Note
Click here to download the full example code
5.1.20.1. SeriesAnalysis: Using Python Embedding
met_tool_wrapper/SeriesAnalysis/SeriesAnalysis_python _embedding.conf
Scientific Objective
None. This is a demonstration of using python embedding to pass and read in external files, which have a data format that MET would not otherwise be able to parse.
Datasets
METplus Components
This use case utilizes the METplus SeriesAnalysis wrapper to search for files as determined by the Python script.
METplus Workflow
SeriesAnalysis is the only tool called in this example. It processes simple text files with no timining or meteorological information to demonstrate how SeriesAnalysis can be run utilizing Python Embedding.
METplus Configuration
METplus first loads all of the configuration files found in parm/metplus_config, then it loads any configuration files passed to METplus via the command line with the -c option, i.e. -c parm/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis_python_embedding.conf
# SeriesAnalysis using Python Embedding METplus Configuration
[config]
PROCESS_LIST = SeriesAnalysis
LOOP_ORDER = processes
LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d%H
INIT_BEG=2005080700
INIT_END=2005080700
INIT_INCREMENT = 12H
LEAD_SEQ = 12
SERIES_ANALYSIS_CUSTOM_LOOP_LIST =
SERIES_ANALYSIS_DESC =
SERIES_ANALYSIS_CAT_THRESH =
SERIES_ANALYSIS_VLD_THRESH =
SERIES_ANALYSIS_BLOCK_SIZE =
SERIES_ANALYSIS_CTS_LIST =
SERIES_ANALYSIS_REGRID_TO_GRID =
SERIES_ANALYSIS_REGRID_METHOD =
SERIES_ANALYSIS_REGRID_WIDTH =
SERIES_ANALYSIS_REGRID_VLD_THRESH =
SERIES_ANALYSIS_REGRID_SHAPE =
SERIES_ANALYSIS_RUNTIME_FREQ = RUN_ONCE_PER_INIT_OR_VALID
SERIES_ANALYSIS_RUN_ONCE_PER_STORM_ID = False
#LOG_SERIES_ANALYSIS_VERBOSITY = 2
SERIES_ANALYSIS_IS_PAIRED = False
SERIES_ANALYSIS_CONFIG_FILE = {CONFIG_DIR}/SeriesAnalysisConfig_wrapped
SERIES_ANALYSIS_STAT_LIST = TOTAL, RMSE, FBAR, OBAR
MODEL = PYTHON
OBTYPE = ANALYS
FCST_SERIES_ANALYSIS_INPUT_DATATYPE = PYTHON_NUMPY
OBS_SERIES_ANALYSIS_INPUT_DATATYPE = PYTHON_NUMPY
FCST_VAR1_NAME = {MET_INSTALL_DIR}/share/met/python/read_ascii_numpy.py MET_PYTHON_INPUT_ARG FCST
OBS_VAR1_NAME = {MET_INSTALL_DIR}/share/met/python/read_ascii_numpy.py MET_PYTHON_INPUT_ARG OBS
[dir]
CONFIG_DIR={PARM_BASE}/met_config
FCST_SERIES_ANALYSIS_INPUT_DIR = {INPUT_BASE}/met_test/data/python
OBS_SERIES_ANALYSIS_INPUT_DIR = {INPUT_BASE}/met_test/data/python
SERIES_ANALYSIS_CLIMO_MEAN_INPUT_DIR =
SERIES_ANALYSIS_CLIMO_STDEV_INPUT_DIR =
SERIES_ANALYSIS_OUTPUT_DIR = {OUTPUT_BASE}/met_tool_wrapper/SeriesAnalysis
[filename_templates]
FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = fcst.txt, fcst.txt
OBS_SERIES_ANALYSIS_INPUT_TEMPLATE = obs.txt, obs.txt
SERIES_ANALYSIS_OUTPUT_TEMPLATE = python_sa.nc
SERIES_ANALYSIS_CLIMO_MEAN_INPUT_TEMPLATE =
SERIES_ANALYSIS_CLIMO_STDEV_INPUT_TEMPLATE =
MET Configuration
METplus sets environment variables based on user settings in the METplus configuration file. See How METplus controls MET config file settings for more details.
YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!
If there is a setting in the MET configuration file that is currently not supported by METplus you’d like to control, please refer to: Overriding Unsupported MET config file settings
Note
See the SeriesAnalysis MET Configuration section of the User’s Guide for more information on the environment variables used in the file below:
////////////////////////////////////////////////////////////////////////////////
//
// Series-Analysis configuration file.
//
// For additional information, see the MET_BASE/config/README file.
//
////////////////////////////////////////////////////////////////////////////////
//
// Output model name to be written
//
${METPLUS_MODEL}
//
// Output description to be written
//
${METPLUS_DESC}
//
// Output observation type to be written
//
${METPLUS_OBTYPE}
////////////////////////////////////////////////////////////////////////////////
//
// Verification grid
// May be set separately in each "field" entry
//
${METPLUS_REGRID_DICT}
////////////////////////////////////////////////////////////////////////////////
censor_thresh = [];
censor_val = [];
${METPLUS_CAT_THRESH}
cnt_thresh = [ NA ];
cnt_logic = UNION;
//
// Forecast and observation fields to be verified
//
fcst = {
${METPLUS_FCST_FILE_TYPE}
${METPLUS_FCST_FIELD}
}
obs = {
${METPLUS_OBS_FILE_TYPE}
${METPLUS_OBS_FIELD}
}
////////////////////////////////////////////////////////////////////////////////
//
// Climatology data
//
//climo_mean = {
${METPLUS_CLIMO_MEAN_DICT}
//climo_stdev = {
${METPLUS_CLIMO_STDEV_DICT}
////////////////////////////////////////////////////////////////////////////////
//
// Confidence interval settings
//
ci_alpha = [ 0.05 ];
boot = {
interval = PCTILE;
rep_prop = 1.0;
n_rep = 0;
rng = "mt19937";
seed = "";
}
////////////////////////////////////////////////////////////////////////////////
//
// Verification masking regions
//
mask = {
grid = "";
poly = "";
}
//
// Number of grid points to be processed concurrently. Set smaller to use
// less memory but increase the number of passes through the data.
//
${METPLUS_BLOCK_SIZE}
//
// Ratio of valid matched pairs to compute statistics for a grid point
//
${METPLUS_VLD_THRESH}
////////////////////////////////////////////////////////////////////////////////
//
// Statistical output types
//
output_stats = {
fho = [];
ctc = [];
${METPLUS_CTS_LIST}
mctc = [];
mcts = [];
${METPLUS_STAT_LIST}
sl1l2 = [];
sal1l2 = [];
pct = [];
pstd = [];
pjc = [];
prc = [];
}
////////////////////////////////////////////////////////////////////////////////
rank_corr_flag = FALSE;
tmp_dir = "/tmp";
//version = "V10.0";
////////////////////////////////////////////////////////////////////////////////
${METPLUS_MET_CONFIG_OVERRIDES}
Python Embedding
This use case calls a Python script to read the input data. The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_numpy.py
Running METplus
This use case can be run two ways:
Passing in SeriesAnalysis_python_embedding.conf then a user-specific system configuration file:
run_metplus.py -c /path/to/METplus/parm/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis_python_embedding.conf -c /path/to/user_system.conf
Modifying the configurations in parm/metplus_config, then passing in SeriesAnalysis_python_embedding.conf:
run_metplus.py -c /path/to/METplus/parm/use_cases/met_tool_wrapper/SeriesAnalysis/SeriesAnalysis_python_embedding.conf
The former method is recommended. Whether you add them to a user-specific configuration file or modify the metplus_config files, the following variables must be set correctly:
INPUT_BASE - Path to directory where sample data tarballs are unpacked (See Datasets section to obtain tarballs). This is not required to run METplus, but it is required to run the examples in parm/use_cases
OUTPUT_BASE - Path where METplus output will be written. This must be in a location where you have write permissions
MET_INSTALL_DIR - Path to location where MET is installed locally
Example User Configuration File:
[dir]
INPUT_BASE = /path/to/sample/input/data
OUTPUT_BASE = /path/to/output/dir
MET_INSTALL_DIR = /path/to/met-X.Y
NOTE: All of these items must be found under the [dir] section.
Expected Output
A successful run will output the following both to the screen and to the logfile:
INFO: METplus has successfully finished running.
Refer to the value set for OUTPUT_BASE to find where the output data was generated. Output for this use case will be found in met_tool_wrapper/SeriesAnalysis (relative to OUTPUT_BASE) and will contain the following file:
python_sa.nc
Keywords
sphinx_gallery_thumbnail_path = ‘_static/met_tool_wrapper-SeriesAnalysis.png’
Total running time of the script: ( 0 minutes 0.000 seconds)