Note
Go to the end to download the full example code
StatAnalysis: Using Python Embedding
met_tool_wrapper/StatAnalysis/StatAnalysis_python_embedding.conf
Scientific Objective
This demonstrates how the Stat-Analysis tool can tie together results from other MET tools (including PointStat, GridStat, EnsembleStat, and WaveletStat) and provide summary statistical information. Matched pair records are passed into Stat-Analysis using python embedding.
Datasets
METplus Components
This use case utilizes the METplus StatAnalysis wrapper to search for files that are valid at a given run time and generate a command to run the MET tool stat_analysis.
METplus Workflow
StatAnalysis is the only tool called in this example. It processes the following run times:
METplus Configuration
METplus first loads all of the configuration files found in parm/metplus_config, then it loads any configuration files passed to METplus via the command line with the -c option, i.e. -c parm/use_cases/met_tool_wrapper/StatAnalysis/StatAnalysis.conf
[config]
# Documentation for this use case can be found at
# https://metplus.readthedocs.io/en/latest/generated/met_tool_wrapper/StatAnalysis/StatAnalysis_python_embedding.html
# For additional information, please see the METplus Users Guide.
# https://metplus.readthedocs.io/en/latest/Users_Guide
###
# Processes to run
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list
###
PROCESS_LIST = StatAnalysis
###
# Time Info
# LOOP_BY options are INIT, VALID, RETRO, and REALTIME
# If set to INIT or RETRO:
# INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set
# If set to VALID or REALTIME:
# VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set
# LEAD_SEQ is the list of forecast leads to process
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control
###
LOOP_BY = VALID
VALID_TIME_FMT = %Y%m%d
VALID_BEG=20070331
VALID_END=20070331
VALID_INCREMENT = 12H
###
# File I/O
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#directory-and-filename-template-info
###
MODEL1 = WRF
MODEL1_OBTYPE = ADPUPA
MODEL1_STAT_ANALYSIS_LOOKIN_DIR = python {MET_INSTALL_DIR}/share/met/python/examples/read_ascii_mpr.py {INPUT_BASE}/met_test/out/point_stat/point_stat_360000L_20070331_120000V.stat
MODEL1_STAT_ANALYSIS_OUT_STAT_TEMPLATE = {model?fmt=%s}_{obtype?fmt=%s}_valid{valid?fmt=%Y%m%d}_fcstvalidhour{valid_hour?fmt=%H}0000Z_out_stat.stat
STAT_ANALYSIS_OUTPUT_DIR = {OUTPUT_BASE}/met_tool_wrapper/StatAnalysis_python_embedding
###
# StatAnalysis Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#statanalysis
###
#LOG_STAT_ANALYSIS_VERBOSITY = 2
STAT_ANALYSIS_CONFIG_FILE = {PARM_BASE}/met_config/STATAnalysisConfig_wrapped
STAT_ANALYSIS_JOB_NAME = aggregate_stat
STAT_ANALYSIS_JOB_ARGS = -out_line_type sl1l2 -by FCST_VAR -out_stat [out_stat_file]
MODEL_LIST = {MODEL1}
DESC_LIST =
FCST_LEAD_LIST =
OBS_LEAD_LIST =
FCST_VALID_HOUR_LIST = 12
FCST_INIT_HOUR_LIST =
OBS_VALID_HOUR_LIST =
OBS_INIT_HOUR_LIST =
FCST_VAR_LIST =
OBS_VAR_LIST =
FCST_UNITS_LIST =
OBS_UNITS_LIST =
FCST_LEVEL_LIST =
OBS_LEVEL_LIST =
VX_MASK_LIST =
INTERP_MTHD_LIST =
INTERP_PNTS_LIST =
FCST_THRESH_LIST =
OBS_THRESH_LIST =
COV_THRESH_LIST =
ALPHA_LIST =
LINE_TYPE_LIST = MPR
GROUP_LIST_ITEMS = FCST_INIT_HOUR_LIST
LOOP_LIST_ITEMS = FCST_VALID_HOUR_LIST, MODEL_LIST
MET Configuration
METplus sets environment variables based on user settings in the METplus configuration file. See How METplus controls MET config file settings for more details.
YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!
If there is a setting in the MET configuration file that is currently not supported by METplus you’d like to control, please refer to: Overriding Unsupported MET config file settings
Note
See the StatAnalysis MET Configuration section of the User’s Guide for more information on the environment variables used in the file below:
////////////////////////////////////////////////////////////////////////////////
//
// STAT-Analysis configuration file.
//
// For additional information, see the MET_BASE/config/README file.
//
////////////////////////////////////////////////////////////////////////////////
//
// Filtering input STAT lines by the contents of each column
//
${METPLUS_MODEL}
${METPLUS_DESC}
${METPLUS_FCST_LEAD}
${METPLUS_OBS_LEAD}
${METPLUS_FCST_VALID_BEG}
${METPLUS_FCST_VALID_END}
${METPLUS_FCST_VALID_HOUR}
${METPLUS_OBS_VALID_BEG}
${METPLUS_OBS_VALID_END}
${METPLUS_OBS_VALID_HOUR}
${METPLUS_FCST_INIT_BEG}
${METPLUS_FCST_INIT_END}
${METPLUS_FCST_INIT_HOUR}
${METPLUS_OBS_INIT_BEG}
${METPLUS_OBS_INIT_END}
${METPLUS_OBS_INIT_HOUR}
${METPLUS_FCST_VAR}
${METPLUS_OBS_VAR}
${METPLUS_FCST_UNITS}
${METPLUS_OBS_UNITS}
${METPLUS_FCST_LEVEL}
${METPLUS_OBS_LEVEL}
${METPLUS_OBTYPE}
${METPLUS_VX_MASK}
${METPLUS_INTERP_MTHD}
${METPLUS_INTERP_PNTS}
${METPLUS_FCST_THRESH}
${METPLUS_OBS_THRESH}
${METPLUS_COV_THRESH}
${METPLUS_ALPHA}
${METPLUS_LINE_TYPE}
column = [];
weight = [];
////////////////////////////////////////////////////////////////////////////////
//
// Array of STAT-Analysis jobs to be performed on the filtered data
//
${METPLUS_JOBS}
////////////////////////////////////////////////////////////////////////////////
//
// Confidence interval settings
//
out_alpha = 0.05;
boot = {
interval = PCTILE;
rep_prop = 1.0;
n_rep = 0;
rng = "mt19937";
seed = "";
}
////////////////////////////////////////////////////////////////////////////////
//
// WMO mean computation logic
//
wmo_sqrt_stats = [ "CNT:FSTDEV", "CNT:OSTDEV", "CNT:ESTDEV",
"CNT:RMSE", "CNT:RMSFA", "CNT:RMSOA",
"VCNT:FS_RMS", "VCNT:OS_RMS", "VCNT:RMSVE",
"VCNT:FSTDEV", "VCNT:OSTDEV" ];
wmo_fisher_stats = [ "CNT:PR_CORR", "CNT:SP_CORR",
"CNT:KT_CORR", "CNT:ANOM_CORR" ];
////////////////////////////////////////////////////////////////////////////////
//hss_ec_value =
${METPLUS_HSS_EC_VALUE}
rank_corr_flag = FALSE;
vif_flag = FALSE;
tmp_dir = "${MET_TMP_DIR}";
//version = "V10.0";
${METPLUS_MET_CONFIG_OVERRIDES}
Python Embedding
This use case calls a Python script to read matched pair lines from an input source. The Python script is stored in the MET repository: /path/to/MET/installation/share/met/python/read_ascii_mpr.py
Running METplus
It is recommended to run this use case by:
Passing in StatAnalysis_python_embedding.conf then a user-specific system configuration file:
run_metplus.py -c /path/to/StatAnalysis_python_embedding.conf -c /path/to/user_system.conf
The following METplus configuration variables must be set correctly to run this example.:
INPUT_BASE - Path to directory where sample data tarballs are unpacked (See Datasets section to obtain tarballs).
OUTPUT_BASE - Path where METplus output will be written. This must be in a location where you have write permissions
MET_INSTALL_DIR - Path to location where MET is installed locally
Example User Configuration File:
[dir]
INPUT_BASE = /path/to/sample/input/data
OUTPUT_BASE = /path/to/output/dir
MET_INSTALL_DIR = /path/to/met-X.Y
NOTE: All of these items must be found under the [dir] section.
Expected Output
A successful run will output the following both to the screen and to the logfile:
INFO: METplus has successfully finished running.
Refer to the value set for OUTPUT_BASE to find where the output data was generated. Output for this use case will be found in met_tool_wrapper/StatAnalysis_python_embedding (relative to OUTPUT_BASE) and will contain the following file:
WRF_ADPSFC_valid20050807_fcstvalidhour120000Z_out_stat.stat
Keywords
Note
StatAnalysisToolUseCase
PythonEmbeddingFileUseCase
Navigate to the METplus Quick Search for Use Cases page to discover other similar use cases.
sphinx_gallery_thumbnail_path = ‘_static/met_tool_wrapper-StatAnalysis.png’
Total running time of the script: (0 minutes 0.000 seconds)