Note
Go to the end to download the full example code
GenEnsProd: Basic Use Case
met_tool_wrapper/GenEnsProd/GenEnsProd.conf
Scientific Objective
Generate ensemble products. This use case demonstrates how to configure the gen_ens_prod tool if you expect that there will occasionally be missing ensembles. 7 ensemble paths are specified but only 6 of them exist in the sample input data set. The wrapper will mark ensembles that are not found with the MISSING keyword in the file-list file that is read by the tool. Also, one of the ensembles is listed as the control member. The gen_ens_prod application will error and exit if the control member is included in the ensemble list, but the GenEnsProd wrapper will automatically remove the control member from the ensemble list. This makes it easier to configure the tool to change the control member without having to change the ensemble list. The number of expected members (defined with GEN_ENS_PROD_N_MEMBERS) is 6 (7 members - 1 control member). The actual number of ensemble members that will be found in this example is 5 (arw-tom-gep4 is not included). The ens.ens_thresh value (defined by GEN_ENS_PROD_ENS_THRESH) is set to 0.8. There are ~0.833 (5/6) valid ensemble members so the application will run.
Datasets
Input: WRF ARW ensemble 24 hour precipitation accumulation
Location: All of the input data required for this use case can be found in the met_test sample data tarball. Click here to the METplus releases page and download sample data for the appropriate release: https://github.com/dtcenter/METplus/releases This tarball should be unpacked into the directory that you will set the value of INPUT_BASE. See the Running METplus section for more information.
METplus Components
This use case utilizes the METplus GenEnsProd wrapper to generate a command to run the MET tool gen_ens_prod if all required files are found.
METplus Workflow
GenEnsProd is the only tool called in this example. It processes the following run time(s):
METplus Configuration
parm/use_cases/met_tool_wrapper/GenEnsProd/GenEnsProd.conf
[config]
# Documentation for this use case can be found at
# https://metplus.readthedocs.io/en/latest/generated/met_tool_wrapper/GenEnsProd/GenEnsProd.html
# For additional information, please see the METplus Users Guide.
# https://metplus.readthedocs.io/en/latest/Users_Guide
###
# Processes to run
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list
###
PROCESS_LIST = GenEnsProd
###
# Time Info
# LOOP_BY options are INIT, VALID, RETRO, and REALTIME
# If set to INIT or RETRO:
# INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set
# If set to VALID or REALTIME:
# VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set
# LEAD_SEQ is the list of forecast leads to process
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control
###
LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d%H
INIT_BEG=2009123112
INIT_END=2009123112
INIT_INCREMENT = 12H
LEAD_SEQ = 24H
###
# File I/O
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#directory-and-filename-template-info
###
GEN_ENS_PROD_INPUT_DIR = {INPUT_BASE}/met_test/data/sample_fcst
GEN_ENS_PROD_INPUT_TEMPLATE =
{init?fmt=%Y%m%d%H}/arw-fer-gep1/d01_{init?fmt=%Y%m%d%H}_{lead?fmt=%3H}00.grib,
{init?fmt=%Y%m%d%H}/arw-sch-gep2/d01_{init?fmt=%Y%m%d%H}_{lead?fmt=%3H}00.grib,
{init?fmt=%Y%m%d%H}/arw-tom-gep3/d01_{init?fmt=%Y%m%d%H}_{lead?fmt=%3H}00.grib,
{init?fmt=%Y%m%d%H}/arw-tom-gep4/d01_{init?fmt=%Y%m%d%H}_{lead?fmt=%3H}00.grib,
{init?fmt=%Y%m%d%H}/arw-fer-gep5/d01_{init?fmt=%Y%m%d%H}_{lead?fmt=%3H}00.grib,
{init?fmt=%Y%m%d%H}/arw-sch-gep6/d01_{init?fmt=%Y%m%d%H}_{lead?fmt=%3H}00.grib,
{init?fmt=%Y%m%d%H}/arw-tom-gep7/d01_{init?fmt=%Y%m%d%H}_{lead?fmt=%3H}00.grib
GEN_ENS_PROD_CTRL_INPUT_DIR = {INPUT_BASE}/met_test/data/sample_fcst
GEN_ENS_PROD_CTRL_INPUT_TEMPLATE =
{init?fmt=%Y%m%d%H}/arw-fer-gep1/d01_{init?fmt=%Y%m%d%H}_{lead?fmt=%3H}00.grib
GEN_ENS_PROD_OUTPUT_DIR = {OUTPUT_BASE}/gen_ens_prod
GEN_ENS_PROD_OUTPUT_TEMPLATE = gen_ens_prod_{valid?fmt=%Y%m%d_%H%M%S}V_ens.nc
###
# Field Info
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#field-info
###
ENS_VAR1_NAME = APCP
ENS_VAR1_LEVELS = A24
ENS_VAR1_THRESH = >0.0, >=10.0
ENS_VAR1_OPTIONS = ensemble_flag = TRUE
ENS_VAR2_NAME = REFC
ENS_VAR2_LEVELS = L0
ENS_VAR2_THRESH = >=35.0
ENS_VAR2_OPTIONS = GRIB1_ptv = 129
ENS_VAR3_NAME = UGRD
ENS_VAR3_LEVELS = Z10
ENS_VAR3_THRESH = >=5.0
ENS_VAR4_NAME = VGRD
ENS_VAR4_LEVELS = Z10
ENS_VAR4_THRESH = >=5.0
ENS_VAR5_NAME = WIND
ENS_VAR5_LEVELS = Z10
ENS_VAR5_THRESH = >=5.0
###
# GenEnsProd Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#genensprod
###
GEN_ENS_PROD_N_MEMBERS = 6
#LOG_GEN_ENS_PROD_VERBOSITY = 2
# MODEL = WRF
# GEN_ENS_PROD_DESC = NA
#GEN_ENS_PROD_REGRID_TO_GRID = NONE
#GEN_ENS_PROD_REGRID_METHOD = NEAREST
#GEN_ENS_PROD_REGRID_WIDTH = 1
#GEN_ENS_PROD_REGRID_VLD_THRESH = 0.5
#GEN_ENS_PROD_REGRID_SHAPE = SQUARE
#GEN_ENS_PROD_REGRID_CONVERT =
#GEN_ENS_PROD_REGRID_CENSOR_THRESH =
#GEN_ENS_PROD_REGRID_CENSOR_VAL =
#GEN_ENS_PROD_CENSOR_THRESH =
#GEN_ENS_PROD_CENSOR_VAL =
#GEN_ENS_PROD_NORMALIZE =
#GEN_ENS_PROD_CAT_THRESH =
#GEN_ENS_PROD_NC_VAR_STR =
GEN_ENS_PROD_ENS_THRESH = 0.8
#GEN_ENS_PROD_VLD_THRESH = 1.0
#GEN_ENS_PROD_NBRHD_PROB_WIDTH = 5
#GEN_ENS_PROD_NBRHD_PROB_SHAPE = CIRCLE
#GEN_ENS_PROD_NBRHD_PROB_VLD_THRESH = 0.0
#GEN_ENS_PROD_NMEP_SMOOTH_VLD_THRESH = 0.0
#GEN_ENS_PROD_NMEP_SMOOTH_SHAPE = CIRCLE
#GEN_ENS_PROD_NMEP_SMOOTH_GAUSSIAN_DX = 81.27
#GEN_ENS_PROD_NMEP_SMOOTH_GAUSSIAN_RADIUS = 120
#GEN_ENS_PROD_NMEP_SMOOTH_METHOD = GAUSSIAN
#GEN_ENS_PROD_NMEP_SMOOTH_WIDTH = 1
#GEN_ENS_PROD_CLIMO_MEAN_FILE_NAME =
#GEN_ENS_PROD_CLIMO_MEAN_FIELD =
#GEN_ENS_PROD_CLIMO_MEAN_REGRID_METHOD =
#GEN_ENS_PROD_CLIMO_MEAN_REGRID_WIDTH =
#GEN_ENS_PROD_CLIMO_MEAN_REGRID_VLD_THRESH =
#GEN_ENS_PROD_CLIMO_MEAN_REGRID_SHAPE =
#GEN_ENS_PROD_CLIMO_MEAN_TIME_INTERP_METHOD =
#GEN_ENS_PROD_CLIMO_MEAN_MATCH_MONTH =
#GEN_ENS_PROD_CLIMO_MEAN_DAY_INTERVAL = 31
#GEN_ENS_PROD_CLIMO_MEAN_HOUR_INTERVAL = 6
#GEN_ENS_PROD_CLIMO_STDEV_FILE_NAME =
#GEN_ENS_PROD_CLIMO_STDEV_FIELD =
#GEN_ENS_PROD_CLIMO_STDEV_REGRID_METHOD =
#GEN_ENS_PROD_CLIMO_STDEV_REGRID_WIDTH =
#GEN_ENS_PROD_CLIMO_STDEV_REGRID_VLD_THRESH =
#GEN_ENS_PROD_CLIMO_STDEV_REGRID_SHAPE =
#GEN_ENS_PROD_CLIMO_STDEV_TIME_INTERP_METHOD =
#GEN_ENS_PROD_CLIMO_STDEV_MATCH_MONTH =
#GEN_ENS_PROD_CLIMO_STDEV_DAY_INTERVAL = 31
#GEN_ENS_PROD_CLIMO_STDEV_HOUR_INTERVAL = 6
#GEN_ENS_PROD_ENSEMBLE_FLAG_LATLON = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_MEAN = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_STDEV = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_MINUS = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_PLUS = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_MIN = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_MAX = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_RANGE = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_VLD_COUNT = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_FREQUENCY = TRUE
#GEN_ENS_PROD_ENSEMBLE_FLAG_NEP = FALSE
#GEN_ENS_PROD_ENSEMBLE_FLAG_NMEP = FALSE
#GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO = FALSE
#GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDP = FALSE
#GEN_ENS_PROD_ENS_MEMBER_IDS =
#GEN_ENS_PROD_CONTROL_ID =
MET Configuration
Note
See the GenEnsProd MET Configuration section of the User’s Guide for more information on the environment variables used in the file below.
parm/met_config/GenEnsProdConfig_wrapped
////////////////////////////////////////////////////////////////////////////////
//
// Gen-Ens-Prod configuration file.
//
// For additional information, please see the MET Users Guide.
//
////////////////////////////////////////////////////////////////////////////////
//
// Output model name to be written
//
//model =
${METPLUS_MODEL}
//
// Output description to be written
// May be set separately in each "obs.field" entry
//
//desc =
${METPLUS_DESC}
////////////////////////////////////////////////////////////////////////////////
//
// Verification grid
// May be set separately in each "field" entry
//
//regrid = {
${METPLUS_REGRID_DICT}
////////////////////////////////////////////////////////////////////////////////
//
// May be set separately in each "field" entry
//
//censor_thresh =
${METPLUS_CENSOR_THRESH}
//censor_val =
${METPLUS_CENSOR_VAL}
//normalize =
${METPLUS_NORMALIZE}
//cat_thresh =
${METPLUS_CAT_THRESH}
//nc_var_str =
${METPLUS_NC_VAR_STR}
//
// Ensemble fields to be processed
//
ens = {
//file_type =
${METPLUS_ENS_FILE_TYPE}
//ens_thresh =
${METPLUS_ENS_THRESH}
//vld_thresh =
${METPLUS_VLD_THRESH}
//field =
${METPLUS_ENS_FIELD}
}
//ens_member_ids =
${METPLUS_ENS_MEMBER_IDS}
//control_id =
${METPLUS_CONTROL_ID}
////////////////////////////////////////////////////////////////////////////////
//
// Neighborhood ensemble probabilities
//
//nbrhd_prob = {
${METPLUS_NBRHD_PROB_DICT}
//
// NMEP smoothing methods
//
//nmep_smooth = {
${METPLUS_NMEP_SMOOTH_DICT}
////////////////////////////////////////////////////////////////////////////////
//
// Climatology data
//
//climo_mean = {
${METPLUS_CLIMO_MEAN_DICT}
//climo_stdev = {
${METPLUS_CLIMO_STDEV_DICT}
////////////////////////////////////////////////////////////////////////////////
//
// Ensemble product output types
// May be set separately in each "ens.field" entry
//
//ensemble_flag = {
${METPLUS_ENSEMBLE_FLAG_DICT}
////////////////////////////////////////////////////////////////////////////////
//version = "V10.1.0";
////////////////////////////////////////////////////////////////////////////////
tmp_dir = "${MET_TMP_DIR}";
${METPLUS_MET_CONFIG_OVERRIDES}
Running METplus
Provide the use case .conf configuration file to the run_metplus.py script.
/path/to/METplus/parm/use_cases/met_tool_wrapper/GenEnsProd/GenEnsProd.conf
See the Running METplus section of the System Configuration chapter for more details.
Expected Output
A successful run will output the following to the screen and the logfile:
INFO: METplus has successfully finished running.
Refer to the value set for OUTPUT_BASE to find where the output data was generated. Output for this use case will be found in met_tool_wrapper/gen_ens_prod (relative to OUTPUT_BASE) and will contain the following file(s):
gen_ens_prod_20100101_120000V_ens.nc
A file-list file will also be generated in stage/file_lists called:
20091231120000_24_gen_ens_prod.txt
It should contain a list of 6 files in {INPUT_BASE} with 1 file marked as missing because it was not found:
file_list
{INPUT_BASE}/met_test/data/sample_fcst/2009123112/arw-sch-gep2/d01_2009123112_02400.grib
{INPUT_BASE}/met_test/data/sample_fcst/2009123112/arw-tom-gep3/d01_2009123112_02400.grib
MISSING/{INPUT_BASE}/met_test/data/sample_fcst/2009123112/arw-tom-gep4/d01_2009123112_02400.grib
{INPUT_BASE}/met_test/data/sample_fcst/2009123112/arw-fer-gep5/d01_2009123112_02400.grib
{INPUT_BASE}/met_test/data/sample_fcst/2009123112/arw-sch-gep6/d01_2009123112_02400.grib
{INPUT_BASE}/met_test/data/sample_fcst/2009123112/arw-tom-gep7/d01_2009123112_02400.grib
Keywords
Note
GenEnsProdToolUseCase
GRIBFileUseCase
EnsembleAppUseCase
Navigate to METplus Quick Search for Use Cases to discover other similar use cases.
sphinx_gallery_thumbnail_path = ‘_static/met_tool_wrapper-GenEnsProd.png’
Total running time of the script: (0 minutes 0.000 seconds)