2. METplus Release Information

Users can view the Stages of the METplus Release Cycle section of the Release Guide for descriptions of the development releases (including beta releases and release candidates), official releases, and bugfix releases for the METplus Components.

2.2. METplus Wrappers Release Notes

When applicable, release notes are followed by the GitHub issue number which describes the bugfix, enhancement, or new feature.

2.2.1. METplus Version 5.0.0 Release Notes (2022-12-??)

Warning

MAJOR CHANGES:

  • The LOOP_ORDER config variable was removed. The variable set in a user’s config file will be ignored in favor of executing the logic that corresponds to LOOP_ORDER = processes, where all times are processed for the first item in the PROCESS_LIST, then all times are processed for the second item in the PROCESS_LIST, etc. This may change the order that commands are executed in a use case, but it should not change the results.

  • The METplus Dockerfile was moved to internal/scripts/docker. It was previously found in scripts/docker.

  • Use cases that include EnsembleStat wrapper will require config file updates. See METplus Wrappers Upgrade Instructions.

  • The default value of SCRUB_STAGING_DIR is now True. This means some intermediate files that are auto-generated by METplus such as file lists and uncompressed files will automatically be removed unless this option is set by the user. These files are typically only used to debug unexpected issues.

  • The default value of METPLUS_CONF now includes the LOG_TIMESTAMP so each METplus run will generate a unique final config file, e.g. metplus_final.conf.20220921121733.

  • Enhancements:

    • Enhance MODE wrapper to support multi-variate MODE (#1585)

    • Allow FCST_IS_PROB variable setting specific to tool (FCST_<tool_name>_IS_PROB) (#1586)

    • Enhance climatology field settings to be consistent with fcst/obs field (#1599)

    • Update Hovmoeller Use case to use updated Hovmoeller plotting (#1650)

    • Update the EnsembleStat wrapper and use case examples to remove ensemble post processing logic (#1816)

    • Enhance logic to consistently create directories (#1657)

    • Create checksum for released code (#262)

    • Add the user ID to the log output at beginning and end of each METplus wrappers run (dtcenter/METplus-Internal#20)

    • Update logic to name final conf and intermediate files with a unique identifier (dtcenter/METplus-Internal#32)

    • Change default logging time information (dtcenter/METplus-Internal#34)

    • Remove LOOP_ORDER config variable (#1687)

    • Add unique identifier for each METplus run to configuration (#1829)

    • StatAnalysis - Support setting multiple jobs (#1842)

    • StatAnalysis - Set MET verbosity (#1772)

    • StatAnalysis - Support using both init/valid variables in string substitution (#1861)

    • StatAnalysis - Allow filename template tags in jobs (#1862)

    • StatAnalysis - Support looping over groups of list items (#1870)

    • StatAnalysis - Allow processing of time ranges other than daily (#1871)

    • StatAnalysis - Add support for using a custom loop list (#1893)

    • Remove MakePlots wrapper (#1843)

    • Add support in EnsembleStat wrapper for setting -ens_mean command line argument (#1569)

    • Enhance METplus to have better signal handling for shutdown events (dtcenter/METplus-Internal#27)

    • TCPairs and TCStat - add support for new config options and command line arguments (#1898)

    • Enhance the GridStat and PointStat wrappers to handle the addition of SEEPS (#1953)

    • SeriesAnalysis - add support for setting mask dictionary (#1926)

    • Update Python requirement to 3.8.6 (#1566)

    • Enhance StatAnalysis wrapper to support now and today (#1669)

    • Clean up use case configuration files (#1402)

    • Add support for creating multiple input datasets (#1694)

  • Bugfixes:

    • PCPCombine - custom loop list does not work for subtract method (#1884)

    • Set level properly in filename template for EnsembleStat forecast input (#1910)

    • Prevent duplicate observation files using a file window if compressed equivalent files exist in same directory (#1939)

    • Allow NA value for <TOOL-NAME>_CLIMO_[MEAN/STDEV]_HOUR_INTERVAL (#1787)

    • Reconcile setting of METPLOTPY_BASE for use cases (#1713)

    • Add support for the {custom} loop string in the MODEL config variable (#1382)

    • Fix PCPCombine extra options removal of semi-colon (#1534)

    • Fix reset of arguments for some wrappers (i.e. GenEnsProd) after each run (#1555)

    • Enhance METDbLoad Wrapper to find MODE .txt files (#1608)

    • Add missing brackets around list variable values for StatAnalysis wrapper (#1641)

    • Allow NA value for <TOOL-NAME>_CLIMO_[MEAN/STDEV]_DAY_INTERVAL (#1653)

  • New Wrappers:

  • New Use Cases:

    • PANDA-C use cases (#1686)

    • MJO-ENSO diagnostics (#1330)

    • Probability of Exceedence for 85th percentile temperatures (#1808)

    • FV3 Physics Tendency plotting via METplotpy (#1852)

    • StatAnalysis Python Embedding using IODA v2.0 (#1453)

    • StatAnalysis Python Embedding to read native grid (u-grid) (#1561)

  • Documentation:

  • Internal:

    • Organize utility scripts used by multiple wrappers (#344)

    • Fix GitHub Actions warnings - update the version of actions and replace set-output (#1863)

    • Update diff logic to handle CSV files that have rounding differences (#1865)

    • Add unit tests for expected failure (dtcenter/METplus-Internal#24)

    • Add instructions in Release Guide for “Recreate an Existing Release” (#1746)

    • Add modulefiles used for installations on various machines (#1749)

    • Document GitHub Discussions procedure for the Contributor’s Guide (#1159)

    • Create a METplus “Release Guide” describing how to build releases for the METplus components (#673)

    • Update documentation about viewing RTD URLs on branches (#1512)

2.3. METplus Wrappers Upgrade Instructions

2.3.1. EnsembleStat/GenEnsProd

Note: If EnsembleStat is not found in the PROCESS_LIST for any use cases, then this section is not relevant.

The METplus v5.0.0 coordinated release includes changes that remove ensemble product generation from EnsembleStat. GenEnsProd is now required to generate ensemble products. There are 3 situations listed below that require slightly different modifications.

2.3.1.1. Case 1: EnsembleStat only generating ensemble products

If the use case had been calling EnsembleStat WITHOUT the -grid_obs or -point_obs command line options, we can assume it was only doing ensemble post-processing. That call to EnsembleStat should be replaced with a call to GenEnsProd instead.

Rename the following variables:

FCST_ENSEMBLE_STAT_INPUT_DIR => GEN_ENS_PROD_INPUT_DIR

FCST_ENSEMBLE_STAT_INPUT_TEMPLATE => GEN_ENS_PROD_INPUT_TEMPLATE

ENSEMBLE_STAT_OUTPUT_DIR => GEN_ENS_PROD_OUTPUT_DIR

ENSEMBLE_STAT_OUTPUT_TEMPLATE => GEN_ENS_PROD_OUTPUT_TEMPLATE and add full filename template for NetCDF output file to end of value, i.e. /gen_ens_prod_{valid?fmt=%Y%m%d_%H%M%S}V_ens.nc

ENSEMBLE_STAT_N_MEMBERS => GEN_ENS_PROD_N_MEMBERS

ENSEMBLE_STAT_ENS_THRESH => GEN_ENS_PROD_ENS_THRESH

ENSEMBLE_STAT_ENS_VLD_THRESH => GEN_ENS_PROD_VLD_THRESH

ENSEMBLE_STAT_ENSEMBLE_FLAG_LATLON => GEN_ENS_PROD_ENSEMBLE_FLAG_LATLON ENSEMBLE_STAT_ENSEMBLE_FLAG_MEAN => GEN_ENS_PROD_ENSEMBLE_FLAG_MEAN ENSEMBLE_STAT_ENSEMBLE_FLAG_STDEV => GEN_ENS_PROD_ENSEMBLE_FLAG_STDEV ENSEMBLE_STAT_ENSEMBLE_FLAG_MINUS => GEN_ENS_PROD_ENSEMBLE_FLAG_MINUS ENSEMBLE_STAT_ENSEMBLE_FLAG_PLUS => GEN_ENS_PROD_ENSEMBLE_FLAG_PLUS ENSEMBLE_STAT_ENSEMBLE_FLAG_MIN => GEN_ENS_PROD_ENSEMBLE_FLAG_MIN ENSEMBLE_STAT_ENSEMBLE_FLAG_MAX => GEN_ENS_PROD_ENSEMBLE_FLAG_MAX ENSEMBLE_STAT_ENSEMBLE_FLAG_RANGE => GEN_ENS_PROD_ENSEMBLE_FLAG_RANGE ENSEMBLE_STAT_ENSEMBLE_FLAG_VLD_COUNT => GEN_ENS_PROD_ENSEMBLE_FLAG_VLD_COUNT ENSEMBLE_STAT_ENSEMBLE_FLAG_FREQUENCY => GEN_ENS_PROD_ENSEMBLE_FLAG_FREQUENCY ENSEMBLE_STAT_ENSEMBLE_FLAG_NEP => GEN_ENS_PROD_ENSEMBLE_FLAG_NEP ENSEMBLE_STAT_ENSEMBLE_FLAG_NMEP => GEN_ENS_PROD_ENSEMBLE_FLAG_NMEP

ENSEMBLE_STAT_REGRID_TO_GRID => GEN_ENS_PROD_REGRID_TO_GRID ENSEMBLE_STAT_REGRID_METHOD => GEN_ENS_PROD_REGRID_METHOD ENSEMBLE_STAT_REGRID_WIDTH => GEN_ENS_PROD_REGRID_WIDTH ENSEMBLE_STAT_REGRID_VLD_THRESH => GEN_ENS_PROD_REGRID_VLD_THRESH ENSEMBLE_STAT_REGRID_SHAPE => GEN_ENS_PROD_REGRID_SHAPE ENSEMBLE_STAT_NBRHD_PROB_WIDTH => GEN_ENS_PROD_NBRHD_PROB_WIDTH ENSEMBLE_STAT_NBRHD_PROB_SHAPE => GEN_ENS_PROD_NBRHD_PROB_SHAPE ENSEMBLE_STAT_NBRHD_PROB_VLD_THRESH => GEN_ENS_PROD_NBRHD_PROB_VLD_THRESH ENSEMBLE_STAT_NMEP_SMOOTH_VLD_THRESH => GEN_ENS_PROD_NMEP_SMOOTH_VLD_THRESH ENSEMBLE_STAT_NMEP_SMOOTH_SHAPE => GEN_ENS_PROD_NMEP_SMOOTH_SHAPE ENSEMBLE_STAT_NMEP_SMOOTH_METHOD => GEN_ENS_PROD_NMEP_SMOOTH_METHOD ENSEMBLE_STAT_NMEP_SMOOTH_WIDTH => GEN_ENS_PROD_NMEP_SMOOTH_WIDTH ENSEMBLE_STAT_NMEP_SMOOTH_GAUSSIAN_DX => GEN_ENS_PROD_NMEP_SMOOTH_GAUSSIAN_DX ENSEMBLE_STAT_NMEP_SMOOTH_GAUSSIAN_RADIUS => GEN_ENS_PROD_NMEP_SMOOTH_GAUSSIAN_RADIUS

If ENS_VAR<n>_ variables are not set:

If no FCST/OBS verification is being performed in the use case using another wrapper, then rename the FCST_VAR<n> variables to ENS_VAR<n>.

e.g.

FCST_VAR1_NAME => ENS_VAR1_NAME FCST_VAR1_LEVELS => ENS_VAR1_LEVELS FCST_VAR2_NAME => ENS_VAR2_NAME FCST_VAR2_LEVELS => ENS_VAR2_LEVELS … etc

If FCST/OBS verification is being performed by another tool, then add ENS_VAR<n> variables using the corresponding FCST_VAR<n> values.

e.g.

ENS_VAR1_NAME = {FCST_VAR1_NAME} ENS_VAR1_LEVELS = {FCST_VAR1_LEVELS} ENS_VAR2_NAME = {FCST_VAR2_NAME} ENS_VAR2_LEVELS = {FCST_VAR2_LEVELS} … etc

Remove the following variables:

Remove any remaining ENSEMBLE_STAT_* variables that are no longer used. Some examples: Remove ENSEMBLE_STAT_ENSEMBLE_FLAG_RANK Remove ENSEMBLE_STAT_ENSEMBLE_FLAG_WEIGHT Remove ENSEMBLE_STAT_MESSAGE_TYPE Remove ENSEMBLE_STAT_OUTPUT_FLAG_ECNT Remove ENSEMBLE_STAT_OUTPUT_FLAG_RPS Remove ENSEMBLE_STAT_OUTPUT_FLAG_RHIST Remove ENSEMBLE_STAT_OUTPUT_FLAG_PHIST Remove ENSEMBLE_STAT_OUTPUT_FLAG_ORANK Remove ENSEMBLE_STAT_OUTPUT_FLAG_SSVAR Remove ENSEMBLE_STAT_OUTPUT_FLAG_RELP Remove ENSEMBLE_STAT_OUTPUT_FLAG_PCT Remove ENSEMBLE_STAT_OUTPUT_FLAG_PSTD Remove ENSEMBLE_STAT_OUTPUT_FLAG_PJC Remove ENSEMBLE_STAT_OUTPUT_FLAG_PRC Remove ENSEMBLE_STAT_OUTPUT_FLAG_ECLV Remove ENSEMBLE_STAT_DUPLICATE_FLAG Remove ENSEMBLE_STAT_SKIP_CONST Remove ENSEMBLE_STAT_OBS_ERROR_FLAG Remove ENSEMBLE_STAT_ENS_SSVAR_BIN_SIZE Remove ENSEMBLE_STAT_ENS_PHIST_BIN_SIZE Remove ENSEMBLE_STAT_CI_ALPHA Remove ENSEMBLE_STAT_MASK_GRID Remove ENSEMBLE_STAT_MASK_POLY Remove ENSEMBLE_STAT_INTERP_FIELD Remove ENSEMBLE_STAT_INTERP_VLD_THRESH Remove ENSEMBLE_STAT_INTERP_SHAPE Remove ENSEMBLE_STAT_INTERP_METHOD Remove ENSEMBLE_STAT_INTERP_WIDTH Remove ENSEMBLE_STAT_OBS_QUALITY_INC/EXC Remove ENSEMBLE_STAT_GRID_WEIGHT_FLAG

2.3.1.2. Case 2: EnsembleStat performing ensemble verification but not generating ensemble products

No changes should be required for this case to continue to work as expected except for removing configuration variables that are no longer used. The use case will no longer generate a _ens.nc file and may create other files (_orank.nc and txt) that contain requested output.

Rename the following variables:

ENSEMBLE_STAT_ENSEMBLE_FLAG_MEAN => ENSEMBLE_STAT_NC_ORANK_FLAG_MEAN ENSEMBLE_STAT_ENSEMBLE_FLAG_RANK => ENSEMBLE_STAT_NC_ORANK_FLAG_RANK ENSEMBLE_STAT_ENSEMBLE_FLAG_WEIGHT => ENSEMBLE_STAT_NC_ORANK_FLAG_WEIGHT ENSEMBLE_STAT_ENSEMBLE_FLAG_VLD_COUNT => ENSEMBLE_STAT_NC_ORANK_FLAG_VLD_COUNT

Remove the following variables:

Remove any ENS_VAR<n>_* variables Remove ENSEMBLE_STAT_ENSEMBLE_FLAG_* ENSEMBLE_STAT_NBRHD_PROB_WIDTH ENSEMBLE_STAT_NBRHD_PROB_SHAPE ENSEMBLE_STAT_NBRHD_PROB_VLD_THRESH ENSEMBLE_STAT_NMEP_SMOOTH_VLD_THRESH ENSEMBLE_STAT_NMEP_SMOOTH_SHAPE ENSEMBLE_STAT_NMEP_SMOOTH_METHOD ENSEMBLE_STAT_NMEP_SMOOTH_WIDTH ENSEMBLE_STAT_NMEP_SMOOTH_GAUSSIAN_DX ENSEMBLE_STAT_NMEP_SMOOTH_GAUSSIAN_RADIUS

2.3.1.3. Case 3: EnsembleStat generating ensemble products and performing ensemble verification

GenEnsProd will need to be added to the PROCESS_LIST in addition to EnsembleStat to generate the ensemble verification output.

PROCESS_LIST = …, EnsembleStat, GenEnsProd, …

Set the input dir and template variables for GenEnsProd to match the values set for FCST input to EnsembleStat. Also set the output dir to match EnsembleStat output dir.

GEN_ENS_PROD_INPUT_DIR = {FCST_ENSEMBLE_STAT_INPUT_DIR} GEN_ENS_PROD_INPUT_TEMPLATE = {FCST_ENSEMBLE_STAT_INPUT_TEMPLATE} GEN_ENS_PROD_OUTPUT_DIR = {ENSEMBLE_STAT_OUTPUT_DIR}

If the EnsembleStat output template is set, then copy the value and add a template for the NetCDF output filename at the end following a forward slash ‘/’ character.

If ENSEMBLE_STAT_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d%H} then set GEN_ENS_PROD_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d%H}/gen_ens_prod_{valid?fmt=%Y%m%d_%H%M%S}V_ens.nc or something similar.

If the EnsembleStat output template is not set, then set GenEnsProd’s template to the desired NetCDF output filename. Here is an example:

GEN_ENS_PROD_OUTPUT_TEMPLATE = gen_ens_prod_{valid?fmt=%Y%m%d_%H%M%S}V_ens.nc

Ensure that any downstream wrappers in the PROCESS_LIST are configured to read the correct GenEnsProd output file instead of the _ens.nc file that was previously generated by EnsembleStat.

If ENS_VAR<n>_ variables are not set, add ENS_VAR<n> variables using the corresponding FCST_ENSEMBLE_STAT_VAR<n> or FCST_VAR<n> values. If FCST_ENSEMBLE_VAR<n>_* variables are set, then use only those values, otherwise use FCST_VAR<n>_*

e.g.

ENS_VAR1_NAME = {FCST_VAR1_NAME} ENS_VAR1_LEVELS = {FCST_VAR1_LEVELS} ENS_VAR2_NAME = {FCST_VAR2_NAME} ENS_VAR2_LEVELS = {FCST_VAR2_LEVELS}

If any of the following ENSEMBLE_STAT_* variables are set in the configuration file, then rename them to the corresponding GEN_ENS_PROD_* variable:

ENSEMBLE_STAT_NBRHD_PROB_WIDTH => GEN_ENS_PROD_NBRHD_PROB_WIDTH ENSEMBLE_STAT_NBRHD_PROB_SHAPE => GEN_ENS_PROD_NBRHD_PROB_SHAPE ENSEMBLE_STAT_NBRHD_PROB_VLD_THRESH => GEN_ENS_PROD_NBRHD_PROB_VLD_THRESH ENSEMBLE_STAT_NMEP_SMOOTH_VLD_THRESH => GEN_ENS_PROD_NMEP_SMOOTH_VLD_THRESH ENSEMBLE_STAT_NMEP_SMOOTH_SHAPE => GEN_ENS_PROD_NMEP_SMOOTH_SHAPE ENSEMBLE_STAT_NMEP_SMOOTH_METHOD => GEN_ENS_PROD_NMEP_SMOOTH_METHOD ENSEMBLE_STAT_NMEP_SMOOTH_WIDTH => GEN_ENS_PROD_NMEP_SMOOTH_WIDTH ENSEMBLE_STAT_NMEP_SMOOTH_GAUSSIAN_DX => GEN_ENS_PROD_NMEP_SMOOTH_GAUSSIAN_DX ENSEMBLE_STAT_NMEP_SMOOTH_GAUSSIAN_RADIUS => GEN_ENS_PROD_NMEP_SMOOTH_GAUSSIAN_RADIUS FCST_ENSEMBLE_STAT_INPUT_GRID_DATATYPE => GEN_ENS_PROD_INPUT_DATATYPE

If any of the following ENSEMBLE_STAT_* variables are set in the configuration file, then set the corresponding GEN_ENS_PROD_* variables to the same value or reference the ENSEMBLE_STAT_* version.

GEN_ENS_PROD_N_MEMBERS = {ENSEMBLE_STAT_N_MEMBERS} GEN_ENS_PROD_ENS_THRESH = {ENSEMBLE_STAT_ENS_THRESH} GEN_ENS_PROD_REGRID_TO_GRID = {ENSEMBLE_STAT_REGRID_TO_GRID} GEN_ENS_PROD_REGRID_METHOD = {ENSEMBLE_STAT_REGRID_METHOD} GEN_ENS_PROD_REGRID_WIDTH = {ENSEMBLE_STAT_REGRID_WIDTH} GEN_ENS_PROD_VLD_THRESH = {ENSEMBLE_STAT_VLD_THRESH} GEN_ENS_PROD_SHAPE = {ENSEMBLE_STAT_SHAPE}

If any of the following ENSEMBLE_STAT_ENSEMBLE_FLAG_* variables are set in the configuration file, then set the corresponding GEN_ENS_PROD_ENSEMBLE_FLAG_* variables to the same value.

ENSEMBLE_STAT_ENSEMBLE_FLAG_LATLON ENSEMBLE_STAT_ENSEMBLE_FLAG_MEAN ENSEMBLE_STAT_ENSEMBLE_FLAG_STDEV ENSEMBLE_STAT_ENSEMBLE_FLAG_MINUS ENSEMBLE_STAT_ENSEMBLE_FLAG_PLUS ENSEMBLE_STAT_ENSEMBLE_FLAG_MIN ENSEMBLE_STAT_ENSEMBLE_FLAG_MAX ENSEMBLE_STAT_ENSEMBLE_FLAG_RANGE ENSEMBLE_STAT_ENSEMBLE_FLAG_VLD_COUNT ENSEMBLE_STAT_ENSEMBLE_FLAG_FREQUENCY ENSEMBLE_STAT_ENSEMBLE_FLAG_NEP ENSEMBLE_STAT_ENSEMBLE_FLAG_NMEP

e.g.

If ENSEMBLE_STAT_ENSEMBLE_FLAG_LATLON = TRUE Add GEN_ENS_PROD_ENSEMBLE_FLAG_LATLON = TRUE

If any of the following ENSEMBLE_STAT_ENSEMBLE_FLAG_* variables are set in the configuration file, then rename them to the corresponding ENSEMBLE_STAT_NC_ORANK_FLAG_* variables.

ENSEMBLE_STAT_ENSEMBLE_FLAG_LATLON => ENSEMBLE_STAT_NC_ORANK_FLAG_LATLON ENSEMBLE_STAT_ENSEMBLE_FLAG_MEAN => ENSEMBLE_STAT_NC_ORANK_FLAG_MEAN ENSEMBLE_STAT_ENSEMBLE_FLAG_VLD_COUNT => ENSEMBLE_STAT_NC_ORANK_FLAG_VLD_COUNT