1. Overview

1.1. Purpose and organization of the User’s Guide

The goal of this User’s Guide is to equip users with the information needed to use the Model Evaluation Tools (MET) and its companion package METplus Wrappers. MET is a set of verification tools developed and supported to community via the Developmental Testbed Center (DTC) for use by the numerical weather prediction community. METplus Wrappers is a suite of Python wrappers and ancillary scripts to enhance the user’s ability to quickly set-up and run MET. Over the next few years, METplus Wrappers will become the authoritative repository for verification of the Unified Forecast System.

The METplus Wrappers User’s Guide is organized as follows. An overview of METplus Wrappers can be found below. Software Installation contains basic information about how to get started with METplus Wrappers - including system requirements, required software, and how to download METplus Wrappers. System Configuration provides information about configuring your environment and METplus Wrappers installation.

1.2. The Developmental Testbed Center (DTC)

METplus Wrappers has been developed, and will be maintained and enhanced, by the Developmental Testbed Center (DTC; http://www.dtcenter.org/ ). The main goal of the DTC is to serve as a bridge between operations and research and to facilitate the activities of these two important components of the numerical weather prediction (NWP) community. The DTC provides an environment that is functionally equivalent to the operational environment in which the research community can test model enhancements; the operational community benefits from DTC testing and evaluation of models before new models are implemented operationally. METplus Wrappers serves both the research and operational communities in this way - offering capabilities for researchers to test their own enhancements to models and providing a capability for the DTC to evaluate the strengths and weaknesses of advances in NWP prior to operational implementation.

METplus Wrappers will also be available to DTC visitors and the NOAA Unified Forecast System (UFS) and NCAR System for Integrated Modeling of the Atmosphere (SIMA) modeling communities for testing and evaluation of new model capabilities, applications in new environments, and so on. The METplus Wrappers release schedule is coincident with the MET release schedule and the METplus Wrappers major release number is six less than the MET major release number (e.g. MET 8.X is released with METplus Wrappers 2.X).

1.3. METplus Wrappers goals and design philosophy

METplus Wrappers is a Python scripting infrastructure for the MET tools. The primary goal of METplus Wrappers development is to provide MET users with a highly configurable and simple means to perform model verification using the MET tools. Prior to the availability of METplus Wrappers, users who had more complex verifications that required the use of more than one MET tool were faced with setting up multiple MET config files and creating some automation scripts to perform the verification. METplus Wrappers provides the user with the infrastructure to modularly create the necessary steps to perform such verifications.

METplus Wrappers has been designed to be modular and adaptable. This is accomplished through wrapping the MET tools with Python and the use of hierarchical configuration files to enable users to readily customize their verification environments. Wrappers can be run individually, or as a group of wrappers that represent a sequence of MET processes. New wrappers can readily be added to the METplus Wrappers package due to this modular design. Currently, METplus Wrappers can easily be applied by any user on their own computer platform that supports Python 3.6. We have deprecated support to Python 2.7.

The METplus Wrappers code and documentation is maintained by the DTC in Boulder, Colorado. METplus Wrappers is freely available to the modeling, verification, and operational communities, including universities, governments, the private sector, and operational modeling and prediction centers through a publicly accessible GitHub repository. Refer to Getting the METplus Wrappers source code for simple examples of obtaining METplus Wrappers.

1.4. METplus Wrappers Components

The major components of the METplus Wrappers package are METplus Python wrappers to the MET tools, MET configuration files and a hierarchy of METplus Wrappers configuration files. Some Python wrappers do not correspond to a particular MET tool, but wrap utilities to extend METplus functionality.

1.5. METplus Release Notes

1.5.2. METplus Version 4.0.0 release notes (20210510)

  • Bugfixes:

    • Changed default values in wrapped MET config files to align with actual default values in MET config files (Reconcile Default Values)

    • Fix bug causing GridStat fatal error (#740)

    • Add support for comparing inputs using a mix of python embedding and non-embedding (#684)

    • Fix quick search links (#687)

    • Align the user guide with get_relativedelta() in time_util.py (#579)

    • Fix CyclonePlotter cartopy mapping issues (#850, #803)

  • Enhancements:

    • Rename master_metplus.py script to run_metplus.py (#794)

    • Update setting of environment variables for MET config files to add support for all to METPLUS_ vars (#768)

    • Add support for many commonly changed MET config variables (#779, #755, #621, #620)

    • Add support for a UserScript wrapper (#723)

    • Create use case subdirectories (#751)

    • Implement [INIT/VALID]EXCLUDE for time looping (#307)

    • Add files to allow installation of METplus wrappers as a Python package (beta) (#282)

    • Generate PDF of User’s Guide (#551)

    • Add support for MET tc_gen changes in METplus (#871, (#801)

    • Add support for 2 fields with same name and different levels in SeriesBy cases (#852)

    • Enhance PCPCombine wrapper to be able to process multiple fields in one command (#718)

    • Update TCStat config options and wrappers to filter data by excluding strings (#857)

    • Support METplus to run from a driver script (#569)

    • Refactor field info parsing to read once then substitute time info for each run time (#880)

    • Enhance Python embedding logic to allow multiple level values (#719)

    • Enhance Python embedding logic to allow multiple fcst and obs variable levels (#708)

    • Add support for a group of files covering multiple run times for a single analysis in GridDiag (#733)

    • Enhance ascii2nc python embedding script for TC dropsonde data (#734, #731)

    • Support additional configuration variables in EnsembleStat (#748)

    • Ensure backwards compatibility for MET config environment variables (#760)

    • Combine configuration file sections into single config section (#777)

    • Add support for skipping existing output files for all wrappers (#711)

    • Add support for multiple instance of the same tool in the process list (#670)

    • Add GFDL build support in build_components (#614)

    • Decouple PCPCombine, RegridDataPlane, and GridStat wrappers behavior (#602)

    • StatAnalysis run without filtering or config file (#625)

    • Enhance User Diagnostic Feature Relative use case to Run Multiple Diagnostics (#536)

    • Enhance PyEmbedIngest to run RegridDataPlane over Multiple Fields in One Call (#549)

    • Filename templates that have other arguments besides a filename for python embedding fails (#581)

    • Add more logging to tc_gen_wrapper (#576)

    • Prevent crash when improperly formatted filename template is used (#674)

  • New Wrappers:

    • PlotDataPlane

    • UserScript

    • METdbLoad

  • New Use Cases:

    • Air Quality and Comp: EnsembleStat_fcstICAP_obsMODIS_aod

    • Medium Range: UserScript_fcstGEFS_Difficulty_Index

    • Convection Allowing Models: MODE_fcstFV3_obsGOES_BrightnessTemp

    • Convection Allowing Models: MODE_fcstFV3_obsGOES_BrightnessTempObjs

    • Convection Allowing Models: GridStat_fcstFV3_obsGOES_BrightnessTempDmap

    • Data Assimilation: StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface

    • Medium Range: SeriesAnalysis_fcstGFS_obsGFS_FeatureRelative_SeriesByLead_PyEmbed_Multiple_Diagnostics

    • Precipitation: EnsembleStat_fcstWOFS_obsWOFS

    • Seasonal to Subseasonal: TCGen_fcstGFSO_obsBDECKS_GDF_TDF

    • Seasonal to Subseasonal: UserScript_fcstGFS_obsERA_Blocking

    • Seasonal to Subseasonal: UserScript_obsERA_obsOnly_Blocking

    • Seasonal to Subseasonal: UserScript_obsERA_obsOnly_WeatherRegime

    • Seasonal to Subseasonal: UserScript_obsPrecip_obsOnly_Hovmoeller

    • Seasonal to Subseasonal: UserScript_obsPrecip_obsOnly_CrossSpectraPlot

    • TC and Extra TC: CyclonePlotter_fcstGFS_obsGFS_OPC

    • TC and Extra TC: UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF

    • TC and Extra TC: GridStat_fcstHAFS_obsTDR_NetCDF

    • Marine and Coastal: PlotDataPlane_obsHYCOM_coordTripolar

    • MET Tool Wrapper: METdbLoad/METdbLoad

    • MET Tool Wrapper: PlotDataPlane/PlotDataPlane_grib1

    • MET Tool Wrapper: PlotDataPlane/PlotDataPlane_netcdf

    • MET Tool Wrapper: PlotDataPlane/PlotDataPlane_python_embedding

    • MET Tool Wrapper: GridStat/GridStat_python_embedding

    • MET Tool Wrapper: PointStat/PointStat_python_embedding

    • MET Tool Wrapper: MODE/MODE_python_embedding

    • MET Tool Wrapper: PyEmbedIngest_multi_field_one_file

  • Internal:

    • Append semi-colon to end of _OPTIONS variables if not found (#707)

    • Ensure all wrappers follow the same conventions (#76)

    • Refactor SeriesBy and ExtractTiles wrappers (#310)

    • Refactor SeriesByLead wrapper (#671, #76)

    • Add the pull request approval process steps to the Contributor’s Guide (#429)

    • Remove jlogger and postmsg (#470)

    • Add unit tests for set_met_config_X functions in CommandBuilder (#682)

    • Define a common set of GitHub labels that apply to all of the METplus component repos (#690)

    • Transition from using Travis CI to GitHub Actions (#721)

    • Improve workflow formatting in Contributers Guide (#688)

    • Change INPUT_BASE to optional (#679)

    • Refactor TCStat and ExtractTiles wrappers to conform to current standards

    • Automate release date (#665)

    • Add documentation for input verification datasets (#662)

    • Add timing tests for Travis/Docker (#649)

    • Set up encrypted credentials in Travis to push to DockerHub (#634)

    • Add to User’s Guide: using environment variables in METplus configuration files (#594)

    • Cleanup version info (#651)

    • Fix Travis tests for pull requests from forks (#659)

    • Enhance the build_docker_images.sh script to support TravisCI updates (#636)

    • Reorganize use case tests so users can add new cases easily (#648)

    • Investigate how to add version selector to documentation (#653)

    • Docker push pull image repository (#639)

    • Tutorial Proofreading (#534)

    • Update METplus data container logic to pull tarballs from dtcenter.org instead of GitHub release assets (#613)

    • Convert Travis Docker files (automated builds) to use Dockerhub data volumes instead of tarballs (#597)

    • Migrate from travis-ci.org to travis-ci.com (#618)

    • Migrate Docker run commands to the METplus ci/jobs scripts/files (#607)

    • Add stage to Travis to update or create data volumes when new sample data is available (#633)

    • Docker data caching (#623)

    • Tutorial testing on supported platforms (#468)

    • Add additional Branch support to the Travis CI pipeline (#478)

    • Change $DOCKER_WORK_DIR from /metplus to /root to be consistent with METplus tutorial (#595)

    • Add all use_cases to automated tests (eg Travis) (#571)

    • Add support to run METplus tests against multiple version of Python (#483)

    • Enhanced testing to use Docker data volumes to supply truth data for output comparisons (#567)

    • Update manage externals for beta5 versions (#832)

    • Create a new METplus GitHub issue template for “New Use Case” (#726)

1.6. Future development plans

METplus Wrappers is an evolving application. New capabilities are planned in controlled, successive version releases that are synchronized with MET releases. Bug fixes and user-identified problems will be addressed as they are found and posted to the known issues section of the METplus Wrappers Users web page (https://dtcenter.org/community-code/model-evaluation-tools-met). Future METplus Wrappers development plans are based on several contributing factors, including the needs of both the operational and research community. Issues that are in the development queue detailed in the “Issues” section of the GitHub repository. Please send questions to met_help@ucar.edu.

1.7. Code support

Support for METplus Wrappers is provided through a MET-help e-mail address: met_help@ucar.edu. We will endeavor to respond to requests for help in a timely fashion. In addition, information about METplus Wrappers and tools that can be used with MET are provided on the MET Users web page (https://dtcenter.org/community-code/model-evaluation-tools-met).

We welcome comments and suggestions for improvements to METplus Wrappers, especially information regarding errors. Comments may be submitted using the MET Feedback form available on the MET website. In addition, comments on this document would be greatly appreciated. While we cannot promise to incorporate all suggested changes, we will certainly take all suggestions into consideration.

METplus Wrappers is a “living” set of wrappers and configuration files. Our goal is to continually enhance it and add to its capabilities. Because our time, resources, and talents can at times be limited, we welcome contributed code for future versions of METplus. These contributions may represent new use cases or new plotting functions. For more information on contributing code to METplus Wrappers, please contact met_help@ucar.edu.