1. Overview
1.1. Purpose and organization of the User’s Guide
The goal of this User’s Guide is to equip users with the information needed to use the Model Evaluation Tools (MET) and its companion package METplus Wrappers. MET is a set of verification tools developed and supported to community via the Developmental Testbed Center (DTC) for use by the numerical weather prediction community. METplus Wrappers is a suite of Python wrappers and ancillary scripts to enhance the user’s ability to quickly set-up and run MET. Over the next few years, METplus Wrappers will become the authoritative repository for verification of the Unified Forecast System.
The METplus Wrappers User’s Guide is organized as follows. An overview of METplus Wrappers can be found below. Software Installation contains basic information about how to get started with METplus Wrappers - including system requirements, required software, and how to download METplus Wrappers. System Configuration provides information about configuring your environment and METplus Wrappers installation.
1.2. The Developmental Testbed Center (DTC)
METplus Wrappers has been developed, and will be maintained and enhanced, by the Developmental Testbed Center (DTC; http://www.dtcenter.org/ ). The main goal of the DTC is to serve as a bridge between operations and research and to facilitate the activities of these two important components of the numerical weather prediction (NWP) community. The DTC provides an environment that is functionally equivalent to the operational environment in which the research community can test model enhancements; the operational community benefits from DTC testing and evaluation of models before new models are implemented operationally. METplus Wrappers serves both the research and operational communities in this way - offering capabilities for researchers to test their own enhancements to models and providing a capability for the DTC to evaluate the strengths and weaknesses of advances in NWP prior to operational implementation.
METplus Wrappers will also be available to DTC visitors and the NOAA Unified Forecast System (UFS) and NCAR System for Integrated Modeling of the Atmosphere (SIMA) modeling communities for testing and evaluation of new model capabilities, applications in new environments, and so on. The METplus Wrappers release schedule is coincident with the MET release schedule and the METplus Wrappers major release number is six less than the MET major release number (e.g. MET 8.X is released with METplus Wrappers 2.X).
1.3. METplus Wrappers goals and design philosophy
METplus Wrappers is a Python scripting infrastructure for the MET tools. The primary goal of METplus Wrappers development is to provide MET users with a highly configurable and simple means to perform model verification using the MET tools. Prior to the availability of METplus Wrappers, users who had more complex verifications that required the use of more than one MET tool were faced with setting up multiple MET config files and creating some automation scripts to perform the verification. METplus Wrappers provides the user with the infrastructure to modularly create the necessary steps to perform such verifications.
METplus Wrappers has been designed to be modular and adaptable. This is accomplished through wrapping the MET tools with Python and the use of hierarchical configuration files to enable users to readily customize their verification environments. Wrappers can be run individually, or as a group of wrappers that represent a sequence of MET processes. New wrappers can readily be added to the METplus Wrappers package due to this modular design. Currently, METplus Wrappers can easily be applied by any user on their own computer platform that supports Python 3.6. We have deprecated support to Python 2.7.
The METplus Wrappers code and documentation is maintained by the DTC in Boulder, Colorado. METplus Wrappers is freely available to the modeling, verification, and operational communities, including universities, governments, the private sector, and operational modeling and prediction centers through a publicly accessible GitHub repository. Refer to Getting the METplus Wrappers source code for simple examples of obtaining METplus Wrappers.
1.4. METplus Wrappers Components
The major components of the METplus Wrappers package are METplus Python wrappers to the MET tools, MET configuration files and a hierarchy of METplus Wrappers configuration files. Some Python wrappers do not correspond to a particular MET tool, but wrap utilities to extend METplus functionality.
1.5. METplus Components Python Requirements
Name |
Version |
METplus Component |
Source |
Description |
Use Cases (only applicable for METplus wrappers component)(followed by python package name) |
---|---|---|---|---|---|
Python 3.6.3+ |
METplus wrappers, |
||||
Python 3.7 |
METplus wrappers |
Multi_Tool: Feature Relative by Lead using Multiple User-Defined Fields (Python 3.7) |
|||
cartopy |
>=0.18.0 |
METplus wrappers, |
Designed for |
TCGen: Genesis Density Function (GDF)
and Track Density Function (TDF)
(cartopy) |
|
cfgrib |
METplus wrappers |
map GRIB files |
Multi_Tool: Feature Relative by Lead using Multiple User-Defined Fields (cfgrib) |
||
cmocean |
METcalcpy, |
contains colormaps |
|||
dateutil |
>=2.8 |
METplus wrappers |
provides powerful |
Most |
|
eofs |
METplus wrappers, |
empirical orthogonal |
WeatherRegime Calculation:
RegridDataPlane,
PcpCombine,
and WeatherRegime python code
(eofs) |
||
h5py |
METplus wrappers |
Pythonic interface |
|||
imutils |
0.5.3 |
METcalcpy, |
A series of convenience |
||
imageio |
METcalcpy, |
provides an easy |
|||
lxml |
METcalcpy, |
a Pythonic binding for |
|||
matplotlib |
>=3.3.4 |
METplus wrappers, |
a comprehensive library |
UserScript:
Make OMI plot from
calculated MJO indices (obs only)
(matplotlib) |
|
metcalcpy |
METplus wrappers, |
a Python version of the |
UserScript:
Make a Hovmoeller plot
(metcalcpy) |
||
metplotpy |
METplus wrappers |
contains packages for |
UserScript:
Make a Hovmoeller plot
(metplotpy) |
||
metpy |
METplus wrappers |
a collection of tools |
Multi_Tool: Feature Relative by Lead using Multiple User-Defined Fields (metpy) |
||
nc-time-axis |
1.4 |
METplotpy |
extension to cftime |
||
netCDF4 |
>=1.5.4 |
METplus wrappers, |
a Python interface to |
For using MET Python embedding functionality in use cases |
|
numpy |
>=1.19.2 |
METplus wrappers, |
NumPy offers |
For using MET Python embedding functionality in use cases |
|
pandas |
>=1.0.5 |
METplus wrappers, |
a fast, powerful, |
For using MET Python embedding functionality in use cases |
|
pint |
>=0.18 |
METcalcpy |
Python package to define, |
||
plotly |
>=4.9.0 |
METcalcpy, |
makes interactive, |
||
psutil |
5.7.2 |
METcalcpy, |
Cross-platform lib for |
||
pygrib |
METplus wrappers |
for reading/writing |
Multi_Tool: Feature Relative by Lead using Multiple User-Defined Fields (pygrib) |
||
pylab |
METplus wrappers |
a convenience module |
WeatherRegime Calculation:
RegridDataPlane, PcpCombine,
and WeatherRegime python code
(pylab) |
||
pymysql |
METcalcpy, |
a pure-Python MySQL |
|||
pyproj |
2.3.1 |
METplus wrappers |
Python interface to PROJ |
GridStat: Python Embedding to read and process ice cover (pyproj) |
|
pyresample |
METplus wrappers |
for resampling geospatial |
GridStat:
Python Embedding to read and
process SST
(pyresample) |
||
pytest |
>=5.2.1 |
METcalcpy, |
a mature full-featured |
||
python-kaleido |
>=0.2.1 |
METcalcpy, |
provides a low-level |
||
pyyaml |
>=5.3.1 |
METcalcpy, |
a full-featured YAML |
||
scikit-image |
>=0.16.2 |
METcalcpy, |
a collection of |
||
scikit-learn |
0.23.2 |
METplus wrappers, |
Open Source library for |
GridStat:
Python Embedding to read and process SST
(scikit-learn) |
|
scipy |
>=1.5.1 |
METplus wrappers, |
wraps highly-optimized |
Blocking Calculation:
ERA RegridDataPlane,
PcpCombine, and
Blocking python code
(scipy) |
|
sklearn |
METplus wrappers |
Simple and efficient |
WeatherRegime Calculation:
RegridDataPlane, PcpCombine, and WeatherRegime python code
(sklearn) |
||
statsmodels |
>=0.11.1 |
METcalcpy, |
provides classes and |
||
xarray |
>=0.17.0 |
METplus wrappers, |
makes working with |
For using MET Python embedding functionality in use cases |
|
xesmf |
METplus wrappers |
NOTE: The xesmf package |
for regridding |
PlotDataPlane: Python Embedding of tripolar coordinate file (xesmf) |
|
yaml |
METcalcpy, |
to load, read, and write |
1.6. METplus Release Notes
Users can view the Stages of the METplus Release Cycle section of the Release Guide for descriptions of the development releases (including beta releases and release candidates), official releases, and bugfix releases for the METplus Components.
1.6.1. METplus Components Release Note Links
1.6.1.1. Release Notes - Latest Official Release
1.6.1.2. Release Notes - Development Release
1.6.2. METplus Wrappers Release Notes
When applicable, release notes are followed by the GitHub issue number which describes the bugfix, enhancement, or new feature: https://github.com/dtcenter/METplus/issues
1.6.2.1. METplus Version 4.1.4 Release Notes (2022-09-09)
Bugfixes:
Allow NA value for <TOOL-NAME>_CLIMO_[MEAN/STDEV]_HOUR_INTERVAL (METplus #1787)
1.6.2.2. METplus Version 4.1.3 Release Notes (2022-07-27)
Bugfixes:
Set Point2Grid -adp and -qc arguments properly (METplus #1623)
Reconcile setting of METPLOTPY_BASE for use cases (METplus #1713)
Allow NA value for <TOOL-NAME>_CLIMO_[MEAN/STDEV]_DAY_INTERVAL (METplus #1653)
Add missing brackets around list variable values for StatAnalysis wrapper (METplus #1641)
Phase Diagram use case writes all output into OUTPUT_BASE instead of INPUT_BASE (METplus #1706)
1.6.2.3. METplus Version 4.1.2 Release Notes (2022-05-16)
Bugfixes:
Add logic to catch error thrown when wrapper has been disabled and update documentation to include instructions to disable UserScript wrapper (METplus-Internal #33)
1.6.2.4. METplus Version 4.1.1 Release Notes (2022-04-21)
Bugfixes:
1.6.2.5. METplus Version 4.1.0 Release Notes (2022-03-14)
Enhancements:
General:
Add support for setting control members in EnsembleStat and GenEnsProd (#1236)
Create an Amazon AMI containing all METplus components (#506)
Modify wrappers that use wrapped MET config files to default to parm/met_config versions if unset (#931)
Add support for setting hss_ec_value in MET config files (#951)
Added support for setting a dictionary value for time_summary.width (#1252)
Properly handle list values that include square braces (#1212)
Update wrapped MET config files to reference MET_TMP_DIR in tmp value (#1101)
Add support for setting INIT_LIST and VALID_LIST for irregular time intervals (#1286)
Support setting the OMP_NUM_THREADS environment variable (#1320)
Add support for commonly changed MET config variables (#896)
Prevent wildcard character from being used in output file path (#1291)
Add support for setting file_type for fcst/obs for applications that process gridded data (#1165)
Enhance logic for setting mask.poly to allow MET list characters (square braces and semi-colon) (#966)
Add support for new climo_cdf.direct_prob flag (#1392)
Implement various enhancements to climatology settings (#1247)
Enhance logic to set climatology info for Python embedding (#944)
Updated logic for handling _CLIMO_MEAN_FIELD variables for specifying climatology fields (#1021)
Incorporate basic zonal and meridional means into METplus (#1230)
Add support for explicitly setting file list file paths in wrappers that support multiple input files (#1289)
Add support for setting -out argument in TCStat and StatAnalysis wrappers (#1102)
PointStat:
GridStat:
Add Grid-Stat configuration options for distance_map dictionary (#1089)
EnsembleStat:
GenEnsProd:
Add support for the normalize option to the Gen-Ens-Prod wrapper (#1445)
TCPairs:
TCGen:
Enhance TCGen wrapper to add support for new configurations (#1273)
SeriesAnalysis:
RegridDataPlane:
Add support for extra field options in RegridDataPlane wrapper (#924)
PCPCombine:
GenVxMask:
Update GenVxMask wrapper to require setting -type (#960)
UserScript:
Enhance UserScript to get a list of files that match the run times instead of using a wildcard (#1002)
ExtractTiles:
Enhance ExtractTiles using MTD input to properly match times (#1285)
TCMPRPlotter:
CyclonePlotter:
Use Cases:
Add stat_analysis to the Blocking and Weather Regime processing (#1001)
Modify user diagnostic feature relative use case to use MetPy Python package (#759)
Reorganize the Cryosphere and Marine and Coastal use case categories into one group (#1200)
Add harmonic pre-processing to the RMM use case (#1019)
New Wrappers:
New Use Cases:
MET Tool Wrapper:
Marine and Cryosphere:
GridStat_fcstRTOFS_obsOSTIA_iceCover (#834)
Satellite verification of sea surface temperature (GHRSST) against RTOFS output (#1004)
Satellite verification of sea surface salinity: SMOS vs RTOFS output (#1116)
Satellite verification of sea surface salinity: AVISO vs RTOFS output HYCOM climo (#1318)
Satellite verification of sea surface salinity: SMAP vs RTOFS output (#1216)
Medium Range:
Feature Relative using MTD output for feature centroid lat/lon (#641)
Precipitation:
Precipitation-type comparison across 3 models (#1408)
Seasonal to Subseasonal (S2S):
Tropical Cyclone and Extra Tropical Cyclone (tc_and_extra_tc):
Bugfixes:
Documentation:
Add list of METplus statistics to documentation (#1049)
Update documentation to reference GitHub Discussions instead of MET Help (#956)
Fix installation instructions in User’s Guide (#1067)
Add instructions to update old METplus configuration files that reference user-defined wrapped MET config files (#1147)
Internal:
Improve approach to obtain additional python packages needed for some use cases (#839)
Make updates to the Release Guide (#935)
Clean up GitHub wiki broken links and out-of-date information (#237)
Add option to override MET version used for automated tests (#936)
Transition Community and Developer Support to Github Discussions (#932)
Add documentation about the Release Guide and Verification Datasets Guide (#874)
Create guidance for memory-intensive use cases, introduce Python memory profiler (#1183)
Identify code throughout METplus components that are common utilities (#799)
Add definitions to the Release Guide for the stages of the release cycle (#934)
Document Continous Integration Functionality in the METplus Contributor’s Guide (#675)
Update Contributor’s Guide for new removing/adding data protocols (#1227)
Add recording of Python packages to Adding Use Cases documentation (#1374)
Remove public-facing access to outdated use case categories (Cryosphere, marine_and_coastal) (#1226)
1.6.2.6. METplus Version 4.0.0 Release Notes (2021-05-10)
Bugfixes:
Changed default values in wrapped MET config files to align with actual default values in MET config files (Reconcile Default Values)
Fix bug causing GridStat fatal error (#740)
Add support for comparing inputs using a mix of python embedding and non-embedding (#684)
Fix quick search links (#687)
Align the user guide with get_relativedelta() in time_util.py (#579)
Enhancements:
Rename master_metplus.py script to run_metplus.py (#794)
Update setting of environment variables for MET config files to add support for all to METPLUS_ vars (#768)
Add support for many commonly changed MET config variables (#779, #755, #621, #620)
Add support for a UserScript wrapper (#723)
Create use case subdirectories (#751)
Implement [INIT/VALID]EXCLUDE for time looping (#307)
Add files to allow installation of METplus wrappers as a Python package (beta) (#282)
Generate PDF of User’s Guide (#551)
Add support for 2 fields with same name and different levels in SeriesBy cases (#852)
Enhance PCPCombine wrapper to be able to process multiple fields in one command (#718)
Update TCStat config options and wrappers to filter data by excluding strings (#857)
Support METplus to run from a driver script (#569)
Refactor field info parsing to read once then substitute time info for each run time (#880)
Enhance Python embedding logic to allow multiple level values (#719)
Enhance Python embedding logic to allow multiple fcst and obs variable levels (#708)
Add support for a group of files covering multiple run times for a single analysis in GridDiag (#733)
Enhance ascii2nc python embedding script for TC dropsonde data (#734, #731)
Support additional configuration variables in EnsembleStat (#748)
Ensure backwards compatibility for MET config environment variables (#760)
Combine configuration file sections into single config section (#777)
Add support for skipping existing output files for all wrappers (#711)
Add support for multiple instance of the same tool in the process list (#670)
Add GFDL build support in build_components (#614)
Decouple PCPCombine, RegridDataPlane, and GridStat wrappers behavior (#602)
StatAnalysis run without filtering or config file (#625)
Enhance User Diagnostic Feature Relative use case to Run Multiple Diagnostics (#536)
Enhance PyEmbedIngest to run RegridDataPlane over Multiple Fields in One Call (#549)
Filename templates that have other arguments besides a filename for python embedding fails (#581)
Add more logging to tc_gen_wrapper (#576)
Prevent crash when improperly formatted filename template is used (#674)
New Wrappers:
PlotDataPlane
UserScript
METdbLoad
New Use Cases:
Air Quality and Comp: EnsembleStat_fcstICAP_obsMODIS_aod
Medium Range: UserScript_fcstGEFS_Difficulty_Index
Convection Allowing Models: MODE_fcstFV3_obsGOES_BrightnessTemp
Convection Allowing Models: MODE_fcstFV3_obsGOES_BrightnessTempObjs
Convection Allowing Models: GridStat_fcstFV3_obsGOES_BrightnessTempDmap
Data Assimilation: StatAnalysis_fcstHAFS_obsPrepBufr_JEDI_IODA_interface
Medium Range: SeriesAnalysis_fcstGFS_obsGFS_FeatureRelative_SeriesByLead_PyEmbed_Multiple_Diagnostics
Precipitation: EnsembleStat_fcstWOFS_obsWOFS
Seasonal to Subseasonal: TCGen_fcstGFSO_obsBDECKS_GDF_TDF
Seasonal to Subseasonal: UserScript_fcstGFS_obsERA_Blocking
Seasonal to Subseasonal: UserScript_obsERA_obsOnly_Blocking
Seasonal to Subseasonal: UserScript_obsERA_obsOnly_WeatherRegime
Seasonal to Subseasonal: UserScript_obsPrecip_obsOnly_Hovmoeller
Seasonal to Subseasonal: UserScript_obsPrecip_obsOnly_CrossSpectraPlot
TC and Extra TC: CyclonePlotter_fcstGFS_obsGFS_OPC
TC and Extra TC: UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF
TC and Extra TC: GridStat_fcstHAFS_obsTDR_NetCDF
Marine and Coastal: PlotDataPlane_obsHYCOM_coordTripolar
MET Tool Wrapper: METdbLoad/METdbLoad
MET Tool Wrapper: PlotDataPlane/PlotDataPlane_grib1
MET Tool Wrapper: PlotDataPlane/PlotDataPlane_netcdf
MET Tool Wrapper: PlotDataPlane/PlotDataPlane_python_embedding
MET Tool Wrapper: GridStat/GridStat_python_embedding
MET Tool Wrapper: PointStat/PointStat_python_embedding
MET Tool Wrapper: MODE/MODE_python_embedding
MET Tool Wrapper: PyEmbedIngest_multi_field_one_file
Internal:
Append semi-colon to end of _OPTIONS variables if not found (#707)
Ensure all wrappers follow the same conventions (#76)
Refactor SeriesBy and ExtractTiles wrappers (#310)
Add the pull request approval process steps to the Contributor’s Guide (#429)
Remove jlogger and postmsg (#470)
Add unit tests for set_met_config_X functions in CommandBuilder (#682)
Define a common set of GitHub labels that apply to all of the METplus component repos (#690)
Transition from using Travis CI to GitHub Actions (#721)
Improve workflow formatting in Contributers Guide (#688)
Change INPUT_BASE to optional (#679)
Refactor TCStat and ExtractTiles wrappers to conform to current standards
Automate release date (#665)
Add documentation for input verification datasets (#662)
Add timing tests for Travis/Docker (#649)
Set up encrypted credentials in Travis to push to DockerHub (#634)
Add to User’s Guide: using environment variables in METplus configuration files (#594)
Cleanup version info (#651)
Fix Travis tests for pull requests from forks (#659)
Enhance the build_docker_images.sh script to support TravisCI updates (#636)
Reorganize use case tests so users can add new cases easily (#648)
Investigate how to add version selector to documentation (#653)
Docker push pull image repository (#639)
Tutorial Proofreading (#534)
Update METplus data container logic to pull tarballs from dtcenter.org instead of GitHub release assets (#613)
Convert Travis Docker files (automated builds) to use Dockerhub data volumes instead of tarballs (#597)
Migrate from travis-ci.org to travis-ci.com (#618)
Migrate Docker run commands to the METplus ci/jobs scripts/files (#607)
Add stage to Travis to update or create data volumes when new sample data is available (#633)
Docker data caching (#623)
Tutorial testing on supported platforms (#468)
Add additional Branch support to the Travis CI pipeline (#478)
Change $DOCKER_WORK_DIR from /metplus to /root to be consistent with METplus tutorial (#595)
Add all use_cases to automated tests (eg Travis) (#571)
Add support to run METplus tests against multiple version of Python (#483)
Enhanced testing to use Docker data volumes to supply truth data for output comparisons (#567)
Update manage externals for beta5 versions (#832)
Create a new METplus GitHub issue template for “New Use Case” (#726)
1.7. Future development plans
METplus Wrappers is an evolving application. New capabilities are planned in controlled, successive version releases that are synchronized with MET releases. Software bugs and user-identified problems will be documented using GitHub issues and fixed either in the next bugfix or official release. Future METplus Wrappers development plans are based on several contributing factors, including the needs of both the operational and research community. Issues that are in the development queue detailed in the “Issues” section of the GitHub repository. Please create a post in the METplus GitHub Discussions Forum with any questions.
1.8. Code support
Support for METplus Wrappers is provided through the METplus GitHub Discussions Forum. We will endeavor to respond to requests for help in a timely fashion. In addition, information about METplus Wrappers and tools that can be used with MET are provided on the MET Users web page.
We welcome comments and suggestions for improvements to METplus Wrappers, especially information regarding errors. Comments may be submitted using the MET Feedback form available on the MET website. In addition, comments on this document would be greatly appreciated. While we cannot promise to incorporate all suggested changes, we will certainly take all suggestions into consideration.
METplus Wrappers is a “living” set of wrappers and configuration files. Our goal is to continually enhance it and add to its capabilities. Because our time, resources, and talents can at times be limited, we welcome contributed code for future versions of METplus. These contributions may represent new use cases or new plotting functions. For more information on contributing code to METplus Wrappers, please create a post in the METplus GitHub Discussions Forum.