Note
Go to the end to download the full example code
StatAnalysis: IODAv2
model_applications/data_assimilation/StatAnalysis_fcstGFS_HofX_obsIODAv2_PyEmbed.conf
Scientific Objective
This use case demonstrates the Stat-Analysis tool and ingestion of HofX NetCDF files that have been output from the Joint Effort for Data assimilation Integration (JEDI) data assimilation system. JEDI uses IODA version 2 formatted files, which are NetCDF files with certain requirements of variables and naming conventions. These files hold observations to be assimilated into forecasts, in this case taken from the JEDI software test data, which contained a small number of Global observation-forecast pairs derived from the hofx application.
UFO is a component of HofX, which maps the background forecast to observation space to form O minus B pairs. The HofX application of JEDI takes the input IODAv2 files and adds an additional variable which is the forecast value as interpolated to the observation location. These HofX files are used as input to form Matched Pair (MPR) formatted lists via Python embedding. In this case, Stat-Analysis then performs an aggregate_stat job and outputs statistics in an ascii file.
This use case adopts the IODAv2 formatted NetCDF files, which replace the previous variable formatting scheme to make use of NetCDF groups.
Datasets
METplus Components
This use case utilizes the METplus StatAnalysis wrapper to search for files that are valid for the given case and generate a command to run the MET tool stat_analysis.
METplus Workflow
StatAnalysis is the only tool called in this example. It processes the following run times:
METplus Configuration
METplus first loads all of the configuration files found in parm/metplus_config, then it loads any configuration files passed to METplus via the command line with the -c option, i.e. -c parm/use_cases/model_applications/data_assimilation/StatAnalysis_fcstGFS_HofX_obsIODAv2_PyEmbed.conf
[config]
# Documentation for this use case can be found at
# https://metplus.readthedocs.io/en/latest/generated/model_applications/data_assimilation/StatAnalysis_fcstGFS_HofX_obsIODAv2_PyEmbed.html
# For additional information, please see the METplus Users Guide.
# https://metplus.readthedocs.io/en/latest/Users_Guide
###
# Processes to run
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list
###
PROCESS_LIST = StatAnalysis
###
# Time Info
# LOOP_BY options are INIT, VALID, RETRO, and REALTIME
# If set to INIT or RETRO:
# INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set
# If set to VALID or REALTIME:
# VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set
# LEAD_SEQ is the list of forecast leads to process
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control
###
LOOP_BY = VALID
VALID_TIME_FMT = %Y%m%d%H
VALID_BEG=2018041500
VALID_END=2018041500
VALID_INCREMENT = 12H
LEAD_SEQ = 0
###
# File I/O
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#directory-and-filename-template-info
###
MODEL1_STAT_ANALYSIS_LOOKIN_DIR = python {PARM_BASE}/use_cases/model_applications/data_assimilation/StatAnalysis_fcstGFS_HofX_obsIODAv2_PyEmbed/read_iodav2_mpr.py {INPUT_BASE}/model_applications/data_assimilation/StatAnalysis_fcstGFS_HofX_obsIODAv2_PyEmbed/sample_hofx_output_sondes.nc4
STAT_ANALYSIS_OUTPUT_DIR = {OUTPUT_BASE}/StatAnalysis_IODAv2
STAT_ANALYSIS_OUTPUT_TEMPLATE = job.out
MODEL1_STAT_ANALYSIS_DUMP_ROW_TEMPLATE = dump.out
###
# StatAnalysis Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#statanalysis
###
MODEL1 = NA
MODEL1_OBTYPE = NA
STAT_ANALYSIS_JOB_NAME = aggregate_stat
STAT_ANALYSIS_JOB_ARGS = -out_line_type CNT -dump_row [dump_row_file] -line_type MPR -by FCST_VAR
MODEL_LIST =
DESC_LIST =
FCST_LEAD_LIST =
OBS_LEAD_LIST =
FCST_VALID_HOUR_LIST =
FCST_INIT_HOUR_LIST =
OBS_VALID_HOUR_LIST =
OBS_INIT_HOUR_LIST =
FCST_VAR_LIST =
OBS_VAR_LIST =
FCST_UNITS_LIST =
OBS_UNITS_LIST =
FCST_LEVEL_LIST =
OBS_LEVEL_LIST =
VX_MASK_LIST =
INTERP_MTHD_LIST =
INTERP_PNTS_LIST =
FCST_THRESH_LIST =
OBS_THRESH_LIST =
COV_THRESH_LIST =
ALPHA_LIST =
LINE_TYPE_LIST =
GROUP_LIST_ITEMS =
LOOP_LIST_ITEMS = MODEL_LIST
MET Configuration
METplus sets environment variables based on user settings in the METplus configuration file. See How METplus controls MET config file settings for more details.
YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!
If there is a setting in the MET configuration file that is currently not supported by METplus you’d like to control, please refer to: Overriding Unsupported MET config file settings
Note
See the StatAnalysis MET Configuration section of the User’s Guide for more information on the environment variables used in the file below:
////////////////////////////////////////////////////////////////////////////////
//
// STAT-Analysis configuration file.
//
// For additional information, see the MET_BASE/config/README file.
//
////////////////////////////////////////////////////////////////////////////////
//
// Filtering input STAT lines by the contents of each column
//
${METPLUS_MODEL}
${METPLUS_DESC}
${METPLUS_FCST_LEAD}
${METPLUS_OBS_LEAD}
${METPLUS_FCST_VALID_BEG}
${METPLUS_FCST_VALID_END}
${METPLUS_FCST_VALID_HOUR}
${METPLUS_OBS_VALID_BEG}
${METPLUS_OBS_VALID_END}
${METPLUS_OBS_VALID_HOUR}
${METPLUS_FCST_INIT_BEG}
${METPLUS_FCST_INIT_END}
${METPLUS_FCST_INIT_HOUR}
${METPLUS_OBS_INIT_BEG}
${METPLUS_OBS_INIT_END}
${METPLUS_OBS_INIT_HOUR}
${METPLUS_FCST_VAR}
${METPLUS_OBS_VAR}
${METPLUS_FCST_UNITS}
${METPLUS_OBS_UNITS}
${METPLUS_FCST_LEVEL}
${METPLUS_OBS_LEVEL}
${METPLUS_OBTYPE}
${METPLUS_VX_MASK}
${METPLUS_INTERP_MTHD}
${METPLUS_INTERP_PNTS}
${METPLUS_FCST_THRESH}
${METPLUS_OBS_THRESH}
${METPLUS_COV_THRESH}
${METPLUS_ALPHA}
${METPLUS_LINE_TYPE}
column = [];
weight = [];
////////////////////////////////////////////////////////////////////////////////
//
// Array of STAT-Analysis jobs to be performed on the filtered data
//
${METPLUS_JOBS}
////////////////////////////////////////////////////////////////////////////////
//
// Confidence interval settings
//
out_alpha = 0.05;
boot = {
interval = PCTILE;
rep_prop = 1.0;
n_rep = 0;
rng = "mt19937";
seed = "";
}
////////////////////////////////////////////////////////////////////////////////
//
// WMO mean computation logic
//
wmo_sqrt_stats = [ "CNT:FSTDEV", "CNT:OSTDEV", "CNT:ESTDEV",
"CNT:RMSE", "CNT:RMSFA", "CNT:RMSOA",
"VCNT:FS_RMS", "VCNT:OS_RMS", "VCNT:RMSVE",
"VCNT:FSTDEV", "VCNT:OSTDEV" ];
wmo_fisher_stats = [ "CNT:PR_CORR", "CNT:SP_CORR",
"CNT:KT_CORR", "CNT:ANOM_CORR" ];
////////////////////////////////////////////////////////////////////////////////
//hss_ec_value =
${METPLUS_HSS_EC_VALUE}
rank_corr_flag = FALSE;
vif_flag = FALSE;
tmp_dir = "${MET_TMP_DIR}";
//version = "V10.0";
${METPLUS_MET_CONFIG_OVERRIDES}
Python Embedding
This use case uses a Python embedding script to read input data
parm/use_cases/model_applications/data_assimilation/StatAnalysis_fcstGFS_HofX_obsIODAv2_PyEmbed/read_iodav2_mpr.py
from __future__ import print_function
import pandas as pd
import os
from glob import glob
import sys
import xarray as xr
import datetime as dt
########################################################################
print('Python Script:\t', sys.argv[0])
# Input is .nc or .nc4 file
if len(sys.argv) == 2:
# Read the input file as the first argument
input_path = os.path.expandvars(sys.argv[1])
try:
print("Input File:\t" + repr(input_path))
# Read all the needed groups
ioda_data = xr.open_dataset(input_path, group = 'MetaData')
ioda_hofx_data = xr.open_dataset(input_path, group = 'hofx')
hofx_vars = list(ioda_hofx_data.keys())
# use dataframes
ioda_df = ioda_data.to_dataframe()
ioda_data.close()
for var_name in hofx_vars:
ioda_df[var_name + '@hofx'] = ioda_hofx_data[var_name]
# Add columns for needed attributes, for each variable present for hofx
for attribute in ['ObsValue', 'ObsType', 'EffectiveQC']:
ioda_attr_data = xr.open_dataset(input_path, group = attribute)
for var_name in hofx_vars:
ioda_df[var_name + '@' + attribute] = ioda_attr_data[var_name]
ioda_attr_data.close()
ioda_hofx_data.close()
nlocs = len(ioda_df.index)
print('Number of locations in set: ' + str(nlocs))
# Decode strings
time = list(ioda_df['datetime'])
for i in range(0,nlocs):
temp = dt.datetime.strptime(time[i], '%Y-%m-%dT%H:%M:%SZ')
time[i] = temp.strftime('%Y%m%d_%H%M%S')
ioda_df['datetime'] = time
#set up MPR data
mpr_data = []
for var_name in hofx_vars:
# Set up the needed columns
ioda_df_var = ioda_df[['datetime','station_id',var_name+'@ObsType',
'latitude','longitude','air_pressure',
var_name+'@hofx',var_name+'@ObsValue',
var_name+'@EffectiveQC']]
# Cute down to locations with valid ObsValues
ioda_df_var = ioda_df_var[abs(ioda_df_var[var_name+'@ObsValue']) < 1e6]
nlocs = len(ioda_df_var.index)
print(var_name+' has '+str(nlocs)+' valid obs.')
# Add additional columns
ioda_df_var['lead'] = '000000'
ioda_df_var['MPR'] = 'MPR'
ioda_df_var['nobs'] = nlocs
ioda_df_var['index'] = range(0,nlocs)
ioda_df_var['varname'] = var_name
ioda_df_var['na'] = 'NA'
# Arrange columns in MPR format
cols = ['na','na','lead','datetime','datetime','lead','datetime',
'datetime','varname','na','lead','varname','na','na',
var_name+'@ObsType','na','na','lead','na','na','na','na','MPR',
'nobs','index','station_id','latitude','longitude',
'air_pressure','na',var_name+'@hofx',var_name+'@ObsValue',
var_name+'@EffectiveQC','na','na']
ioda_df_var = ioda_df_var[cols]
# Into a list and all to strings
mpr_data = mpr_data + [list( map(str,i) ) for i in ioda_df_var.values.tolist() ]
print("Total Length:\t" + repr(len(mpr_data)))
except NameError:
print("Can't find the input file.")
print("HofX variables in this file:\t" + repr(hofx_vars))
else:
print("ERROR: read_iodav2_mpr.py -> Must specify input file.\n")
sys.exit(1)
########################################################################
Running METplus
It is recommended to run this use case by:
Passing in StatAnalysis_fcstGFS_HofX_obsIODAv2_PyEmbed.conf then a user-specific system configuration file:
run_metplus.py -c /path/to/StatAnalysis_fcstGFS_HofX_obsIODAv2_PyEmbed.conf -c /path/to/user_system.conf
The following METplus configuration variables must be set correctly to run this example.:
INPUT_BASE - Path to directory where sample data tarballs are unpacked (See Datasets section to obtain tarballs).
OUTPUT_BASE - Path where METplus output will be written. This must be in a location where you have write permissions
MET_INSTALL_DIR - Path to location where MET is installed locally
Example User Configuration File:
[dir]
INPUT_BASE = /path/to/sample/input/data
OUTPUT_BASE = /path/to/output/dir
MET_INSTALL_DIR = /path/to/met-X.Y
NOTE: All of these items must be found under the [dir] section.
Expected Output
A successful run will output the following both to the screen and to the logfile:
INFO: METplus has successfully finished running.
Refer to the value set for OUTPUT_BASE to find where the output data was generated. Output for this use case will be found in StatAnalysis_IODAv2 (relative to OUTPUT_BASE) and will contain the following file:
dump.out
Keywords
Note
StatAnalysisToolUseCase
PythonEmbeddingFileUseCase
DataAssimilationUseCase
IODA2NCToolUseCase
Navigate to the METplus Quick Search for Use Cases page to discover other similar use cases.
sphinx_gallery_thumbnail_path = ‘_static/data_assimilation-StatAnalysis_fcstGFS_HofX_obsIODAv2_PyEmbed.png’
Total running time of the script: (0 minutes 0.000 seconds)