Note

Click here to download the full example code

5.2.2.1. Grid-Stat and MODE: Sea Ice Validation

Runs bot GridStat and MODE tools to compare two ice concentration analyses and generate statistics and diagnostics. (IMS:NCEPAnalysis:Grib2)

Scientific Objective

Run Grid-Stat and MODE to compare the National Ice Center (NIC) Interactive Multisensor Snow and Ice Mapping System (IMS) and the National Centers for Environmental Prediction (NCEP) sea ice analysis. This is a validation and diagnostics use case because it is limited to a comparison between IMS analysis to NCEP analysis.

Datasets

Both IMS and NCEP sea ice analyses are observation datasets. For the purposes of MET, IMS is referred to as “forecast” and NCEP is referred to as “observation”.

  • Forecast dataset: IMS Sea Ice Concentration

    • Variable of interest: ICEC; ICEC is a binary field where “1” means a sea ice concentration of >=0.40 and “0” means a sea ice concentration of <0.40.

    • Level: Z0 (surface)

    • Dates: 20190201 - 20190228

    • Valid time: 22 UTC

    • Format: Grib2

    • Projection: 4-km Polar Stereographic

  • Observation dataset: NCEP Sea Ice Concentration

    • Variable of interest: ICEC; ICEC is the sea ice concentration with values from 0.0 - 1.0. Values >1.0 && <=1.28 indicate flagged data to be included and should be set to ==1.0 when running MET. Values <1.28 should be ignored as that indicates an invalid observation.

    • Level: Z0 (surface)

    • Dates: 20190201 - 20190228

    • Valid time: 00 UTC

    • Format: Grib2

    • Projection: 12.7-km Polar Stereographic

  • Data source: Received from Robert Grumbine at EMC. IMS data is originally from the NIC. NCEP data is originally from NCEP.

  • Location: IMS: https://www.natice.noaa.gov/ims/index.html; IMS - (https://polar.ncep.noaa.gov/seaice/Analyses.shtml)

METplus Components

This use case runs the MET GridStat and MODE tools.

METplus Workflow

The workflow processes the data by valid time, meaning that each tool will be run for each time before moving onto the next valid time. The GridStat tool is called first followed by the MODE tool. It processes analysis times from 2019-02-01 to 2019-02-05. The valid times for each analysis are different from one another (please see ‘Dataset’ section for more information).

METplus Configuration

METplus first loads all of the configuration files found in parm/metplus_config. Then, it loads any configuration files passed to METplus by the command line with the -c option.

# IMS Ice Concentration (fcst) vs. NCEP Ice Concentration (obs)
# IMS and NCEP are both observation analyses. For the purpose of running MET, IMS is referred to as
# the forecast and NCEP as the obs.
# Written by Lindsay Blank, NCAR. January 2020
####################################################################################################
[config]
# Loop by analysis time
LOOP_BY = VALID

# Format of VALID_BEG and VALID_END
VALID_TIME_FMT = %Y%m%d

# L: Available dates are 20190201 - 20190228
# Start time for METplus run
VALID_BEG=20190201

# End time for METplus run
VALID_END=20190205

# Increment between METplus runs in seconds. Must be >= 60
VALID_INCREMENT=86400

# Options are times, processes
# times = run all items in the PROCESS_LIST for a single initialization
# time, then repeat until all times have been evaluated.
# processes = run each item in the PROCESS_LIST for all times
# specified, then repeat for the next item in the PROCESS_LIST.
LOOP_ORDER = times

# List of applications to run
PROCESS_LIST = GridStat, Mode

###################################################################################################
# GridStat Configurations
###################################################################################################
# List of variables to compare
# "THRESH" refers to "cat_thresh"
FCST_VAR1_NAME = ICEC
FCST_VAR1_LEVELS = Z0
FCST_VAR1_THRESH = ==1.0 

OBS_VAR1_NAME = ICEC
OBS_VAR1_LEVELS = Z0
OBS_VAR1_THRESH = >=0.40
OBS_VAR1_OPTIONS = censor_thresh = [ >1.00 && <=1.28, >1.28 ]; censor_val    = [ 1.00 , -9999 ];

####################################################################################################
# MODE Configurations
####################################################################################################
# regridding domain for MODE
MODE_REGRID_TO_GRID = OBS

# Turn on quilting
MODE_QUILT = TRUE

# Convolution radius list
MODE_CONV_RADIUS = 10, 20, 30, 40, 50

# Convolution threshold list
# L: IMS is a binary field where a value of "1" is equivalent to >=0.40 sea ice concentration.
FCST_MODE_CONV_THRESH = ==1.00
OBS_MODE_CONV_THRESH = >=0.40

# Merge flag: options are NONE, THRESH, ENGINE, or BOTH
MODE_MERGE_FLAG = NONE

# Merge threshold list
MODE_MERGE_THRESH = >=1.25

####################################################################################################

# Description of data to be processed
# used in output file path
MODEL = IMS
OBTYPE = NCEP

# regridding domain for GridStat
GRID_STAT_REGRID_TO_GRID = OBS

# Location of grid_stat MET config file
GRID_STAT_CONFIG_FILE = {CONFIG_DIR}/GridStatConfig_sea_ice

# prefix to add to GridStat output filenames
GRID_STAT_OUTPUT_PREFIX = {MODEL}_{CURRENT_FCST_NAME}_vs_{OBTYPE}_{CURRENT_OBS_NAME}_{CURRENT_FCST_LEVEL}

# Location of mode MET config file
MODE_CONFIG_FILE = {CONFIG_DIR}/MODEConfig_sea_ice

# prefix to add to MODE output filenames
MODE_OUTPUT_PREFIX = {MODEL}_{CURRENT_FCST_NAME}_vs_{OBTYPE}_{CURRENT_OBS_NAME}

# IMS Options:
# Set to true if forecast data is probabilistic
FCST_IS_PROB = false

# NCEP Options:

[dir]
# location of configuration files used by MET applications
CONFIG_DIR={PARM_BASE}/use_cases/model_applications/cryosphere

# input and output data directories for each application in PROCESS_LIST
OBS_GRID_STAT_INPUT_DIR = {INPUT_BASE}/model_applications/cryosphere/sea_ice/NCEP_data
FCST_GRID_STAT_INPUT_DIR = {INPUT_BASE}/model_applications/cryosphere/sea_ice/IMS_data

OBS_MODE_INPUT_DIR = {INPUT_BASE}/model_applications/cryosphere/sea_ice/NCEP_data
FCST_MODE_INPUT_DIR = {INPUT_BASE}/model_applications/cryosphere/sea_ice/IMS_data

GRID_STAT_OUTPUT_DIR = {OUTPUT_BASE}/model_applications/cryosphere/sea_ice/GridStat
MODE_OUTPUT_DIR = {OUTPUT_BASE}/model_applications/cryosphere/sea_ice/MODE

[filename_templates]
# format of filenames

# IMS
FCST_GRID_STAT_INPUT_TEMPLATE = imssnow96.{valid?fmt=%Y%m%d}.grb.grib2
FCST_MODE_INPUT_TEMPLATE = imssnow96.{valid?fmt=%Y%m%d}.grb.grib2

# NCEP
OBS_GRID_STAT_INPUT_TEMPLATE = seaice.t00z.north12psg.grib2.{valid?fmt=%Y%m%d}
OBS_MODE_INPUT_TEMPLATE = seaice.t00z.north12psg.grib2.{valid?fmt=%Y%m%d}

GRID_STAT_VERIFICATION_MASK_TEMPLATE = {INPUT_BASE}/model_applications/cryosphere/sea_ice/seaice_nland127.nc

MODE_VERIFICATION_MASK_TEMPLATE = {INPUT_BASE}/model_applications/cryosphere/sea_ice/seaice_nland127.nc

GRID_STAT_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d}/grid_stat
MODE_OUTPUT_TEMPLATE = {valid?fmt=%Y%m%d}/mode

MET Configuration

METplus sets environment variables based on the values in the METplus configuration file. These variables are referenced in the MET configuration file. YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS! If there is a setting in the MET configuration file that is not controlled by an environment variable, you can add additional environment variables to be set only within the METplus environment using the [user_env_vars] section of the METplus configuration files. See the ‘User Defined Config’ section on the ‘System Configuration’ page of the METplus User’s Guide for more information.

GridStatConfig_sea_ice

////////////////////////////////////////////////////////////////////////////////
//
// Grid-Stat configuration file.
//
// For additional information, see the MET_BASE/config/README file.
//
////////////////////////////////////////////////////////////////////////////////

//
// Output model name to be written
//
model = "${MODEL}";

//
// Output description to be written
// May be set separately in each "obs.field" entry
//
desc = "NA";

//
// Output observation type to be written
//
obtype = "${OBTYPE}";

////////////////////////////////////////////////////////////////////////////////

//
// Verification grid
//
regrid = {
   to_grid    = ${REGRID_TO_GRID};
   method     = NEAREST;
   width      = 1;
   vld_thresh = 0.5;
   shape      = SQUARE;
}

////////////////////////////////////////////////////////////////////////////////

//
// May be set separately in each "field" entry
//
cnt_thresh       = [ NA ];
cnt_logic        = UNION;
wind_thresh      = [ NA ];
wind_logic       = UNION;
eclv_points      = 0.05;
nc_pairs_var_name = "";
nc_pairs_var_suffix = "";
rank_corr_flag   = FALSE;

//
// Forecast and observation fields to be verified
//

fcst = {
   field = [ ${FCST_FIELD} ];
   };

obs = {
    field = [ ${OBS_FIELD} ];
      };
	

////////////////////////////////////////////////////////////////////////////////

//
// Climatology data
//
climo_mean = {

   file_name = [];
   field     = [];

   regrid = {
      method     = NEAREST;
      width      = 1;
      vld_thresh = 0.5;
      shape      = SQUARE;
   }

   time_interp_method = DW_MEAN;
   match_month        = TRUE;
   match_day          = FALSE;
   time_step          = 21600;
}

climo_stdev = climo_mean;
climo_stdev = {
   file_name = [];
}

climo_cdf_bins = 1;

////////////////////////////////////////////////////////////////////////////////

//
// Verification masking regions
// May be set separately in each "obs.field" entry
//
mask = {
   grid = [ "" ];
   poly = [ ${VERIF_MASK} ];
}

////////////////////////////////////////////////////////////////////////////////

//
// Confidence interval settings
// May be set separately in each "obs.field" entry
//
ci_alpha  = [ 0.05 ];

boot = {
   interval = PCTILE;
   rep_prop = 1.0;
   n_rep    = 0;
   rng      = "mt19937";
   seed     = "";
}

////////////////////////////////////////////////////////////////////////////////

//
// Data smoothing methods
// May be set separately in each "obs.field" entry
//
interp = {
   field      = BOTH;
   vld_thresh = 1.0;
   shape      = SQUARE;

   type = [
      {
         method = NEAREST;
         width  = 1;
      }
   ];
}

////////////////////////////////////////////////////////////////////////////////

//
// Neighborhood methods
// May be set separately in each "obs.field" entry
//
nbrhd = {
   field      = BOTH;
   vld_thresh = 1.0;
   shape      = SQUARE;
   width      = [ 3, 5, 7, 9 ];
   cov_thresh = [ >=0.5 ];
}

////////////////////////////////////////////////////////////////////////////////

//
// Fourier decomposition
// May be set separately in each "obs.field" entry
//
fourier = {
   wave_1d_beg = [];
   wave_1d_end = [];
}

////////////////////////////////////////////////////////////////////////////////

//
// Gradient statistics
// May be set separately in each "obs.field" entry
//
gradient = {
   dx = [ 1 ];
   dy = [ 1 ];
}

////////////////////////////////////////////////////////////////////////////////

//
// Statistical output types
// May be set separately in each "obs.field" entry
//
output_flag = {
   fho    = STAT;
   ctc    = STAT;
   cts    = STAT;
   mctc   = NONE;
   mcts   = NONE;
   cnt    = STAT;
   sl1l2  = STAT;
   sal1l2 = NONE;
   vl1l2  = NONE;
   val1l2 = NONE;
   vcnt   = NONE;
   pct    = STAT;
   pstd   = STAT;
   pjc    = NONE;
   prc    = NONE;
   eclv   = NONE;
   nbrctc = NONE;
   nbrcts = NONE;
   nbrcnt = STAT;
   grad   = NONE;
}

//
// NetCDF matched pairs output file
// May be set separately in each "obs.field" entry
//
nc_pairs_flag = {
   latlon     = TRUE;
   raw        = TRUE;
   diff       = TRUE;
   climo      = TRUE;
   weight     = FALSE;
   nbrhd      = TRUE;
   fourier    = FALSE;
   gradient   = FALSE;
   apply_mask = TRUE;
}

////////////////////////////////////////////////////////////////////////////////

grid_weight_flag = NONE;
tmp_dir          = "/tmp";
output_prefix    = "${OUTPUT_PREFIX}";
version          = "V9.0";

////////////////////////////////////////////////////////////////////////////////
See the following file for more information about the environment variables set in this configuration file:

parm/use_cases/met_tool_wrapper/GridStat/GridStat.py

MODEConfig_sea_ice

////////////////////////////////////////////////////////////////////////////////
//
// MODE configuration file.
//
// For additional information, see the MET_BASE/config/README file.
//
////////////////////////////////////////////////////////////////////////////////

//
// Output model name to be written
//
model = "${MODEL}";

//
// Output description to be written
//
desc = "NA";

//
// Output observation type to be written
//
obtype = "${OBTYPE}";

////////////////////////////////////////////////////////////////////////////////

//
// Verification grid
//
regrid = {
   to_grid    = ${REGRID_TO_GRID};
   method     = NEAREST;
   width      = 1;
   vld_thresh = 0.5;
   shape      = SQUARE;
}

////////////////////////////////////////////////////////////////////////////////

//
// Approximate grid resolution (km)
//
grid_res = 12.7;

////////////////////////////////////////////////////////////////////////////////

//
// Run all permutations of radius and threshold
//
quilt = ${QUILT};

//
// Forecast and observation fields to be verified
//
fcst = {

   field = ${FCST_FIELD}

   censor_thresh      = [];
   censor_val         = [];
   conv_radius        = [${FCST_CONV_RADIUS}]; //in grid squares
   conv_thresh        = [${FCST_CONV_THRESH}];
   vld_thresh         = 0.5;
   filter_attr_name   = [];
   filter_attr_thresh = [];
   merge_thresh       = [${FCST_MERGE_THRESH}];
   merge_flag         = ${FCST_MERGE_FLAG};
}

obs = {
   field = ${OBS_FIELD};
  
   censor_thresh	= [ >1.00 && <=1.28, >1.28 ];
   censor_val		= [ 1.00 , -9999 ];
   conv_radius		= [${OBS_CONV_RADIUS}]; //in grid squares
    conv_thresh		= [${OBS_CONV_THRESH}];
    vld_thresh		= 0.5;
    filter_attr_name	= [];
    filter_attr_thresh	= [];
    merge_thresh	= [${OBS_MERGE_THRESH}];
    merge_flag		= ${OBS_MERGE_FLAG};
}

////////////////////////////////////////////////////////////////////////////////

//
// Handle missing data
//
mask_missing_flag = BOTH;

//
// Match objects between the forecast and observation fields
//
match_flag = NO_MERGE;

//
// Maximum centroid distance for objects to be compared
//
max_centroid_dist = 800.0/grid_res;

////////////////////////////////////////////////////////////////////////////////

//
// Verification masking regions
//
mask = {
   grid      = "";
   grid_flag = NONE; // Apply to NONE, FCST, OBS, or BOTH
   poly      = ${VERIF_MASK};
   poly_flag = BOTH; // Apply to NONE, FCST, OBS, or BOTH
}

////////////////////////////////////////////////////////////////////////////////

//
// Fuzzy engine weights
//
weight = {
   centroid_dist    = 2.0;
   boundary_dist    = 4.0;
   convex_hull_dist = 0.0;
   angle_diff       = 1.0;
   aspect_diff      = 0.0;
   area_ratio       = 1.0;
   int_area_ratio   = 2.0;
   curvature_ratio  = 0.0;
   complexity_ratio = 0.0;
   inten_perc_ratio = 0.0;
   inten_perc_value = 50;
}

////////////////////////////////////////////////////////////////////////////////

//
// Fuzzy engine interest functions
//
interest_function = {

   centroid_dist = (
      (            0.0, 1.0 )
      (  60.0/grid_res, 1.0 )
      ( 600.0/grid_res, 0.0 )
   );

   boundary_dist = (
      (            0.0, 1.0 )
      ( 400.0/grid_res, 0.0 )
   );

   convex_hull_dist = (
      (            0.0, 1.0 )
      ( 400.0/grid_res, 0.0 )
   );

   angle_diff = (
      (  0.0, 1.0 )
      ( 30.0, 1.0 )
      ( 90.0, 0.0 )
   );

   aspect_diff = (
      (  0.00, 1.0 )
      (  0.10, 1.0 )
      (  0.75, 0.0 )
   );

   corner   = 0.8;
   ratio_if = (
      (    0.0, 0.0 )
      ( corner, 1.0 )
      (    1.0, 1.0 )
   );

   area_ratio = ratio_if;

   int_area_ratio = (
      ( 0.00, 0.00 )
      ( 0.10, 0.50 )
      ( 0.25, 1.00 )
      ( 1.00, 1.00 )
   );

   curvature_ratio = ratio_if;

   complexity_ratio = ratio_if;

   inten_perc_ratio = ratio_if;
}

////////////////////////////////////////////////////////////////////////////////

//
// Total interest threshold for determining matches
//
total_interest_thresh = 0.8;

//
// Interest threshold for printing output pair information
//
print_interest_thresh = 0.0;

////////////////////////////////////////////////////////////////////////////////

//
// Plotting information
//
met_data_dir = "MET_BASE";

fcst_raw_plot = {
   color_table      = "MET_BASE/colortables/met_default.ctable";
   plot_min         = 0.0;
   plot_max         = 0.0;
   colorbar_spacing = 1;
}

obs_raw_plot = {
   color_table      = "MET_BASE/colortables/met_default.ctable";
   plot_min         = 0.0;
   plot_max         = 0.0;
   colorbar_spacing = 1;
}

object_plot = {
   color_table      = "MET_BASE/colortables/mode_obj.ctable";
}

//
// Boolean for plotting on the region of valid data within the domain
//
plot_valid_flag = FALSE;

//
// Plot polyline edges using great circle arcs instead of straight lines
//
plot_gcarc_flag = FALSE;

////////////////////////////////////////////////////////////////////////////////

//
// NetCDF matched pairs, PostScript, and contingency table output files
//
ps_plot_flag    = TRUE;
nc_pairs_flag   = {
   latlon       = TRUE;
   raw          = TRUE;
   object_raw   = TRUE;
   object_id    = TRUE;
   cluster_id   = TRUE;
   polylines    = TRUE;
}
ct_stats_flag   = TRUE;

////////////////////////////////////////////////////////////////////////////////

shift_right = 0;   //  grid squares

////////////////////////////////////////////////////////////////////////////////

output_prefix    = "${OUTPUT_PREFIX}";
version        = "V9.0";

////////////////////////////////////////////////////////////////////////////////
See the following file for more information about the environment variables set in this configuration file:

parm/use_cases/met_tool_wrapper/MODE/MODE.py

Running METplus

The command to run this use case is:

master_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/cryosphere/GridStat_MODE_fcstIMS_obsNCEP_sea_ice.conf

Expected Output

A successful run of this use case will output the following to the screen and logfile:

INFO: METplus has successfully finished runing.

A successful run will have the following output files in the location defined by {OUTPUT_BASE}, which is located in the metplus_system.conf configuration file located in /path/to/METplus/parm/metplus_config. This list of files should be found for every time run through METplus. Using the output for 20190201 as and example.

GridStat output:

  • grid_stat_IMS_ICEC_vs_NCEP_ICEC_Z0_000000L_20190201_220000V_pairs.nc

  • grid_stat_IMS_ICEC_vs_NCEP_ICEC_Z0_000000L_20190201_220000V.stat

MODE output:

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R1_T1_cts.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R1_T1_obj.nc

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R1_T1_obj.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R1_T1.ps

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R2_T1_cts.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R2_T1_obj.nc

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R2_T1_obj.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R2_T1.ps

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R3_T1_cts.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R3_T1_obj.nc

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R3_T1_obj.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R3_T1.ps

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R4_T1_cts.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R4_T1_obj.nc

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R4_T1_obj.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R4_T1.ps

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R5_T1_cts.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R5_T1_obj.nc

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R5_T1_obj.txt

  • mode_IMS_ICEC_vs_NCEP_ICEC_000000L_20190201_220000V_000000A_R5_T1.ps