Point2Grid: Calculate Practically Perfect Probabilities

model_applications/ short_range/ Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.conf

Scientific Objective

To use storm reports as observations to calculate Practically Perfect probabilities.

Datasets

Relevant information about the datasets that would be beneficial include:

  • Observation dataset: Local Storm Reports

METplus Components

This use case runs ASCII2NC to get the storm reports in netcdf format, runs Point2Grid to get those netcdf observations onto a grid, runs RegridDataPlane to use that gridded data as a mask to calculate probabilities

METplus Workflow

The following tools are used for each run time:

ASCII2NC > Point2Grid > RegridDataPlane

This example runs on a single time/file at a time. Each storm report is assumed to have no more than 24 hours of data inside

Run times:

2020-02-05

METplus Configuration

METplus first loads all of the configuration files found in parm/metplus_config, then it loads any configuration files passed to METplus via the command line with the -c option, i.e. -c parm/use_cases/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.conf

[config]

# Documentation for this use case can be found at
# https://metplus.readthedocs.io/en/latest/generated/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.html

# For additional information, please see the METplus Users Guide.
# https://metplus.readthedocs.io/en/latest/Users_Guide

###
# Processes to run
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#process-list
###

PROCESS_LIST = ASCII2NC, Point2Grid, RegridDataPlane 


###
# Time Info
# LOOP_BY options are INIT, VALID, RETRO, and REALTIME
# If set to INIT or RETRO:
#   INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set
# If set to VALID or REALTIME:
#   VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set
# LEAD_SEQ is the list of forecast leads to process
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#timing-control
###

LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d%H
INIT_BEG = 2020020500
INIT_END = 2020020500
INIT_INCREMENT = 24H

LEAD_SEQ = 12H


###
# File I/O
# https://metplus.readthedocs.io/en/latest/Users_Guide/systemconfiguration.html#directory-and-filename-template-info
###

CONFIG_DIR = {PARM_BASE}/use_cases/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect


# ASCII2NC

ASCII2NC_INPUT_DIR = {INPUT_BASE}/model_applications/short_range/practically_perfect
ASCII2NC_INPUT_TEMPLATE = "{CONFIG_DIR}/read_ascii_storm.py {ASCII2NC_INPUT_DIR}/200205_rpts_filtered.csv"

ASCII2NC_OUTPUT_TEMPLATE = {OUTPUT_BASE}/model_applications/short_range/practically_perfect/StormReps.{init?fmt=%Y%m%d%H}.nc


# Point2Grid

POINT2GRID_INPUT_DIR = {OUTPUT_BASE}/model_applications/short_range/practically_perfect
POINT2GRID_INPUT_TEMPLATE = {OUTPUT_BASE}/model_applications/short_range/practically_perfect/StormReps.{init?fmt=%Y%m%d%H}.nc

POINT2GRID_OUTPUT_DIR = {OUTPUT_BASE}/model_applications/short_range/practically_perfect


# RegridDataPlane

OBS_REGRID_DATA_PLANE_INPUT_DIR = {POINT2GRID_OUTPUT_DIR}
OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE = StormReps_211.{init?fmt=%Y%m%d%H}.nc

OBS_REGRID_DATA_PLANE_OUTPUT_DIR = {POINT2GRID_OUTPUT_DIR}
OBS_REGRID_DATA_PLANE_OUTPUT_TEMPLATE = StormReps_211_Probs.{init?fmt=%Y%m%d}.nc

POINT2GRID_OUTPUT_TEMPLATE = {OUTPUT_BASE}/model_applications/short_range/practically_perfect/StormReps_211.{init?fmt=%Y%m%d%H}.nc


###
# ASCII2NC Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#ascii2nc
###

ASCII2NC_WINDOW_BEGIN = 0
ASCII2NC_WINDOW_END = 0

ASCII2NC_INPUT_FORMAT = python
ASCII2NC_TIME_SUMMARY_FLAG = False
ASCII2NC_TIME_SUMMARY_RAW_DATA = False
ASCII2NC_TIME_SUMMARY_BEG = 000000
ASCII2NC_TIME_SUMMARY_END = 235959
ASCII2NC_TIME_SUMMARY_STEP = 300
ASCII2NC_TIME_SUMMARY_WIDTH = 600
ASCII2NC_TIME_SUMMARY_GRIB_CODES = 11, 204, 211
ASCII2NC_TIME_SUMMARY_VAR_NAMES =
ASCII2NC_TIME_SUMMARY_TYPES = min, max, range, mean, stdev, median, p80
ASCII2NC_TIME_SUMMARY_VALID_FREQ = 0
ASCII2NC_TIME_SUMMARY_VALID_THRESH = 0.0


###
# Point2Grid Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#point2grid
###

POINT2GRID_REGRID_TO_GRID = G211

POINT2GRID_INPUT_FIELD =Fscale
POINT2GRID_INPUT_LEVEL =

POINT2GRID_ADP =

POINT2GRID_REGRID_METHOD = MAX

POINT2GRID_GAUSSIAN_DX = 81.271
POINT2GRID_GAUSSIAN_RADIUS = 120

POINT2GRID_PROB_CAT_THRESH =

POINT2GRID_VLD_THRESH =


###
# RegridDataPlane Settings
# https://metplus.readthedocs.io/en/latest/Users_Guide/wrappers.html#regriddataplane
###

OBS_REGRID_DATA_PLANE_RUN = True

REGRID_DATA_PLANE_ONCE_PER_FIELD = False

OBS_REGRID_DATA_PLANE_VAR1_INPUT_FIELD_NAME = Fscale_mask
OBS_REGRID_DATA_PLANE_VAR1_INPUT_LEVEL = "(*,*)"
OBS_REGRID_DATA_PLANE_VAR1_OUTPUT_FIELD_NAME = PP_probs

REGRID_DATA_PLANE_VERIF_GRID = G211

REGRID_DATA_PLANE_METHOD = MAXGAUSS

REGRID_DATA_PLANE_WIDTH = 1

REGRID_DATA_PLANE_GAUSSIAN_DX = 81.271
REGRID_DATA_PLANE_GAUSSIAN_RADIUS = 120

MET Configuration

METplus sets environment variables based on the values in the METplus configuration file. These variables are referenced in the MET configuration file. YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS! If there is a setting in the MET configuration file that is not controlled by an environment variable, you can add additional environment variables to be set only within the METplus environment using the [user_env_vars] section of the METplus configuration files. See the ‘User Defined Config’ section on the ‘System Configuration’ page of the METplus User’s Guide for more information.

////////////////////////////////////////////////////////////////////////////////
//
// Default ascii2nc configuration file
//
////////////////////////////////////////////////////////////////////////////////

//
// The parameters listed below are used to summarize the ASCII data read in
//

//
// Time periods for the summarization
// obs_var (string array) is added and works like grib_code (int array)
// when the obs name is given instead of grib_code
//
${METPLUS_TIME_SUMMARY_DICT}

//
// Mapping of input little_r report types to output message types
//
message_type_map = [
   { key = "FM-12 SYNOP";  val = "ADPSFC"; },
   { key = "FM-13 SHIP";   val = "SFCSHP"; },
   { key = "FM-15 METAR";  val = "ADPSFC"; },
   { key = "FM-18 BUOY";   val = "SFCSHP"; },
   { key = "FM-281 QSCAT"; val = "ASCATW"; },
   { key = "FM-32 PILOT";  val = "ADPUPA"; },
   { key = "FM-35 TEMP";   val = "ADPUPA"; },
   { key = "FM-88 SATOB";  val = "SATWND"; },
   { key = "FM-97 ACARS";  val = "AIRCFT"; }
];

//
// Indicate a version number for the contents of this configuration file.
// The value should generally not be modified.
//
//version = "V10.0";

tmp_dir = "${MET_TMP_DIR}";

${METPLUS_MET_CONFIG_OVERRIDES}

See the following files for more information about the environment variables set in this configuration file.

parm/use_cases/met_tool_wrapper/Point2Grid/Point2Grid.py parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane.py

Python Embedding

This use case uses a Python embedding script to read input data

parm/use_cases/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect/read_ascii_storm.py


import pandas as pd
import os
import sys

print(f'Python Script: {sys.argv[0]}')

# input file specified on the command line
# load the data into the numpy array

if len(sys.argv) < 2:
    script_name = os.path.basename(sys.argv[0])
    print(f"ERROR: {script_name} -> Must specify exactly one input file.")
    sys.exit(1)

# Read the input file as the first argument
input_file = os.path.expandvars(sys.argv[1])
print(f'Input File: {input_file}')

if not os.path.exists(input_file):
    print("ERROR: Could not find input file")
    sys.exit(2)

# Read and format the input 11-column observations
COLUMN_NAMES = (
    "Message_Type",     # (1)  string
    "Station_ID",       # (2)  string
    "Valid_Time",       # (3)  string (YYYYMMDD_HHMMSS)
    "Lat",              # (4)  numeric (Deg North)
    "Lon",              # (5)  numeric (Deg East)
    "Elevation",        # (6)  numeric (msl)
    "Var_Name",         # (7)  string (or GRIB_Code)
    "Level",            # (8)  numeric
    "Height",           # (9)  numeric (msl or agl)
    "QC_String",        # (10) string
    "Observation_Value" # (11) numeric
)

# Create a blank dataframe based on the 11 column standard
point_frame = pd.DataFrame(columns=COLUMN_NAMES,dtype='str')

#Read in the Storm report, 8 columns not matching the 11 column standard
temp_data = pd.read_csv(input_file,names=['Time', 'Fscale', 'Location', 'County','Stat','Lat', 'Lon', 'Comment'], dtype=str ,skiprows=1)

#Strip out any rows in the middle that are actually header rows
#Allows for concatenating storm reports together
temp_data = temp_data[temp_data["Time"] != "Time"]

#Change some columns to floats and ints
temp_data[["Lat","Lon"]] = temp_data[["Lat","Lon"]].apply(pd.to_numeric)

#Assign approprite columns to point_frame leaving missing as empty strings
point_frame["Lat"] = temp_data["Lat"]
point_frame["Lon"] = temp_data["Lon"]
#point_frame["Station_ID"] = temp_data["County"]
point_frame["Station_ID"] = "NA"
point_frame["Var_Name"] = "Fscale"
point_frame["Message_Type"] = "StormReport"

#Assign 0.0 values to numeric point_frame columns that we don't have in the csv file
point_frame["Elevation"] = 0.0
point_frame["Level"] = 0.0
point_frame["Height"] = 0.0

#Change Comments into a "QC" string Tornado=1, Hail=2, Wind=3, Other=4
point_frame["QC_String"] = "4"
mask = temp_data["Comment"].str.contains('TORNADO')
point_frame.loc[mask,"QC_String"] = "1"
mask = temp_data["Comment"].str.contains('HAIL')
point_frame.loc[mask,"QC_String"] = "2"
mask = temp_data["Comment"].str.contains('WIND')
point_frame.loc[mask,"QC_String"] = "3"

#Time is HHMM in the csv file so we need to use a piece of the filename and
#this value to create a valid date string
file_without_path = os.path.basename(input_file)
year_month_day = "20"+file_without_path[0:6]
point_frame["Valid_Time"] = year_month_day+"_"+temp_data["Time"]+"00"

#Currently we are only interested in the fact that we have a report at that locaton
#and not its actual value so all values are 1.0
point_frame["Observation_Value"] = 1.0

#Ascii2nc wants the final values in a list
point_data = point_frame.values.tolist()

print("Data Length:\t" + repr(len(point_data)))
print("Data Type:\t" + repr(type(point_data)))

########################################################################

Running METplus

This use case can be run two ways:

  1. Passing in EnsembleStat_fcstHRRRE_obsHRRRE_Sfc_MultiField.conf then a user-specific system configuration file:

    run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.conf -c /path/to/user_system.conf
    
  2. Modifying the configurations in parm/metplus_config, then passing in EnsembleStat_fcstHRRRE_obsHRRRE_Sfc_MultiField.conf:

    run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/short_range/Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.conf
    

The former method is recommended. Whether you add them to a user-specific configuration file or modify the metplus_config files, the following variables must be set correctly:

  • INPUT_BASE - Path to directory where sample data tarballs are unpacked (See Datasets section to obtain tarballs). This is not required to run METplus, but it is required to run the examples in parm/use_cases

  • OUTPUT_BASE - Path where METplus output will be written. This must be in a location where you have write permissions

  • MET_INSTALL_DIR - Path to location where MET is installed locally

Example User Configuration File:

[dir]
INPUT_BASE = /path/to/sample/input/data
OUTPUT_BASE = /path/to/output/dir
MET_INSTALL_DIR = /path/to/met-X.Y

NOTE: All of these items must be found under the [dir] section.

Expected Output

A successful run will output the following both to the screen and to the logfile:

INFO: METplus has successfully finished running.

Refer to the value set for OUTPUT_BASE to find where the output data was generated. Output for this use case will be found in model_applications/short_range/practically_perfect/ (relative to OUTPUT_BASE) and will contain the following files:

  • StormReps_211_Probs.20200205.nc

Keywords

Note

  • ASCII2NCToolUseCase

  • Point2GridToolUseCase

  • RegridDataPlaneToolUseCase

  • RegriddingInToolUseCase

  • NetCDFFileUseCase

  • PythonEmbeddingFileUseCase

  • ShortRangeAppUseCase

  • NCAROrgUseCase

  • ProbabilityGenerationUseCase

  • MaskingFeatureUseCase

  • HMTOrgUseCase

  • HWTOrgUseCase

Navigate to the METplus Quick Search for Use Cases page to discover other similar use cases.

sphinx_gallery_thumbnail_path = ‘_static/short_range-Point2Grid_obsLSR_ObsOnly_PracticallyPerfect.png’

Total running time of the script: (0 minutes 0.000 seconds)

Gallery generated by Sphinx-Gallery