decoders54.README Prepared by Peggy Bruehl/NWS & Steve Chiswell/Unidata 10/96 ------------------------------------------------------------------ NAWIPS/GEMPAK 5.4 Decoder Programs NAWIPS/GEMPAK decoders have a new look beginning with version 5.4. The new NAWIPS/GEMPAK decoders are called "DC"-style decoders. They can be used interactively from the command line or can be run in conjunction with the LDM software from Unidata. Below, you will find a listing of all the DC-style decoders, including the standard NAWIPS decoders and some supplemental decoders provided by Steve Chiswell from the Unidata Program Center. The DC-style decoders replace the LDMBridge satogem, uatogem and sytogem decoders. All of the "DC"-style decoders described below include code that supports the AFOS data stream. I, however, have not tested them with AFOS data. Therefore, it is possible that you may see some incompatibilities with the AFOS data. This is particularly true of the Unidata supplemental decoders. If you do run into trouble, please contact Peggy Bruehl (peggy@comet.ucar.edu). Each DC-style decoder is run with the following syntax: dcprogram [options] output_file The available options are described below. Each individual decoder has it's own set of defaults for these options. For more information about these defaults, please see the on-line help file for each program (type "phelp dchrly" for example, at any GEMPAK prompt), or see the Decoder Appendix in the NAWIPS Users Manual. The output file name can be specified using a template. This allows better control of file name creation within the decoders, rather than relying on a name entirely created from the LDM. The template uses the date and time of the bulletin or report to replace the following characters: YY Year without the century MM Month number DD Day HH Hour NN Minute For example, YYMMDD_surface.gem Decoder Options: -v N Set the verbosity level to "N". As N goes up, more verbose logging occurs. The default for all decoders is "0". -c curtim "curtim" is the pseudo-current time. This is a complete GEMPAK format date/time (YYMMDD/HHMM) and is used as the system time when decoding archive data. The default value for this variable for all decoders is the system time. -b nhours "nhours" is the number of hours prior to the current time. Only reports within this time period are decoded into the data file. -d decoder_log "decoder_log" is the name and path of the decoder log file. -t time_out "time_out" is the time out value in seconds of the decoder. If no data is received within the specified time limit, the program will exit. The default time out value for all decoders is 600 seconds. -n The default behavior of the decoder is to store the raw text reports in the decoded data file. Use this flag to NOT store the raw data. -h Print the usage statement and exit -p prmfil Set the parameter packing table to "prmfil". See below for a description of the GEMPAK parameter packing tables. -s stntbl Set the station table to "stntbl". See below for a description of the GEMPAK station tables. -a iadstn Define the maximum number of additional stations (beyond those in "prmfil") that can be added to the data file as "iadstn". -m maxtim Set the maximum number of times in a single data file to "maxtim". The maximum times allowable in a single surface file in NAWIPS 5.4 is 300. ------------------------------------------------------------------ Running the NAWIPS/GEMPAK 5.4 DC-style decoders As stated previously, the DC-style decoders can be run interactively from the command line, or they can be run in conjunction with the LDM software. To run the programs interactively, you must provide the raw data to be decoded as standard input. (That means use the "<" redirect in symbol to pipe the raw data into the decoder). You must also use the -c option to set the current time. Using the DCHRLY decoder as an example: dchrly -c YYMMDD/HHNN -b 24 -d logfile YYMMDD_hrly.gem < input_file You can also configure the DC-style decoders to run automatically upon receipt of raw data from the LDM. To do this, you must configure the pattern action file "pqact.conf" that is included with the LDM software. There is a complete set of sample pattern actions for all the DC-style decoders included at the end of this document. ------------------------------------------------------------------ The Complete List of DC-style Decoders: NAWIPS 5.4 "DC" Decoders: ------------------------- o DCHRLY: Convert Surface SA & METAR observations (replaces satogem) o DCUAIR: Convert Radiosonde upper air data (replaces uatogem) o DCNMOS: Decode NGM MOS reports o DCAMOS: Decode AVN MOS reports o DCMMOS: Decode MRF MOS reports o DCFFG: Decode flash flood guidance bulletins Unidata NAWIPS 5.4 Supplemental Decoders: ----------------------------------------- These "DC"-style decoders are available as contributions from the contrib/ directory on the SAC FTP server in the files dcunidata.tar.Z and dcunidata.README. o DCSYNOP: Convert Synoptic Code data (replaces sytogem) o DCGRIB: Convert HRS feed GRIB or raw GRIM message data (Note, this decoder does not have an AFOS interface.) o DCNLDN: Convert NLDN lightning data o DCPROF: Convert BUFR format profiler data o DCWATCH: Decode severe thunderstorm WWUS40 bulletins o DCTROP: Decode Tropical storm location and forecast bulletins o DCSTORM: Decode SPC storm report WWUS60 bulletins Coming Soon: DCSHEF for decoding SHEF reports. ----------------------------------------------------------------------- GEMPAK Station Tables When creating GEMPAK data files for observational data, you must supply a station table containing a list of all the possible stations that will be reporting data. These station tables are kept in the directory $GEMTBL/stns: avnmos.stn msfstns.tbl snstns.tbl sfworld_upc.tbl ffgzon.stn ngmmos.stn snworld.tbl syworld.tbl metar.tbl sfstns.tbl systns.tbl mrfmos.stn sfstns_ncep.tbl systns_upc.tbl You can see that each data type has it's own station table. When using most NAWIPS/GEMPAK DC-style decoders, you must select the best station table for the data type you are decoding. Each decoder has a default station table. In NAWIPS 5.4, you can have up to 4700 stations in a station table. The station tables are text files and contain a list of station IDs, station numbers, station names, station state and country, station lats & lons (both in degrees*10), station elevations (m), and station priorities. The station priorities are not used at this time. The standard surface station table is $GEMTBL/stns/sfstns.tbl and here are a few entries from that table: HGT 690020 HUNTER LIGGET/TUSI CA US 3600 -12123 310 0 OAR 690070 FORT_ORD/FRITZSCHE& CA US 3668 -12177 41 0 NZJ 690140 EL TORO MCAS CA US 3367 -11773 117 0 NXP 690150 TWENTYNINE PALMS CA US 3430 -11617 626 0 NTK 690160 TUSTIN MCAS CA US 3370 -11783 17 0 L63 690170 IND SPRNG RANGE 63 NV US 3653 -11557 972 0 DYS 690190 DYESS AFB/ABILENE TX US 3243 -9985 545 0 NUW 690230 WHIDBEY ISLAND NAS WA US 4835 -12265 14 0 ... The format is column sensitive. ----------------------------------------------------------------------- GEMPAK Packing Files When creating GEMPAK data files for observational data, you must supply something called a "Packing Table" which defines the parameters (Temp, Pres, Wind Speed, etc.) that are stored in the data file. These packing tables are kept in the directory $GEMTBL/pack: avnmos.pack hrly.pack mrfmos.pack ngmmos.pack sfsyn.pack ffg.pack metar.pack msfc.pack sfship.pack snmerg.pack You can see there is one packing table for each kind of data. This is true, because normally each data type has it's own set of variables. When using most NAWIPS/GEMPAK DC-style decoders, you must select the best packing table for the data type you are decoding. Each decoder has a default packing table. The packing tables are text files and contain a list of parameter names, ranges, and resolutions to be stored in the associated file. The standard surface data packing file is hrly.tbl and looks like: !HRLY.PACK ! !This table is the packing table for DCHRLY, which decodes surface !airways & METAR reports. ! !*Log: ! !S. Maxwell 6/96 Renamed SF52.PACK to SAO.PACK !S. Schotz 8/96 Renamed SAO.PACK to HRLY.PACK ! PMSL 950. 1049. .1 ALTI 24. 34. .01 TMPF -100. 150. .1 DWPF -100. 150. .1 SKNT 0. 126. .1 DRCT 0. 360. 10. GUST 0. 250. 1. WNUM 0. 512000. 1. CHC1 0. 18009. 1. CHC2 0. 8009. 1. CHC3 0. 8009. 1. VSBY 0. 51. .1 P03D 0. 8999. 1. P03I 0. 12.99 .01 P06I 0. 12.99 .01 HSUN 0. 999. 1. CSYL 0. 9. 1. CSYM 10. 19. 1. CSYH 20. 29. 1. SNOW 0. 1000.0 1. WEQS 0. 100.0 .1 T06X -100. 150. .1 T12X -100. 150. .1 T24X -100. 150. 1. TMDX -100. 150. 1. T06N -100. 150. .1 T12N -100. 150. .1 T24N -100. 150. 1. TMDN -100. 150. 1. P24I 0. 40. .01 The format is column sensitive. The first column contains a four character parameter identifier. There is an advantage to choosing parameters that are defined in GEMPAK (see Appendix A in the NAWIPS Users Manual for a list). However, this is not required, and you can use any 4 letter character string to name a variable type. The next three columns are the minimum, maximum, and resolution of the data respectively. ----------------------------------------------------------------------- Using DC-style decoders with the LDM The DC decoders are typically controlled by the LDM software from Unidata. The decoders are configured in the pqact.conf pattern action file of the LDM. Here is an example pqact.conf file containing the actions for all the DC-style decoders listed above. PLEASE NOTE THE FOLLOWING: 1) These pattern action entries are referring to the Family of Services (FOS) data stream, and are using WMO headers. To use these actions with the AFOS data stream, you must convert the pattern to fit the AFOS headers. 2) There are TABS in the sample pattern actions below which are very very very important. Do not "cut-and-paste" these entries into your pqact.conf file. "cut-and-paste" does not preserve TABS. 3) These pattern actions use different directory structures for data and table files than most SOOs use. If you use any of these actions, please be sure to configure the directories according to your system set up. ################################################################ # # Sample GEMPAK decoder actions # # Patterns match data headers on the FOS DATA STREAM # # Most pattern actions use VERBOSE (-v) logging # ################################################################ # # Observational Data # # Surface obs and specials # # DCHRLY # DDS|IDS ^S[AP].* .... ([0-3][0-9]) PIPE /usr/local/ldm/decoders/dchrly -v 1 -b 9 -m 24 -d data/gempak/logs/dchrly.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/hrly.pack -s /home/gempak/NAWIPS-5.4/gempak5.4/tables/stns/sfmetar_sa.tbl data/gempak/surface/YYMMDD_sao.gem # # Upper air reports # # DCUAIR # DDS|IDS ^U[ABDEFGHIJKLMPQRSTXZ].... .... ([0-3][0-9]) PIPE /usr/local/ldm/decoders/dcuair -v 1 -b 12 -n -m 16 -d data/gempak/logs/dcuair.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/snmerg.pack -s /home/gempak/NAWIPS-5.4/gempak5.4/tables/stns/snworld.tbl data/gempak/upperair/YYMMDD_upa.gem # # # Synoptic surface reports # # DCSYNOP # DDS|IDS ^S([IMN][^VW].*|XUS2[0-3]) .... ([0-3][0-9]) PIPE /usr/local/ldm/decoders/dcsynop -a 2000 -v 1 -d data/gempak/logs/dcsynop_syn.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/sfsyn.pack -s /home/gempak/NAWIPS-5.4/gempak5.4/tables/stns/systns_upc.tbl data/gempak/surface/YYMMDD_syn.gem # # Buoy and Ship reports # # DCSYNOP # DDS|IDS ^S([INS]V.*|MV[^NS].*) .... ([0-3][0-9]) PIPE /usr/local/ldm/decoders/dcsynop -v 1 -d data/gempak/logs/dcsynop_sb.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/sfship.pack data/gempak/surface/YYMMDD_sb.gem # ################################################################ # # Model Output Statistics (MOS) Data # # NGM MOS Data # # DCNMOS # DDPLUS ^FO(US14|AK1[34]) .... ([0-3][0-9])([0-2][0-9]) PIPE /usr/local/ldm/decoders/dcnmos -v 2 -d data/gempak/logs/dcnmos.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/ngmmos.pack -s /home/gempak/NAWIPS-5.4/gempak5.4/tables/stns/ngmmos.stn data/gempak/mos/YYMMDDHH_nmos.gem # # MRF MOS # # DCMMOS # DDPLUS ^(FOX[CESW].*|FOAK20|FECN21) .... ([0-3][0-9])([0-2][0-9]) PIPE /usr/local/ldm/decoders/dcmmos -v 1 -d data/gempak/logs/dcmmos.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/mrfmos.pack -s /home/gempak/NAWIPS-5.4/gempak5.4/tables/stns/mrfmos.stn data/gempak/mos/YYMMDDHH_mmos.gem # # AVN MOS # # DCAMOS # DDPLUS ^(FEX[CESW].*|FEAK20|FECN20) .... ([0-3][0-9])([0-2][0-9]) PIPE /usr/local/ldm/decoders/dcamos -v 1 -d data/gempak/logs/dcamos.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/avnmos.pack -s /home/gempak/NAWIPS-5.4/gempak5.4/tables/stns/avnmos.stn data/gempak/mos/YYMMDDHH_amos.gem # # BUFR profiler data # # DCPROF # HRS ^IUPT.* .... ([0-3][0-9])([0-2][0-9]) PIPE /usr/local/ldm/decoders/dcprof -d data/gempak/logs/dcprof.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/profiler.pack -s /home/gempak/NAWIPS-5.4/gempak5.4/tables/stns/profiler.stn /home/gempak/NAWIPS-5.4/gempak5.4/tables/grid/tableb /home/gempak/NAWIPS-5.4/gempak5.4/tables/grid/tabled data/gempak/profiler/YYMMDD_pro.gem # # Flash flood guidance # # DCFFG # DDPLUS (^RGU[EMWS]..)|(^RGAK1[012])|(^FOAK4[38])|(^FOUS4[1-4]) PIPE /usr/local/ldm/decoders/dcffg -v 1 -d data/gempak/logs/dcffg.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/ffg.pack -s /home/gempak/NAWIPS-5.4/gempak5.4/tables/stns/ffgzon.stn data/gempak/storm/ffg/YYMMDD_ffg.gem # ################################################################ # # GEMPAK decoder for NLDN Lightning data stream # # DCNLDN # # Decode NLDN data into a GEMPAK ship file (named YYMMDDHH.gem) # NLDN ^(9[3-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9]) PIPE /usr/local/ldm/decoders/dcnldn -m 10000 -s minute10 -d data/gempak/logs/dcnldn.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/nldn.pack data/gempak/nldn/YYMMDDHH.gem # ################################################################ # # Miscellaneous Other Data # # SPC storm reports # # DCSTORM # DDPLUS ^WWUS60 KMKC ([0-3][0-9])([0-2][0-9]) PIPE -close /usr/local/ldm/decoders/dcstorm -m 2000 -d data/gempak/logs/dcstorm.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/sels.pack data/gempak/storm/sels/YYMMDD_sels.gem # # Watch box coordinates # # DCWATCH # DDPLUS ^WWUS40 KMKC ([0-3][0-9])([0-2][0-9]) PIPE /usr/local/ldm/decoders/dcwatch -v 1 -t 30 -d data/gempak/logs/dcwatch.log -s /home/gempak/NAWIPS-5.4/gempak5.4/tables/stns/sfworld.tbl -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/watch.pack data/gempak/storm/watches/watches_YYYY.gem # # Hurricane/tropical storm positions and forecasts # # DCTROP # DDPLUS ^WTPZ4.* .... ([0-3][0-9])([0-2][0-9]) PIPE -close /usr/local/ldm/decoders/dctrop -v 1 -d data/gempak/logs/dctrop.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/tropic.pack data/gempak/storm/tropic/epacific/YYYY_@@.gem DDPLUS ^WTPN3.* .... ([0-3][0-9])([0-2][0-9]) PIPE -close /usr/local/ldm/decoders/dctrop -v 1 -d data/gempak/logs/dctrop.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/tropic.pack data/gempak/storm/tropic/wpacific/YYYY_@@.gem DDPLUS ^WTNT4.* .... ([0-3][0-9])([0-2][0-9]) PIPE -close /usr/local/ldm/decoders/dctrop -v 1 -d data/gempak/logs/dctrop.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/tropic.pack data/gempak/storm/tropic/atlantic/YYYY_@@.gem DDPLUS ^WTPA4.* .... ([0-3][0-9])([0-2][0-9]) PIPE -close /usr/local/ldm/decoders/dctrop -v 1 -d data/gempak/logs/dctrop.log -p /home/gempak/NAWIPS-5.4/gempak5.4/tables/pack/tropic.pack data/gempak/storm/tropic/cpacific/YYYY_@@.gem # ################################################################ # # Gridded data Section # # NMC GRIB products # NGM output for continental US # Decode into into yymmddhh_ngm.gem # # ONLY ONE SAMPLE ENTRY INCLUDED. NORMALLY, each model ingested # must have it's own pattern action file. # HRS ^Y..... KWB. ([0-3][0-9])([0-2][0-9]).*/mETA PIPE /usr/local/ldm/decoders/dcgrib -v -d data/gempak/logs/dcgrib.log -g /home/gempak/NAWIPS-5.4/gempak5.4/tables PACK data/gempak/hds/YYMMDDHH_ngm.gem # ------------------------------------------------------------------