 |
Modsnd Users Guide |
Users Guide Table of Contents I. Change Summary for Modsnd 5.0 II. Installation Instructions III. Running get_bufr.pl IV. A few get_bufr.pl examples V. Summary of Modsnd-generated files VI. Scheduling Cronjobs for Automatic Data Fetching VII. Bufr File Parameter Summary
I. Change Summary for Modsnd version 5.0
a. The Modsnd package script has been completely rewritten to make it more efficient
and to provide functionality that was either missing or not working in earlier versions.
Enhancements, features, functionality, and bugs in version 5.0 include, but are not limited to:
i. Command line options are now more intuitive. ii. The ability to download BUFR files via http has been integrated into get_bufr.pl iii. More robust downloading options. You may now specify any or all methods of BUFR file download (ftp, http, nfs) in order to obtain those station files that you deserve. iv. Uploading of processed files to other servers may be accomplished via ftp, copy, secure copy (scp), and secure ftp (sftp). v. Users can now ftp BUFKIT files to as many remote locations as they like. This "friends and family" option is described in the modsnd.conf file. vii. More dynamic personal pronouns!
b. Support for the following data sets is included:
NAM BUFR data sets:
nam Operational NAM Model BUFR files namm The "off-hour" (06 and 18 UTC only) runs of the NAM.
GFS BUFR data sets:
gfs3 Operational GFS Model BUFR files (3-hourly)
RUC BUFR data sets:
ruc Hourly operational RUC Model BUFR files Note that although support for these data sets is included in Modsnd 5.0, the data may
not be available on the servers and are subject to change, being moved, or worse.
c. Lots of error checking has been added. As stated earlier more dynamic verbs too.
II. Installation instructions for version 5.0
NOTES:The home directory for version 5.0 and future releases of modsnd is anywhere you want, but
for the sake of these instructions I will assume you are installing the distribution in /usr1.a. If you have a previous version of modsnd installed on your machine rename the
directory to modsnd.old or better yet, just delete it.
b. Move the modsnd4 package to the /usr1 directory and unpack:% mv modsnd5.tgz /usr1
% cd /usr1
% tar -xvzf modsnd5.tgz
c. Create a link from modsnd to the newly created modsnd42 directory
% ln -s modsnd5 modsnd
d. You will need to edit 2 files before you download BUFR files, get_bufr.pl and
config/modsnd.confi. Change the path name at the top of modsnd/get_bufr.pl to reflect the location of
modsnd on your system.ii. Make any changes necessary to the modsnd/config/modsnd.conf file and make
sure you have read/write permission to the top level of your "metdat" directory.
e. All the model configuration files are located in config directory. All you need to do
to add/remove support for a BUFR data set is to create and edit an appropriately
named file. You may use one of the existing data set information files as a template.
a. USAGE:
When processing BUFR files, the typical get_bufr.pl usage will be:
% get_bufr.pl --dset <data set> --stations <station list> [ACQUISITION METHOD(S)] [other options]
Where "--dset <data set>" defines the BUFR data set you wish to process, "--stations <station list>"
is the list of BUFR stations, "[acquisition method]" is the method you wish you use to acquire the
BUFR files (ftp, http, and/or nfs), and "[other options]" are the other available options. All
the available options are described below
b. QUASI-MANDATORY OPTIONS:
--dset DATA SET The "--dset" flag specifies the BUFR data set you wish to acquire and process. The DATA SET must be one of the supported BUFR data sets and have a corresponding <data set>_bufrinfo.conf file in the conf directory. See the "--dslist" option below for listing out the available data sets. Example:
% get_bufr.pl --dset nam [other options]
--stations STATIONS The argument list for the "--stations" option is a list of BUFR stations, separated by a comma (,) and without spaces. Either a station number or ID may be used. See the "--stnlist" option below for more information on list out the available stations. Examples:
% get_bufr.pl --dset nam --stations KRDU,041001,KGSO,723170 (yes)
% get_bufr.pl --dset nam --stations KRDU, 041001, KGSO, 723170 (no)
% get_bufr.pl --dset nam --stations KRDU 041001 KGSO 723170 (no)
Notice that in the above examples the BUFR station ID KGSO is also identified by the station number (723170). The get_bufr.pl routine will eliminate any duplicate stations in the list.
c. ACQUISITION METHODS:
The current version of Modsnd supports the acquisition of BUFR files via FTP (--ftp), HTTP (--http), and NFS (cp and scp; --nfs). Multiple methods may be specified on the command line, in which case get_bufr.pl will use each method in the presented order.
USAGE:
--ftp|http|nfs [SERVER ID or <server:directory/filename>]:
DESCRIPTION
Passing the --ftp, --http, and/or --nfs flags will cause get_bufr.pl to search for BUFR files on a server using ftp, http, or nfs commands respectively. Not passing any arguments, E.g, just "--http|ftp|nfs", will instruct the script to use each server specified in the <dataset>_bufrinfo.conf file by SERVER-[FTP|HTTP|NFS]. If no servers for a method are defined in the _bufrinfo.conf file, then the next method in the command line list, if any, will be used. E.g.:
% get_bufr.pl --dset nam --nfs --http --ftp
i. FTP and HTTP (--ftp and --http)
An argument to --ftp|http may either be a SERVER ID that is used to identify a remote system or
string that specifies the IP/hostname of a server followed by the path to the file and naming convention.If a SERVER ID is passed then it must have a corresponding entry in the <dataset>_bufrinfo.conf and also be
defined in the modsnd.conf file. E.g.:% get_bufr.pl --dset nam --http STRC
Where STRC has an entry in the nam_bufrinfo.conf file as:
SERVER-HTTP = STRC:/data/bufr/YYYYMMDD/nam/bufr.tCCz/bufr.STNM.YYYYMMDDCC
And is also defined in the conf/modsnd.conf file as:
STRC = soostrc.comet.ucar.edu
It is also possible to specify a hostname and directory/filename string separated by a
colon(:) as an argument to --ftp|http|nfs. E.g.:% get_bufr.pl --dset nam --ftp soostrc.comet.ucar.edu:/data/bufr/YYYYMMDD/nam/bufr.tCCz/bufr.STNM.YYYYMMDDCC
or
% get_bufr.pl --dset nam --ftp 128.117.110.214:/data/bufr/YYYYMMDD/nam/bufr.tCCz/bufr.STNM.YYYYMMDDCCNote that in each of the above examples, the YYYY, MM, DD, FF, and CC will be replaced with the
appropriate values.
ii. NFSPassing --nfs will cause get_bufr.pl to search for BUFR files on a locally-accessible system using either
secure copy (scp) or copy (cp) commands. Not passing any arguments, i.e, just "--nfs", will instruct the
to use each machine listed in the appropriate <dataset>_bufrinfo.conf file by a SERVER-NFS identifier.An argument to --nfs may either be a SERVER ID that is used to identify a system. The SERVER ID must
have a corresponding entry in the _bufrinfo.conf and be defined in the wrf_prep.conf file. E.g.:% get_bufr.pl --dset nam --nfs system_a
Where SYSTEM_A has an entry in the nam_bufrinfo.conf file as:
SERVER-NFS = SYSTEM_A:/data/bufr/YYYYMMDD/nam/bufr.STNM.YYYYMMDDCC
And is also defined in the conf/modsnd.conf file as:
SYSTEM_A = systema.comet.ucar.edu
or
SYSTEM_A = user@systema.comet.ucar.edu
The SERVER-NFS entry does not have to include a server ID. You may include the hostname information
directly on the SERVER-NFS line. E.g.:SERVER-NFS = systema.comet.ucar.edu:/data/bufr/YYYYMMDD/nam/bufr.STNM.YYYYMMDDCC
or
SERVER-NFS = user@systema.comet.ucar.edu:/data/bufr/YYYYMMDD/nam/bufr.STNM.YYYYMMDDCC
In each of the examples above secure copy (scp) is used to access the requested files
on another system; however, if your files are locally available in a directory you don't
need to include the [user@]hostname information. E.g.:SERVER-NFS = /data/bufr/YYYYMMDD/nam/bufr.STNM.YYYYMMDDCC
or
SERVER-NFS = LOCAL:/data/bufr/YYYYMMDD/nam/bufr.STNM.YYYYMMDDCCIn which case the copy (cp) command will be used to access the files.
It is also possible to specify a hostname and directory/filename string separated by a
colon(:) as an argument to --nfs. E.g.:% get_bufr.pl --dset nam --nfs servera.comet.ucar.edu:/data/bufr/YYYYMMDD/nam/bufr.STNM.YYYYMMDDCC
or
% get_bufr.pl --dset nam --nfs user@servera.comet.ucar.edu:/data/bufr/YYYYMMDD/nam/bufr.STNM.YYYYMMDDCC
or
% get_bufr.pl --dset nam --nfs /data/bufr/YYYYMMDD/nam/bufr.STNM.YYYYMMDDCC
d. OTHER USEFUL AND NOT SO USEFUL USEFUL OPTIONS
--date [YY]YYMMDD Passing the "--date" option defines the date of the BUFR forecast files to use.
The argument to this option is a 4- or 2-digit year, 2-digit month (01 to 12),
and 2-digit day (01 to 36).Not passing the "--date" option will cause get_bufr.pl to use the current date on
the system.DO NOT use the --date option for real-time processing of BUFR files.
--cycle CYCLE Not passing the --cycle option will cause the script to use the cycle time of the
most recent model run from which data are available. In determining the cycle time
of the most recently available BUFR files, get_bufr.pl accounts for the amount of time
required to run the operational model and process the BUFR files for distribution.For example, if it takes NCEP 2 hours to run and process grib files then the script
will not attempt to obtain data from the 12Z run until after 14Z. The delay (DELAY)
and available cycles parameters for each data set are defined in each bufrinfo.conf
file.DO NOT use the --cycle option for real-time processing of BUFR files.
--previous Download and process BUFR files from the previous cycle of a model run rather than
the current one.--nodelay Passing the --nodelay option will turn off (set to 0 hours) the default DELAY
value defined in each _bufrinfo.conf file.--noarch Passing --noarch will turn off the archiving of BUFR, GEMPAK, and BUFKIT files as defined
by the BUFR_ARCHIVE, GEMPAK_ARCHIVE, and BUFKIT_ARCHIVE parameters respectively in the
<data set>_bufrinfo.conf file.--nobufkit Passing --nobufkit turns off processing of BUFR files into BUFKIT format. --noascii Passing --noascii turns off processing of BUFR files into text files. --prepend Passing --prepend will result in YYMMDDCC being added to the beginning of the newly
minted BUFKIT file. Due to popular demand a second file without the YYMMDDCC will
also be created.--forced Passing --forced will force get_bufr.pl to download the requested BUFR files regardless
of whether the files already exist on the local system. The default behavior is to not
attempt to download files that already exist locally. After the files have been
downloaded they are processed as expected.--forcep Passing --forced will force get_bufr.pl to process the requested BUFR files regardless
of whether the files already exist on the local system. The default behavior is to not
attempt to process files that exist locally.--[no]verbose Turn on/off the printing of interesting stuff to the screen. Overrides the setting in
modsnd.conf.
e. STILL MORE USEFUL OPTIONS
--stnlist DSET [str] List out bufr stations for the DSET data set. The [str] argument provides a matching
string that helps to cut down on the amount of output. For example. whereas% get_bufr.pl --stnlist nam
will dump out the entire list od 1000+ stations,
% get_bufr.pl --stnlist nam rdu
will print out only those station descriptions that match the "RDU" string:
% get_bufr.pl --stnlist nam rdu
Matches for rdu from the nam BUFR station list:
723060 35.87N 78.78W KRDU 10 RALEIGH-DURHAM NC 134 FOUS TAF 2-99
724386 40.41N 86.93W KLAF 10 LAFAYETTE/PURDUE_U IN 185 TAF 2-99
--dslist Passing --dslist to get_bufr.pl will print all the supported BUFR data sets with
a brief description.
IV. A few get_bufr.pl examples
1. % get_bufr.pl --ftp --dset nam --stations KRDU,KGSO,723140
The above get_bufr.pl call will download via ftp and process the most current operational NAM
BUFR files for Raleigh, Greensboro, and Charlotte, NC. You can mix station numbers
and IDs. Duplicate and non-existing stations will be ignored. If the time is 1600 UTC
when the above command is executed then the script will attempt to download data
from the 1200 UTC cycle run. Note that if the time is 1330 UTC then the script will
attempt to download 0600 UTC BUFR data since it is unlikely that the 1200 UTC data
are available. See the DELAY setting in the model configuration file for more information.2. % get_bufr.pl --dset nam --ftp --cycle 00 --stations KRDU,KGSO,723140
Same as (1.) above except download data from the 0000 UTC cycle run of the operational nam model.
3. % get_bufr.pl --dset nam --http STRC --ftp --date 20070311--cycle 00 --stations KRDU,KGSO,723140
Same as (2.) above except initially attempt to download the BUFR files from 11 March 2007
from the STRC http server in the nam_bufrinfo.conf file. If that should fail then proceed to
the list of FTP servers.4. % get_bufr.pl--dset nam --nfs --date 20070311--cycle 00 --stations KRDU,KGSO,723140
Same as (3.) above except obtain the files from the list of local server(s) defined in the the bufrinfo.conf file.
5. % get_bufr.pl--dset nam --nfs LOCAL:/data/nam/bufr.STNM.YYYYMMDDCC --date 20070311--cycle 00 --stations KRDU,KGSO,723140
Same as (4.) above except obtain the files from the of local server in the the modsnd.conf file.
V. Summary of Modsnd-generated files
The above commands will create the following files:
In $METDAT/bufr
bufr.STMN.YYYYMMDDCC - the raw BUFR file
In $METDAT/gempak
YYMMDDCC.<model type>_bufr.sfc -- a GEMPAK surface file
YYMMDDCC.<model type>_bufr.sfc_aux -- an auxiliary GEMPAK surface file
YYMMDDCC.<model type>_bufr.snd -- a GEMPAK sounding file GREAT FOR NSHARPIn $METDAT/bufkit
<model type>_STID.buf - BUFKIT file
YYMMDDCC.<model type>_STID.buf - BUFKIT file ( if the -prepend argument is passed)
VI. Scheduling Cronjobs for automatic data fetching
Unlike the previous Modsnd release, You may place your get_bufr.pl commands directly in a crontab file.
For example:
49 * * * * /usr1/modsnd/get_bufr.pl --dset nam --http --ftp --stations KRDU,KGSO > /tmp/get_bufr.log 2>&1
TABLE 1. Eta and RUC Sounding and Surface Parameter Names
PRES SND Pressure mb TMPC SND Temperature C DWPC SND Dew point C SPED SND Speed m/s DRCT SND Direction degrees OMEG SND Omega pa/s HGHT SND Height m DTCP SND Temp tndncy due to conv phase change K / day DTGP SND Temp tndncy due to grid-scale phase change K / day DTSW SND Temp tndncy due to short-wave radiation K / day DTLW SND Temp tndncy due to long-wave radiation K / day DTAR SND Temp tndncy due to all radiation K / day CFRL SND Cloud fraction in layer % CWTR SND Cloud water Kg / Kg TKEL SND Turbulent kinetic energy J / Kg IMXR SND Cloud ice mixing ratio Kg / Kg PMSL SFC Mean sea-level pressure mb PRES SFC Surface pressure mb TMPC SFC First level temperature C DWPC SFC First level dewpoint temperature C SPED SFC First level wind speed m/s DRCT SFC First level wind direction degrees SKTC SFC Earth surface temperature C SBTC SFC Bottom soil temperature C TMIN SFC Minimum lowest layer air temperature over 1 hr C TMAX SFC Maximum lowest layer air temperature over 1 hr C FXLH SFC Flux of latent heat (evaporation) W / m**2 FXLP SFC Potential flux of latent heat W / m**2 FXSH SFC Flux of sensible heat W / m**2 FXSS SFC Flux of sub-surface heat W / m**2 FXSN SFC Flux of snow phase change heat W / m**2 SWRD SFC Short-wave radiation downward flux W / m**2 SWRU SFC Short-wave radiation upward flux W / m**2 LWRD SFC Long-wave radiation downward flux W / m**2 LWRU SFC Long-wave radiation upward flux W / m**2 NDRF SFC Net downward radiative flux at the surface W / m**2 RNET SFC Net radiative flux at the surface W / m**2 FXTT SFC Total surface energy budget residual W / m**2 P01M SFC Total precip over 1 hr mm (kg/m**2) C01M SFC Convective precip over 1 hr mm (kg/m**2) S01M SFC Stable precip over 1 hr mm (kg/m**2) WTNS SFC Surface moisture availability unitless ratio SNFL SFC Accumulated snowfall over 1 hr mm (kg/m**2) SNRA SFC Snow mm (kg/m**2) SLMM SFC Soil moisture ratio % SWEM SFC Snow water equivalent mm (kg/m**2) N01M SFC Snow water equivalent mm (kg/m**2) R01M SFC Total snow melt over 1 hr mm (kg/m**2) BFGR SFC Storm sfc runoff over 1 hr mm (kg/m**2) SLLH SFC Baseflow-groundwater runoff over 1 hr mm (kg/m**2) SLMM SFC Surface evaporation over 1 hr mm (kg/m**2) SLLP SFC Potential surface evaporation over 1 hr mm (kg/m**2) SWBL SFC Total water budget residual mm (kg/m**2) LWRT SFC Long-wave radiation at top W / m**2 SWRT SFC Short-wave radiation at top W / m**2 WXTS SFC Categorical weather type snow 0,1 WXTP SFC Categorical weather type ice pellets 0,1 WXTZ SFC Categorical weather type freezing rain 0,1 WXTR SFC Categorical weather type rain 0,1 WSYM SFC Weather type symbol number LCLD SFC Low-cloud amount % MCLD SFC Middle-cloud amount % HCLD SFC High-cloud amount % CDBP SFC Cloud base pressure mb VSBK SFC Visibility km UWND SFC 10 m u component m/s VWND SFC 10 m v component m/s TH10 SFC 10 m potential temperature K Q10M SFC 10 m specific humidity g/kg HLCY SFC Estimated storm-relative helicity m*m/s*s USTM SFC Estimated storm motion u component m/s VSTM SFC Estimated storm motion v component m/s T2MS SFC 2 m temperature C TD2M SFC Estimated 2 m dewpoint temperature C Q2MS SFC 2 m specific humidity g/kg SRLM SFC Surface roughness length m SEXC SFC Surface exchange coefficient m/s VEGF SFC Green vegetation cover % CNPW SFC Canopy water mm (kg/m**2) SMCx SFC Layer x=1,4 volumetric soil moisture STCx SFC Layer x=1,4 soil temperature K SMSK SFC Land/water mask (0=land, 1=water) SELV SFC Surface elevation (model terrain) m SLAT SFC Station latitude degrees SLON SFC Station longitude degrees STNM SFC Station number STID SFC Station character ID