 |
Valuable Gribmaster Information |
December 2006 - New: Download Gribmaster 5.0 Note: Gribmaster version 5.0 documentation will be available in January 2007.
I. Introduction
II. Change Summary for Gribmaster version 4.2
III. Installation instructions
IV. Using gribmaster
V. Scheduling Cronjobs for automatic data fetching
Gribmaster version 5.0: What is it?
Gribmaster is a perl utility that allows users to download and process operational and non-operational data sets from selected servers via ftp, http, or scp. The package comes preconfigured for files on the NOMADS, TOC, NCEP and STRC data servers although users can easily add additional sites to meet thier needs. Downloaded data sets may then be processed for use in may different software packages including NAWIPS and GrADS, shipped off to other systems or archived for future use.
Why should you care?
Due to operational bandwidth and software limitations, the NWS offices get a subset of all the model fields that are available on the data servers. Gribmaster allows offices to get model runs at full resolution along with diagnostic fields that are not currently available through the operational feed. You can use Gribmaster to get UKMET model runs, Ensemble forecasts, full resolution hourly 13km ruc data, 3hourly NAM, and GFS data out to 384 hours, RTMA fields. In addition, the 12km NAM tiles and 8km non-hydrostatic hybrid model (HiRres Window) data are available. Are you salivating yet? Remember, too much of a good thing is wonderful.
II. Change Summary for Gribmaster version 5.0
a. Grimbaster version 5.0 has been completely rewritten to accommodate the new datasets, filenames, and directory structures used on remote data servers. All that remains from from previous versions are a few conjunctions.
b. The ability to use secure (scp) has been included for moving files around locally.
c. Gribmaster now supports the conversion of files from GRIB 2 <-> GRIB 1.
d. The gribmaster package is preconfigured for downloading files from the NOMADS, TOC, STRC, and NCEP servers. You may easily add additional servers as necessary. You can request primary and fail-over servers for the data. Gribmaster will automatically fail-over if data are not available from the primary server.
e. Datasets can be converted into GEMPAK format for viewing in NAWIPS or GrADS-compatible format.
f. Lots of error checking has been added.
g. Gribmaster will not over-state profits since the intellectual proceeds achieved by running gribmaster are infinite.
III. Installation instructions
Notes:You may put the gribmaster package anywhere your heart desires; however, for the sake of this discussion we will assume the package is located under the /usr1 directory. If "usr1" is offensive to you in any way, feel free to substitute another directory.
a. If you have a previous version of gribmaster installed on your machine rename the directory to gribmaster.old for the time being. You can whack it later:
% mv /usr1/gribmaster /usr1/gribmaster.old
After the updated distribution is installed and and working, you may delete the gribmaster.old directory.
b. Move the gribmaster package to the /usr1 directory and unpack:
% mv gribmaster5.tgz /usr1
% cd /usr1
% tar -xzvf gribmaster5.tgzA gribmaster5 directory will be created.
c. Create a link from gribmaster to gribmaster5
% ln -s /usr1/gribmaster5 /usr1/gribmaster
d. You should edit the following line near the top of the gribmaster5/gribmaster file to reflect your (gribmaster) package location:
$Gmaster{HOME} = "/usr1/gribmaster"
Note: (Almost) All configuration for gribmaster is accomplished by editing the conf/gribmaster.conf and various conf/<dataset>_gribinfo.conf files. Although there are default settings in place, you are strongly encouraged to read and edit these files as necessary.
Also, most of the default settings may be overridden with command line arguments.
Running "% gribmaster --guide" will provide the user's guide.
Running "% gribmaster --help" will give a brief summary:
Optional flags [optional]
--dset DATASTET The moniker for the requested data set identified
by <data set>_gribinfo.confIn the absence of the --dset DATASET option, gribmaster will read each of the dataset files located in the "active" directory. If you plan on running gribmaster for real-time data downloading then it is recommended that you copy the data set configuration files from the "conf" directory to the "active" directory and then then edit them as necessary.
--help --[no]verbose Turns verbose mode on [off]
--date [YY]YYMMDD Date of the data file to access. Default is current date
--cycle HOUR Cycle hour of data set. See --guide for more arguments to --cycle
--length HOURS The period, in hours, that you wish to download and process
--nodelay Ignore the DELAY setting in <data set>_gribinfo.conf. You want your data now
--tiles LIST list of tile number separated by a comma (03,23,33,...) and without spaces
--members LIST list of ensemble members separated by a comma (mean,n1,p1,...) (not working yet)
--ftp [SERVER] Obtain files via ftp from the list of available servers or [SERVER]
--http [SERVER] Obtain files via http from the list of available servers or [SERVER]
--nfs [SYSTEM] Obtain files crom a local system via the copy (cp) or remore copy (rcp) commands
--noarchive Do not archive the files to the location specified in the <data set>_gribinfo.conf file
--noaltdir Do not move the files to the alternate location specified in the <data set>_gribinfo.conf file
--[no]gempak [Do not] process the incoming grib files to gempak format
--[no]grads [Do not] process the incoming grib files to GrADS format
--[no]convert Do [not] convert downloaded grib 1 (2) files to version 2 (1)
--force Force processing of all grib files after downloading
--query [DATASET] list information in the data set configuration file
--dslist list the available data sets.
--guide Print out much more semi-usefull information than can be provided here
a. Configuring the gribmaster/conf/gribmaster.conf file
Take a look at this file. Chances are that you will not need to edit anything.
b. Configuring the various grib information files
In the conf directory you will find a _gribinfo.conf file for selected model data set on the NCEP and TOC servers. You may add additional data sets by simply copying one of these files and making the appropriate changes to reflect the new set. Note that the <data set> portion of the <data set>_gribinfo.conf file will be used when executing gribmaster to identify the desired data, so name it wisely.
The file is relatively well-documented so the configuration should be straight forward. The exception may be in the CYCLES parameter simply due to the many different options available to the user, some of which I failed to document.
Feel free to send email if you have problems.
c. Running gribmaster
Most of the gribmaster options (flags) override the default values for a particular data set; however, there are a minimum number of arguments that you must pass to gribmaster. Here are a few examples:
1) % gribmaster --ftp --dset gfs
Will use all the default values in the gfs_gribinfo.conf file to download grib files from the most current (date and cycle) run and place them in the default directory on your local machine. Because there are no arguments passed to --ftp the program will seach ALL defined remote servers (SERVER-FTP) looking for the data.
2) % gribmaster --ftp NCEP --dset gfs
Just like (1) above except that gribmaster will only access the NCEP ftp server.
3) % gribmaster --ftp --gempak --dset gfs
Just like (1) above except that the downloaded grib files will be converted
to gempak format.4) % gribmaster --http --gempak --dset gfs
Just like (3) above except use the http servers identified in the gfs_gribinfo.conf file to access the data files.
3) % gribmaster --http --gempak --dset gfs
Just like (1) above except that the downloaded grib files will be converted
to gempak format.4) % gribmaster --nfs --gempak --dset gfs
Just like (3) above except that gribmaster will try to get the files from a NFS mounted partition.
5) % gribmaster --nfs /data/incoming --gempak --dset gfs
Just like (4) above except that gribmaster will try to get the files from a the /data/incoming directory
6) % gribmaster --ftp --cycle 00 --dset gfs
As in (1) above except that gribmaster will get the data from the 00Z cycle run from the current date (UTC on machine).
7) % gribmaster --ftp --cycle 00:24 --dset gfs
As in (6) above except that gribmaster will start with the 24 hour forecast from todays 00Z run and download through the FINLFH hour as defined in the gfs_gribinfo.conf file. Frequency of forecast grids are also defined in the configuration file. See the documentation in the _gribinfo.conf file for more information on the dark powers of the --cycle option.
8) % gribmaster --ftp --cycle 00:24:FINLFH:12 --dset gfs
As in (7) above except use the default final forecast hour (FINLFH) and override the default forecast grid file frequency with 12 hours. See the documentation in the _gribinfo.conf file for more information on the dark powers of the --cycle option.
9) % gribmaster --ftp --cycle 00:24 --length 48 --dset gfs
As in (7) above except override the default value for FINLFH and get 48 hours worth of forecast grids beginning with the 24 hour forecast. Thus you get the 24 to 72 hour forecast.
10) % gribmaster --query gfs
List out the default settings for the gfs data set.
11) % gribmaster --verbose --ftp NCEP --dset gfs
Just like (2) above except print out information about what is going on.
12) % gribmaster --ftp --cycle 00 --date 20040820 --dset gfs
Just like one of the examples above but get the 00Z cycle data from 20 August 2004.
V. Scheduling Cronjobs for automatic data fetching
Here, for example, is a complete set of crontab file entries that you would need in order to look at both cycles of output for a single station:
1 * * * * PATH/gribmaster --dset eta212 --ftp NCEP > PATH/logs/gribmaster.log 2>&1
Note that PATH must be replaced by the actual path to the scripts (i.e., /usr1/gribmaster).
Since gribmaster checks to see if the files it creates already exist, it is possible to run the script frequently without wasting much CPU time.