Gridmaster FAQ
|
|
Questions |
|
|
|
| Q | I've noticed that there is a script called "runmetajobs" in gribmaster 2.0. What does it do? |
| A | Runmetajobs is a new script that is run from cronmaster. In the old gribmaster (version 1.X), gribmaster used to run the "makemeta" script. Unfortunately, when multiple copies of gribmaster were queued up in the process table, there was no guarantee which gribmaster would wake up first and process the metafiles. The result of this was metafiles with graphics out of order. Jim Cowie (NWS/COMET) implemented a solution to this problem. Instead of having gribmaster run the makemeta jobs, Jim wrote runmetajobs to do the metafile generation. After gribmaster has finished, cronmaster runs runmetajobs. Runmetajobs checks the file system for all the metajob.HHMM files created by gribmaster, and runs them in the correct time order. So now, even if multiple gribmaster scripts are queued up in your process table, the makemeta scripts are always run in the right order. The runmetajobs script uses a different log file than gribmaster does. So, in the log/ directory, you will now see two sets of logs for each run of gribmaster: griblog.HHMM and metalog.HHMM. Jim's solution is used in the gribmaster 2.0 software. You don't have to do anything to install this feature. Cronmaster is already set up to run runmetajobs automatically.
|
| Q | There are two kinds of operational Meso-Eta data on the OSO server these days. The full data set is labelled as 03 and 15Z, and a much smaller data set is available at 0Z and 12Z. Can I configure Gribmaster to download only one of these? |
| A | Yes, Gribmaster 2.0 allows you lots of flexibility in what data you wish to download. In the XXXCONF lines in the conf/grib.conf file, you can specify which model cycle times you want. For the case of the two varieties of Meso-Eta data:
You can see that the first entry specifies "03/15" in the 6th field of the XXXCONF line. This entry will download the full data set. The entry in MESO2CONF specifies the supplemental data set available from 00 and 12Z ("00/12") You can specify the model cycle times of *any* model in the 6th field of the XXXCONF line. For example, if you are only interested in the 9, 12, and 18Z RUC data, you can specify:
|
| Q | There are some models that are so large, I can't download them efficiently over the frame relay network. On the other hand, I would like to see at least some of the forecast data from these models. Is there a way that I can just download some of the forecast hours from a given model? |
| A | Yes, Gribmaster 2.0 allows you lots of flexibility in what data you wish to download. In the XXXCONF lines in the conf/grib.conf file, you can specify just the forecast hours you want. For example, the Meso-Eta "AWIPS3D" 25mb vertical resolution forecast grids are available at 3 hour intervals out to 33 hours, but the data set is huge (144MB total). You might decide only to download the F00-F12 forecast hours rather than the full data set. You can specify the forecast hours of interest in the 7th field of the XXXCONF line. In this case:
If an event merits downloading all the forecast hours, simply replace the list of forecast hours with "ALL":
You can specify the forecast hours to download for *any* model by modifying the 7th field of the XXXCONF line in grib.conf. |
| Q | In the old gribmaster (version 1.X), there were lots of problems using the "PCFTP" functionality to FTP data around my LAN. Has that been fixed? |
| A | Yes. The FTP'ing on the local LAN functionality of gribmaster 2.0 is much more robust. |
| Q | In the old gribmaster (version 1.x), sometimes I'd end up missing some data if the file on the server was updated after I had already downloaded it. Has that been fixed? |
| A | Yes. Gribmaster 2.0 has code to check the size of your local data file and the size of the data file on the server. If the data file on the server is larger, the whole data file is downloaded again. Note, there are two caveats to this functionality. 1) In order for this function to work, you must not be deleting your GRIB files. So, the variable $GRIBDEL in conf/grib.conf must be set to NO. 2) If a data file is downloaded again, and you are creating metafile graphics with that data, your metafile will likely have some frames out of order. For a solution to that problem, please see the Q&A about out-of-order metafile frames below. |
| Q | There was a problem using gribmaster 1.X to get the GRIB files of experimental data from my regional data server. Is this still a problem? |
| A | No. As long as your region is using gribmaster 2.0 to download experimental data to their server, you can use gribmaster 2.0 to download the experimental data to your own computer with no additional changes. |
| Q | Is there any way I can use gribmaster to download one specific model? |
| A | Yes, gribmaster will take one argument:
where "XXX" is any model that has a XXXCONF line in the file conf/grib.conf. So, if you normally do no download the Meso-Eta, but decide that you would like to see it for today, you can run "gribmaster meso" (as long as MESOCONF is a valid configuration line in grib.conf). Given the argument, gribmaster will just download the specified model. The "XXX" model does not even have to be in the $MODELLIST variable. You can use this feature interactively (from the command line), or you can even configure it in cron if you'd like. |
| Q | Every now and then, gribmaster stops working and I don't have any new data anymore. What happened? |
| A | There are two reasons why gribmaster might stop working:
The first thing to check is #1. If something changes at OSO, I'll send out a message on the soo_STRC mailing list. If something changes at your regional data server, watch for an email message from your data focal point at SSD. If all is well at the data server, then you must check your own system. The first place to start is in the griblog.HHMM files in the log/ directory. Check the most recent log file. (Use the "ls -lst | more" command to see a time ordered list of the log files.) The log file might be able to tell you where the problem is. The most common problem is a missing " in one of the conf files. If your log file says:
then it that is most likely the problem. There's an easy way to find exactly where the missing " should be. Edit the gribmaster script and add a "-v" to the top line:
Now, run gribmaster interactively from the command line and watch the output come to the screen:
The last thing on the screen should be the "Unmatched " " line. Look at the line above it. That is the line that is missing the ". Most likely this line is in conf/path.conf or conf/grib.conf. Try grib.conf first. Find the line, and make sure it contains the correct number and location of "'s. Don't forget to remove the "-v" from the first line of gribmaster. The "-v" makes the gribmaster output very verbose. You might also have this problem with the makemeta script. Check the log/metalog.HHMM files. If you see the same "Unmatched " " message, then the problem is most likely in the meta.conf file. Finally, if you have just upgraded your operating system, or if you have never run gribmaster on this computer before, make sure you have installed the necessary C-Shell patch on your computer. See the Installation Instructions for more info on this. |
| Q | I want to subset the Meso-Eta model output because it uses so much disk space. How do I do that? Also, how do I configure a metafile script to be run for the subsetted meso-eta model? |
| A | Gribmaster 2.0 allows you to define whether or not you want to subset model output. You can specify as many subset areas as you'd like. This is defined in the XXXCONF line in conf/grib.conf. For example, for the 3Z & 15Z Meso-Eta, specify "GRID" for no subsetting:
Or, specify "co" to subset the data over Colorado:
You can use any valid GEMPAK "GAREA" to specify your subset area. For example, you can use a station ID (DEN), a state (CO), or any geographic area defined in the $GEMTBL/stns/geog.tbl table (WEST). You can also specify a lat/lon box such as 35;-110;45;-90. You can specify multiple subset areas in the XXXCONF line, separated by "/":
When you specify any subsetting (anything other than "GRID"), the name of the resulting GEMPAK data file will be modified by that subset area. Using the example above, you will end up with three GEMPAK file names:
(You can see why using a lat/lon box for the subset area may not be such a good idea. While it is perfectly acceptable by the program, it does result in a very strange GEMPAK file name. As an alternative, I suggest that you add a line to the $GEMTBL/stns/geog.tbl table that specifies the lat/lon box of interest. That way, you can pick a readable name for the area which can be used on the XXXCONF line and the GEMPAK file name.) Normally, however, you will not be selecting multiple subset areas. (Although your regional data server may do this.) Instead, you'll likely chose a single subset area. Let's go back to the Colorado example:
This will result in one GEMPAK data file with the name "YYMMDDHH_meso_co.gem". Now, if you wish to create some metafile graphics from this GEMPAK data file, you must configure the conf/meta.conf file. If you hadn't subsetted the model output, you'd specify the metafile generation as follows:
This specification will run the eta_meta_na script using the YYMMDDHH_meso.gem data file. (The "meso" comes from the first field in the XXXCONF line in grib.conf.) However, in your case, you want to use the YYMMDDHH_meso_co.gem data file. So, the entry in meta.conf must be:
The metafile will be named "eta_co_YYMMDD_HH_co_na". Please note, depending on how the metafile generation script is written, you may have to create special NTRANS metafile directory for these metafiles. For example, in the metafile generation script, look for the DEVICE line:
If the DEVICE line uses the variable ${mdl_lc} to specify the metafile directory (as is shown above), then the metafile will be written to a directory called "eta_co". You will likely have to create this directory under $METDAT/meta. However, if you change the DEVICE line to use the variable ${MDL_DIR} to specify the metafile directory, then the metafile will be written to the regular "eta" directory:
The metafile itself will still be named "eta_co_YYMMDD_HH_co_na". So the important points are:
|
| Q | Sometimes, when part of a model comes in late, gribmaster will create metafile frames out of order. How can I fix this? |
| A | This is a tricky issue, because NTRANS doesn't know anything about time ordering. NTRANS simply displays the frames in the same order that they were added to the metafile. So, the only way to re-order the graphics in a metafile is to completely recreate the metafile. Of course, this can be a time consuming process. In Gribmaster 2.0, there is a way to specify that if a forecast hour comes in late, gribmaster should totally recreate the entire metafile. There are a couple of things that you must do to enable this function. First of all, in $RTDATA/grib.conf, you must specify "ALL" in the 9th field of the XXXCONF line for the model in question. Using the Eta model as an example:
The next part is a little tricky. The name of the metafile created is defined *inside* the metafile generation script. The gribmaster script does not have any information about how the metafile is named. Therefore, it is impossible for gribmaster to know if it should *recreate* the metafile or just create it. The only way that gribmaster can determine the name of the metafile is from the name of the script. So, the second thing you must do to implement the recreation of metafiles is to verify that the name of the metafile corresponds to the name of the metafile generation script. Again taking an example from the Eta model, let's say I wish to run the eta_meta_main script. In the eta_meta_main script, I find the line:
This line tells me that the metafile is named "eta_YYMMDD_HH_us_main". The script is named "eta_meta_main". As long as the string "main" in the script name matches the last part of the metafile name, gribmaster will know when it needs to recreate this metafile. On the other hand, let's say I want to run the script eta_meta_na. In the eta_meta_na script, I find the line:
This metafile is also named "eta_YYMMDD_HH_us_main", but the script name is "eta_meta_na". As it is, gribmaster will not know when to recreate the metafile. So, in order to implement this functionality, you'll either have to change the name of the script to "eta_meta_main", or you'll have to change the name of the metafile to:
This seems like an inordinate amount of work to implement this functionality. However, given that we have a large number of metafile generation scripts in the field, there's no other way to get the metafile script name into the gribmaster program. |
| Q | Sometimes the metafiles that I expect do not get generated. When I look in the log/meta.log file, I see the message "floating exception". Also, a "core" file shows up in my $RTDATA directory. What's wrong? |
| A | Well, usually a floating exception means that either their something wrong with the script, or you've run out of memory/swap space on your machine. The best place to start is to try to recreate the error so you can see exactly where it is occurring. To do this, try running the "makemeta" script (part of gribmaster) by hand. For example, if I expect the metafile "eta_971110_00_us_main" to be generated, but it is not, then the first thing to do is to run the metafile generation script again:
and watch the output on the screen. If you find the floating exception error, then there may be a problem with the script and you should check the section just above the crash. Often, these crashes can be eliminated by adding a smoothing function ("SM5S()") around the function in GFUNC. This is particularly true of RELH and precip fields. If the "SM5S()" doesn't help, and there is nothing else wrong with the section of the script (i.e. if you run just that part alone, it does not crash), then it is likely that the script has exceeded the available memory or swap space on your workstation. You'll have to decrease the memory requirements by making the graphics area smaller, plotting fewer fields, adding a "SM5S()" smoothing function to the GFUNC, or using less color fill. Occasionally, you won't get a floating exception when running the script interactively. Usually this means that other scripts/programs on your system are causing the memory limitation and subsequent crash. To solve this problem, you might consider creating the metafiles at a later time, when the workstation is less busy. Other alternatives to these problems are to move processing of these metafile on to another CPU (perhaps a Linux PC), or to add more swap space or memory to the computer. Also, I found that when I upgraded from HP-UX 9.0.5 to 10.2, many of the mysterious floating exceptions disappeared. |
 |
Go to the Soo STRC Data Page |
|
Go to the SOO/STRC Home Page |