>From: Bob Rozumalski Date: Fri, 8 Nov 2002 13:04:21 -0500 (EST) To: soo_strc@comet.ucar.edu Subject: Message From Geoff DiMego Regarding UA QC FYI - Bob ------------- Begin Forwarded Message ------------- Date: Fri, 08 Nov 2002 13:02:08 -0500 From: "Geoff Dimego" Folks: I'd like to remind everyone that there is a ton of quality control applied to data within all of NCEP's data processing and analysis systems. You can see alot of this described under authors Collins (Bill) and Keyser (Dennis) in the Research Papers section of EMC's Mesoscale Modeling Branch web page at http://www.emc.ncep.noaa.gov/mmb/ The capability and sensitivity of our QC has grown steadily especially in light of how dramatically our obs database has grown over the years. I'm not sure if all forecasters are aware of just how much non RAOB data goes into our analyses these days. While we still consider them to be one of our most accurate and complete data sources, overall, RAOB data do NOT dominate our analyses anymore. The vast diversity of our observation database can be seen from the information in Suru Saha's displays of "GFS SSI data distributions" and "GFS SSI data dump counts" viewable from Verification and Diagnostics section of the Global Climate and Weather Modeling Branch web page at http://www.emc.ncep.noaa.gov/gmb/ With that said, let me address the facts in the case referred to in the WX-TALK item. There was some very moist levels in the Salem Oregon RAOB at 00z Wednesday 6 November 2002. The SDM saw bullseye features in two charts that he mistakenly associated with the Eta model analysis. In fact, these charts depict simple contoured analyses of the raw RAOB precipitable water and mean RH. By raw, I mean data that has not gone through ANY quality control at all. These charts are not made from Eta analyses. In fact, the Eta analyses did NOT reflect these anomalously moist levels - demonstrating the power of our objective quality control to discern and ignore dangerous departures. So rd's comment and the SDM's implication that bad data got into the Eta because of lack of human qc are in fact false. Now on to the general issue of human quality control. NCEP has conducted impact tests where the subjective quality control decisions were disabled and the Global Data Assimilation System and Global Forecasts System runs were made and compared with the operational runs which had the human quality decisions. We found no impact. The experiments were made with the global system because relatively few decisions are made in the brief time between receipt of RAOB data and the start of the Eta runs and, therefore, most of the subjective decisions are made outside of North America and in time for the GFS (what used to be known as AVN). These results show clearly that the vast majority of bad data identified and flagged by the human was also rejected by the objective quality control steps within the global analysis. These results are the reason why consideration is being given to modifying the practice of a human making quality control decisions. This is not to say that a human won't be monitoring the data flow and the quality of data AND the performance of the quality control. No consideration is being given to discontinuing the use of reject lists of consistently bad stations (which are and would continue to be maintained by a human). We do envision a much stronger synergy between the developers of the objective quality control algorithms in EMC and the human monitors who would provide periodic feedback on their perceptions of where and when the algorithms work poorly and need refinement. I do want to stress that there has been no final decision made yet. Kirby Cook wrote: > > Hi Geoff, > > A quick question that has come up from the field in responce to a > recent notice regarding the QC of Raobs before data ingest... > > I seemed to remember (I don't recall from where) that the role of the > SDM was being changed... does this fall under that... or is this > something else. Should we be concerned about this? > > Thanks! > > Kirby > -------- Original Message -------- > Subject: NCEP question > Date: Wed, 06 Nov 2002 10:02:06 -0700 > From: "Craig Schmidt" > Organization: NOAA/NWS/WR > To: Kirby Cook > > Kirby, > Do you have any idea who I could send the following messages from the Wx-Talk list? > Obviously they're a little over the top and emotional, but a good point is made. > I, too, would like to know why manual QC of the upper air data is going to end. > > Thanks, > Craig > > Date: Tue, 5 Nov 2002 22:17:42 -0500 > From: Rob Dale > Subject: Bad models now? > > Before I ask "what in the world was NWS thinking when they did this" -- can > someone confirm that what is being stated here is actually the case, and not > my mis-reading of tonight's model status product? > > === > > FYI...THE ABOVE STATIONS BAD RH GOT INTO THE ETA AND > PRODUCED A SPURIOUS BULLSEYE FOR AVG RH AND PCPTBL WATR > IN THE ANALYSIS OVR THE PACIFIC NW. THIS DATA WAS PURGED > FOR THE GFS. EXPECT MORE OF THESE KINDS OF ERRORS AFTER 12/01 > DUE TO THE FACT THE SDM WILL NO LONGER BE PERFORMING MANUAL > QC TO THE UA DATA AFTER THAT DATE. > > === > > So bad data (normally filtered out by human quality control) got in > to the > model and screwed the Eta in the northwest US, and that's something we > should see much more often? Why would human QC be removed from the process? > Are all met sensors now running at 100% quality? > > - rd > > ------------------------------ > > Date: Tue, 5 Nov 2002 22:21:26 -0600 > From: Steve Adams > Subject: Re: Bad models now? > > >So bad data (normally filtered out by human quality control) got into the > >model and screwed the Eta in the northwest US, and that's something we > >should see much more often? Why would human QC be removed from the process? > >Are all met sensors now running at 100% quality? > > I guess they're spending so much money on weather.noaa.gov and their ftp > infrastructure that they don't have enough left over to fund a warm body > to QC model inputs. ;-) > > Actually, this is surprising. It seems the computing facility has seen > good support over the years, with money provided for regular upgrades to > the hardw > are and for programming personnel to upgrade the models software. > What good would faster computers and superior model software do if the > initial conditions fed into it are cr.p? I'll be the first to admit that > automated QC processes can be very thorough, however... they certainly > can't catch everything. And it seems by the message that NWS knows that. > I realize cutbacks are sometimes necessary, but wouldn't model QC be > considered an "essential"?! > > This is a continuing trend at NWS. Many data products now go out without > a human being looking at them. Cooperative observer data, state/regional > temp and precip tables, hydrologic data... and now, it seems, model data. > Garbage in = Garbage out. Once again, quality is sacrificed in the name > of "efficiency". Thanks a bunch NWS. This will certainly bolster the > confidence of the users of your "new and improved" models. > > Steve > > ------------------------------ > > Date: Tue, 5 Nov 2002 23:07:55 -0600 > From: Gilbert > Sebenste > Subject: Re: Bad models now? > > On Tue, 5 Nov 2002, Steve Adams wrote: > > > >So bad data (normally filtered out by human quality control) got into the > > >model and screwed the Eta in the northwest US, and that's something we > > >should see much more often? Why would human QC be removed from the process? > > >Are all met sensors now running at 100% quality? > > > > > > > > I guess they're spending so much money on weather.noaa.gov and their ftp > > infrastructure that they don't have enough left over to fund a warm body > > to QC model inputs. ;-) > > > > Actually, this is surprising. It seems the computing facility has seen > > good support over the years, with money provided for regular upgrades to > > the hardware and for programming personnel to upgrade the models software. > > What good would faster computers and superior model software do if the > > initial conditions fed in > to it are cr.p? I'll be the first to admit that > > automated QC processes can be very thorough, however... they certainly > > can't catch everything. And it seems by the message that NWS knows that. > > I realize cutbacks are sometimes necessary, but wouldn't model QC be > > considered an "essential"?! > > > > This is a continuing trend at NWS. Many data products now go out without > > a human being looking at them. Cooperative observer data, state/regional > > temp and precip tables, hydrologic data... and now, it seems, model data. > > Garbage in = Garbage out. Once again, quality is sacrificed in the name > > of "efficiency". Thanks a bunch NWS. This will certainly bolster the > > confidence of the users of your "new and improved" models. > > Good grief, Steve. I just read the original post and yours and my head is > still spinning. If you have to consolidate a NWS forecast office or fire > someone to get two warm bodies to QC the data, then DO IT! Holy smokes, > what a > disaster!!! > > I don't know who to write to try to change this, but using the word > "lunacy" doesn't even come close to describe the incredible lack of > wisdom it took to come to this decision. Don't say money---whatever you > do, don't say money. Fire everyone at NCEP if you do this...at least you > will spare them from being a laughing stock of the WMO when the models > head straight for the toilet in terms of quality. If I were a modeller at > NCEP, I'd take that as a kick in the groin instead of a vote of > confidence. Please, if someone knows who to write to fix this horrid > decision, I'm all ears. I'm not NWS bashing, but I am holding the feet > under the fire to those who made a decision that, I firmly believe, will > ultimately destroy a considerable amount of integrity that comes from > the National Weather Service! > > ******************************************************************************* > Gilbert Sebenste ******** > Internet: gilbert@niu.edu (My opinions only!) ****** > Staff Meteorologist, Northern Illinois University **** > E-mail: sebenste@weather.admin.niu.edu *** > web: http://weather.admin.niu.edu ** > Work phone: 815-753-5492 * > ******************************************************************************* > > ------------------------------ > > End of WX-TALK Digest - 2 Nov 2002 to 5 Nov 2002 (#2002-264) > ************************************************************ --------------340F8DFE1AD41DD9F421F9F7 Content-Type: text/x-vcard; charset=us-ascii; name="Geoff.DiMego.vcf" Content-Transfer-Encoding: 7bit Content-Description: Card for Geoff DiMego Content-Disposition: attachment; filename="Geoff.DiMego.vcf" begin:vcard n:DiMego;Geoff tel;fax:301-763-8545 tel;home:410-451-3437 tel;work:310-763-8000 ext 7221 x-mozilla-html:FALSE org:DOC/NOAA/NWS;NWS/NCEP/EMC version:2.1 email;internet:geoff.dimego@noaa.gov title:Chief, Mesoscale Modeling Branch adr;quoted-printable:;;NOAA/NWS/NCEP/EMC=0D=0AW/NP22 5200 Auth Rd., Rm 205;Suitland;MD;20746-4304;USA fn:Geoff DiMego end:vcard --------------340F8DFE1AD41DD9F421F9F7-- ------------- End Forwarded Message -------------