[wrfems] Using the adaptive time step with nested runs (Was: Major Temperature Problem)

Tue Apr 3 08:06:39 MDT 2012

Hello Jon,

My primary conclusion after 2 weeks of testing was that the adaptive 
time step is unreliable when running nested simulations.

I have not had as many problems with single domain simulations. The few 
I've seen can be corrected by changing the adaptive
time step options in the configuration file.

However, I've exhausted all my options in attempting to provide a robust 
workaround to the problem of  the model crashing
when nested domains are included.  There appears to be two sources of 
error, a violation of the vertical CFL condition and a
disruption in the  of physics timing.  It is possible to tweak  the 
adaptive TS  configuration settings for a specific domain/case
but those settings are probably not a panacea for all simulations, which 
is  rather frustrating  for me.

The issues appear to primarily affect the child domains, but the 
consequences may take down the entire simulation.

I'm hoping that WRF V3.3+ has a fix for these problems but I'm not that 
far along in the testing.

Bob

4/3/12 7:38 AM, Case, Jonathan (MSFC-ZP11)[ENSCO INC] wrote:
>
> Hi Bob,
>
> Were you able to determine whether this adaptive time-step problem 
> occurs only on the inner nests, or on all domains including d01?
>
> Thanks,
>
> Jon
>
> *From:*wrfems-bounces at comet.ucar.edu 
> [mailto:wrfems-bounces at comet.ucar.edu] *On Behalf Of *Robert Rozumalski
> *Sent:* Tuesday, March 27, 2012 9:06 AM
> *To:* wrfems at comet.ucar.edu
> *Subject:* Re: [wrfems] Using the adaptive time step with nested runs 
> (Was: Major Temperature Problem)
>
>
> Good morning all,
>
> I'm just following up to Don's issue regarding a serious problem with 
> the near-surface fields when running a nested simulation with an 
> adaptive time step.
>
> Symptom of the problem:
>
>   Many near surface fields within a nested domain, such as 2m 
> temperature, stop updating during the course of a simulation. This 
> problem may occur from
>   the beginning, or start at  some point during a simulation.  It also 
> may start/stop multiple times over an entire simulation.  This issue 
> tends to be sporadic,
>   sometimes occurring  over consecutive simulations and then disappear 
> for an extended period of time.
>
>   The problem only appears when the adaptive time step is used in a 
> nested simulation.
>
>
> After days of scouring through the code and running many dozen test 
> simulations, I've determined that the nested domain tendencies coming 
> out of the
> LSM and PBL schemes are not being added to the model perturbation 
> fields.  It appears this problem is due to the timing between 
> ever-changing model
> time step and the frequency of calls to the physics schemes.  My guess 
> is that the tendency fields are being zero'd out before they are added 
> to the
> model perturbation fields but I'm not completely certain if this is 
> the exact cause.
>
> I've been able to correct the problem by changing the frequency of 
> calls to the physics schemes (CUDT, BLDT and RADT) and modifying the 
> adaptive
> time step  parameters for Don's case but I need to run some additional 
> simulations in order to develop some general configuration guidelines.
>
> I hope to have this information available this afternoon.
>
> Bob
>
>
>
>
> On 3/19/12 12:16 PM, Don Van Dyke wrote:
>
> Thanks for your help.  Unfortunately, I only saved the grib files from 
> a failed run and didn't think to save the raw netCDF files, so I don't 
> have any failed netCDF files to look at for the moment.  The 12z run 
> today worked fine.  The only change we've made so far is to remove the 
> RUC from the initialization and just go with the NAMPTILE and 
> NAMPTILELSM with the sport SSTs for initialization.  If another run 
> fails, I'll try to get the netCDF copied off to look at.  I really 
> hope the cause is not the adaptive time-step somehow, since that's the 
> only way we can cram 4 runs per day on that computer.  However, we've 
> had the adaptive running for around a month now without any problems 
> at all.
>
> This issue brought up another question with the RUC data.  The RUC is 
> scheduled to be discontinued quite soon and replaced by the RAP.  Will 
> be able to use the RAP data right away, or will we need to wait for 
> another build and resort to using something other than RUC in the 
> meantime?  Thanks again!
>
> -Don
>
> Robert Rozumalski wrote:
>
>
> Hello Don,
>
> I've attempted to run a simulation using your configuration and got an 
> interesting result, a segmentation fault after 23 hours.
>
> Typically, a segmentation fault that occurs well into a simulation 
> suggests a problem with the WRF code. It's possible that you
> are getting the same errors without a seg fault since the fault will 
> occur when the model is attempting to access a block of
> memory on a system that is not accessible.  In your case it may be 
> that wrf is accessing memory from an allowed area but
> it's not the correct area, so the simulation continues.
>
> For a failed run, take a look at the domain 2 WRF netCDF output files 
> in the  wrfprd directory  with "ncview":
>
>    %  ncview <netCDF file>
>
>        Look at:   2D Vars -> 2T    (2m temp)
>
> Start at the beginning of the run and work forward in time. Do you see 
> anything unusual?
>
> I suspect you will see an increase in the amount of white area with 
> time. The white indicates that the data are bad.
>
> I'm working  to determine the cause of the issue.
>
> Bob
>
>
>
> On 3/18/12 6:19 AM, Don Van Dyke wrote:
>
> Bob and others,
>
> We have developed a major temperature problem with our local WRF here 
> at WFO Tallahassee in the last 2 days.  For some unknown reason, the 
> surface and boundary layer temperatures are occasionally not rising 
> during the daylight hours for our inner 4 km nest.  We run a 12 km/4 
> km configuration.  What is even stranger about this problem is that 
> the 12 km outer nest is just fine, and the problem seems 
> intermittent.  The nest is configured as a one-way nest though, so I 
> guess the problem must be isolated to the 4 km version of the model 
> somehow.  I first noticed this problem with the 12z March 17th run, 
> then the subsequent 18z and 00z runs were fine, then the problem 
> reappeared on the 06z March 18th run.  I've attached some screenshots 
> as an example.  On the 12z March 17th run, the 4km model forecast 
> areas of fog all day for the 18th with highs near 60 while the 12 km 
> parent nest was normal with highs in the 80s.  I've also attached the 
> log files from the 06z March 18th run.   We've never seen this happen 
> before, and we're at a loss to explain why it's only affecting the 4 
> km nest and not the 12 km run, and why the problem seems to be 
> intermittent.  Also, to my knowledge, nothing has changed on our 
> computer other than I recently added crons for 00z and 12z runs.  (We 
> previously only did 06z and 18z runs.)  However, I did not change 
> anything with the model configuration.  This just suddenly started 
> happening.  Thanks for any help anyone can give!
>
> Don Van Dyke
> General Forecaster
> NWS Tallahassee, FL
>
>
> _______________________________________________
> wrfems mailing list
> wrfems at comet.ucar.edu <mailto:wrfems at comet.ucar.edu>
>
>
>
> -- 
> Robert A. Rozumalski, PhD
> NWS National SOO Science and Training Resource Coordinator
>
> COMET/UCAR PO Box 3000   Phone:  303.497.8356
> Boulder, CO 80307-3000
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> wrfems mailing list
> wrfems at comet.ucar.edu <mailto:wrfems at comet.ucar.edu>
>
>
>
>
> _______________________________________________
> wrfems mailing list
> wrfems at comet.ucar.edu  <mailto:wrfems at comet.ucar.edu>
>
>
>
>
> -- 
> Robert A. Rozumalski, PhD
> NWS National SOO Science and Training Resource Coordinator
>   
> COMET/UCAR PO Box 3000   Phone:  303.497.8356
> Boulder, CO 80307-3000
>   

-- 
Robert A. Rozumalski, PhD
NWS National SOO Science and Training Resource Coordinator

COMET/UCAR PO Box 3000   Phone:  303.497.8356
Boulder, CO 80307-3000

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.comet.ucar.edu/pipermail/wrfems/attachments/20120403/2959d9ed/attachment-0001.html>