Testing WRF v4.0 for NDARB

Below, you will find how the ND Cloud Modification Project runs data through WRF 4.0 instead of WRF 3.7. This is a general instruction on downloading, coding, and running WRF 4.0 but will also include specific examples that are useful for this project in particular. The WRF 4.0 outputs will be compared to the WRF 3.7 original outputs and real-time observations to reveal the differences between the two versions.

If you ever get stuck to the point of no repair … turn it off and on again! Most of the errors I ran into were fixed by simply logging off, getting a snack, and logging back in.

• Use the instructions found at https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php#STEP1 if this is your first time working with WRF.
• To download files onto a server use the command 'wget' before the weblink.
• Always make sure the environmental variables are defined everytime you try to run WRF!
• Don't use the command, export PATH=$DIR/mpich/bin:$PATH, it will cause errors down the road.
• I used the Linux x86_64 option '32. (Serial)' for this project's purposes and '1=basic' for the following nesting question.
• A complete guide to WPS's namelist can be found here, https://www2.mmm.ucar.edu/wrf/users/namelist_best_prac_wps.html
• Download WPS so it is in the same directory as WRF. Don't put WPS inside the WRF directory.
• If you would like to learn more about WRF, check out this User's Guide, https://homepages.see.leeds.ac.uk/~lecrrb/wrf/aRWUsersGuide.pdf

ssh -X -A forecast@wopr.atmos.und.edu : In this directory you will find,

• 00Z WSM6 WRF outputs at '/home/forecast/western_fwrf370'
• 00Z WDM6 WRF outputs at '/home/forecast/western_fwrf370_wdm6'
• 12Z WSM6 WRF outputs at '/home/forecast/western_fwrf370_12Z'

ssh -X -A forecast@western.atmos.und.edu : In this directory you will find,

• 00Z Thompson at '/home/forecast/western_fwrf370_MP_8_alt'
• 12Z Thompson at '/home/forecast/western_fwrf370_12Z_MP_8_alt'

All data can be found at '/home/data/nam/' in the respective directories.

For this project specifically, I have downloaded WRF v4.0 in /wrf within the Calgary server. Calgary is accessible at 'ssh first.last@134.129.222.140', and the password is specific to each person with an account on Calgary.

To move files from one server to another use 'scp -r original file server:pathway to file in original server .' Don't forget to include a period at the end! Example=scp -r forecast@western.atmos.und.edu:/home/data/wrfout_west_370_MP_8_alt/20220914 .

Find WRF namelist options at https://esrl.noaa.gov/gsd/wrfportal/namelist_input_options.html

To see the namelist specifications that a wrf output file was run with, type: ncdump -h wrfout_d0…

The following was taken from the README files for each version. Most of the specifics have remained the same but there are two primary differences that are used in 4.0 but not 3.7: the Hybrid-sigma vertical coordinate coordinates and flux-adjusting surface data nudging.

Advanced Research WRF (ARW) solver: Eulerian mass, hydrostatic and non-hydrostatic

• Arakawa C-grid staggering
• Runge-Kutta 2nd and 3rd order timestep options
• scalar-conserving flux form for prognostic variables
• 2nd to 6th order advection options (horizontal and vertical)
• time-split small step for acoustic modes
• small step horizontally explicit, vertically implicit
• divergence damping option and vertical time off-centering
• external-mode filtering option for mass model
• hydrostatic option via namelist option
• positive-definite and monotonic advection for moisture, scalar, tke and chemical tracers
• global modeling capability on latitude-longitude grid
• digital filter initialization
• WENO advection options
• WRF4.0/4.2 ONLY Hybrid sigma-pressure vertical coordinate (since V3.9)

Two-way nesting:

• multiple domains and multiple nest levels
• supports integer nest grid ratio
• feedback option for both odd and even nest grid ratios
• smoothing options

Moving nest:

• Specified move
• Automatic move using a mid-level vortex-following algorithm

Physics options:

• microphysics (Kessler/ WRF Single Moment 3, 5 and 6 classes / Lin et al./ Mibrandt 2-moment / Eta Ferrier / Thompson / Goddard / 2-moment Morrison / WRF Double Moment 5 and 6 classes / SBU-Lin 5-classes / NSSL 2-moment and 1-moment / CAM 5.1 ) / Thompson aerosol-aware / HUJI full and fast SBM
• cumulus parameterization (Kain-Fritsch with shallow convection / Betts-Miller-Janjic / Grell-Devenyi ensemble / Grell 3D (with shallow convection option) / Grell-Freitas ensemble / Tiedtke (with shallow conv and momentum transport) / NSAS (with shallow conv and momentum transport) / SAS (with shallow conv for ARW) / Zhang-McFarlane (with momentum transport) ) / New Tiedtke (with shallow conv and momentum transport) / Multi-scale KF (with shallow convection)
• UW shallow convection / GRIMS shallow convection
• planetary boundary layer (Yosei University / Mellor-Yamada-Janjic / ACM2 / QNSE-EDMF / MYNN / BouLac / UW / TEMF / Grenier-Bretherton-McCaa ) / Shin-Hong
• slab soil model (5-layer thermal diffusion / Noah land-surface model (4 levels) / RUC LSM (6 levels) / Pleim-Xu (2 levels / Noah-MP (4 levels) / SSiB (3 levels) / CLM4 (10 levels) )
• Urban canopy model, BEP multi-layer and BEM (works with Noah LSM, BEP and BEM also requires MYJ or BouLac)
• longwave radiation (RRTM / CAM / RRTMG / new Goddard / FLG )
• shortwave radiation (Dudhia / old Goddard / CAM / RRTMG / new Goddard / FLG ), terrain slope and shading effect
• sub-grid turbulence (constant K diffusion/ 2-D Smagorinsky/ predicted TKE / 2-D, 6th order diffusion / Nonlinear Backscatter Anisotropic (NBA) sub-grid turbulence stress for LES )
• Rayleigh damping for w at the upper boundary layer
• gravity wave drag (including flow blocking)
• land-use categories determine surface properties; support for 24 category USGS and 20 category MODIS
• Options for modifying SST, sea ice, vegetation fraction, albedo, and deep soil temp for long simulations
• fractional sea ice option for polar regions; modified snow/ice physics
• single-column ocean mixed layer model / 3D Price-Weller-Pinkel (PWP) ocean model
• drag and enthalpy flux formulation for hurricane applications
• CLM lake model
• windfarm drag

Nudging:

• three-dimensional and surface analysis nudging
• observation nudging
• spectral nudging
• WRF4.0/4.2 ONLY flux-adjusting surface data nudging

Thompson Microphysics options:

• WRF 4.2 ONLY significant alteration is introduced for diagnosing the Y-intercept parameter for the one- moment graupel. A few adjustments made for rain number concentration tendencies, two of which are bug fixes. Also a minor adjustment for consistency to minimum cloud ice size in event a balance of ice number is needed. Lastly a bug fix for partly-melted snow. The most notable changes are: (1) a larger mean rain size from cloud water to rain autoconversion that helps to produce a stronger leading edge of convective squall lines; (2) partly-melted snow was falling too rapidly while this fix produces very realistic progression of melting snow falling gradually toward rain velocity as it melts.

  Hybrid Vertical Coordinate: The HVC option is a "hybrid" vertical coordinate, in that the eta levels 
  are terrain following near the surface, and then relax towards an isobaric surface aloft.  The 
  purpose of this coordinate option is to reduce the artificial influence of topography towards the top 
  of the model.

The following are steps for compiling WRF and WPS. Detailed instructions can be found at https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php and Linux instructions can be found at https://metclim.ucd.ie/2017/06/wrf-installation-on-a-linux-machine/.

Source code information can be found at https://www2.mmm.ucar.edu/wrf/users/download/get_sources.html This site is for all versions of WRF so make sure you are downloading the correct version. For this project, we are upgrading to v4.0 so I downloaded the tar files for WRF Version 4.0 and WPS Version 4.0.

You can skip this step if you are compiling WRF on a supercomputer such as Cheyenne. It should already be done in a shared directory.

In the /WRF directory, type:

  mkdir Build_WRF
  mkdir TESTS

In the /WRF/TESTS directory, type:

  wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/Fortran_C_tests.tar
  tar -xf Fortran_C_tests.tar
  gfortran TEST_1_fortran_only_fixed.f
  ./a.out

Output: SUCCESS test 1 fortran only fixed format

  gfortran TEST_2_fortran_only_free.f90
  ./a.out

Output: Assume Fortran 2003: has FLUSH, ALLOCATABLE, derived type, and ISO C Binding SUCCESS test 2 fortran only free format

  gcc TEST_3_c_only.c
  ./a.out

Output: SUCCESS test 3 c only

  gcc -c -m64 TEST_4_fortran+c_c.c
  gfortran -c -m64 TEST_4_fortran+c_f.f90
  gfortran -m64 TEST_4_fortran+c_f.o TEST_4_fortran+c_c.o
  ./a.out

Output: C function called by Fortran Values are xx = 2.00 and ii = 1 SUCCESS test 4 fortran calling c

  ./TEST_csh.csh

Output: SUCCESS csh test

  ./TEST_perl.pl

Output: SUCCESS perl test

  ./TEST_sh.sh

Output: SUCCESS sh test

You can skip this step if you are compiling WRF on a supercomputer such as Cheyenne. It should already be done in a shared directory.

The WRF Libraries required for configuring and compiling are version specific and need to be 100% correct. If you download an incorrect version of a library or simply put the download in the wrong folder then WRF will not compile correctly. Environmental Variables seen below need to be defined every time you want to run WRF or permanently saved.

Type:

  export DIR=/wrf/WRF/Build_WRF/LIBRARIES
  export CC=gcc
  export CXX=g++
  export FC=gfortran
  export FCFLAGS=-m64
  export F77=gfortran
  export FFLAGS=-m64
  export JASPERLIB=$DIR/grib2/lib
  export JASPERINC=$DIR/grib2/include
  export LDFLAGS=-L$DIR/grib2/lib
  export CPPFLAGS=-I$DIR/grib2/include
  export PATH=$DIR/netcdf/bin:$PATH
  export NETCDF=$DIR/netcdf
  export LD_LIBRARY_PATH=$DIR/lib:$LD_LIBRARY_PATH

Include “export LD_LIBRARY_PATH=$DIR/grib2/lib:$LD_LIBRARY_PATH” for errors including “libpng12.so.0”.

In the /WRF/Build_WRF directory, type:

  mkdir LIBRARIES

In the /WRF/Build_WRF/LIBRARIES directory, type:

  wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/jasper-1.900.1.tar.gz
  wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/libpng-1.2.50.tar.gz
  wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/zlib-1.2.7.tar.gz
  wget http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/netcdf-4.1.3.tar.gz

Find other Netcdf-c versions at https://github.com/Unidata/netcdf-c/releases. Find other Netcdf-fortran version at https://github.com/Unidata/netcdf-fortran/releases.

In the /WRF/Build_WRF/LIBRARIES directory, type:

  tar xzvf netcdf-4.1.3.tar.gz
  cd netcdf-4.1.3
  ./configure --prefix=$DIR/netcdf --disable-dap --disable-netcdf-4 --disable-shared
  make
  make install
  cd ..

  tar xzvf zlib-1.2.7.tar.gz
  cd zlib-1.2.7
  ./configure --prefix=$DIR/grib2
  make
  make install
  cd ..

  tar xzvf libpng-1.2.50.tar.gz
  cd libpng-1.2.50
  ./configure --prefix=$DIR/grib2
  make
  make install
  cd ..

  tar xzvf jasper-1.900.1.tar.gz
  cd jasper-1.900.1
  ./configure --prefix=$DIR/grib2
  make
  make install
  cd ..

In the /WRF/TESTS directory, type:

  wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/Fortran_C_NETCDF_MPI_tests.tar
  tar -xf Fortran_C_NETCDF_MPI_tests.tar
  cp ${NETCDF}/include/netcdf.inc .
  gfortran -c 01_fortran+c+netcdf_f.f
  gcc -c 01_fortran+c+netcdf_c.c
  gfortran 01_fortran+c+netcdf_f.o 01_fortran+c+netcdf_c.o -L${NETCDF}/lib -lnetcdff -lnetcdf
  ./a.out

Output: C function called by Fortran Values are xx = 2.00 and ii = 1 SUCCESS test 1 fortran + c + netcdf

In the /WRF directory, type:

  ./configure

Compilation Options:

• em_real (3d real case)
• em_quarter_ss (3d ideal case)
• em_b_wave (3d ideal case)
• em_les (3d ideal case)
• em_heldsuarez (3d ideal case)
• em_tropical_cyclone (3d ideal case)
• em_hill2d_x (2d ideal case)
• em_squall2d_x (2d ideal case)
• em_squall2d_y (2d ideal case)
• em_grav2d_x (2d ideal case)
• em_seabreeze2d_x (2d ideal case)
• em_scm_xy (1d ideal case)

Type:

  ./compile em_real >& log.compile

Replace em_real with the compilation option of your choice. Compilation might take 20-30 minutes.

Type:

  ls -ls main/*.exe

If the executable files, wrf.exe OR real.exe have not appeared in the /WRF/main directory then stop now! There is an issue with your libraries and further progress is not possible.

Type (if executable files are missing):

  ./clean -a

Then, redo STEPS 3, 4, and 5 and ensure the libraries are matching what version of WRF you are using. The issue could also be a missing environmental variable. Check out the WRF & MPAS-A Forum https://forum.mmm.ucar.edu/.

In the /WPS directory, type:

  ./clean
  ./configure
  ./compile >& log.compile

If the executable files, geogrid.exe, ungrib.exe, OR metgrid.exe have not appeared in the /WPS directory then stop now! There is an issue with your libraries or WRF compilation and further progress is not possible. My solution to this issue is deleting the untarred WPS directory and redoing the WRF compilation steps. Check out the WRF & MPAS-A Forum https://forum.mmm.ucar.edu/ for other solutions.

Find all Geographical Input Data Mandatory Field Downloads at https://www2.mmm.ucar.edu/wrf/users/download/get_sources_wps_geog.html

In the /WRF/Build_WRF directory, type:

  wget https://www2.mmm.ucar.edu/wrf/src/wps_files/geog_high_res_mandatory.tar.gz
  gunzip geog.tar.gz
  tar -xf geog.tar

Edit the /WPS/namelist.wps file:

  &geogrid
      geog_data_path = path_to_directory/Build_WRF/WPS_GEOG

At this point, we will need data to run through WRF. To test if WRF was compiled correctly, you can use a Case Study available online at https://www2.mmm.ucar.edu/wrf/OnLineTutorial/CASES/SingleDomain/index.php. The following steps can be modified in order to accommodate the data set you are using.

On the same level as the /WPS and /WRF directories, type:

  mkdir DATA

Put all data files within the /DATA directory.

In the /WPS directory, type:

  ln -sf ungrib/Variable_Tables/Vtable.NAM Vtable
  ./link_grib.csh /pathway-to-DATA/DATA/

Output: GRIBFILE.AAA, GRIBFILE.AAB, GRIBFILE.AAC, …, GRIBFILE.AAZ files in the /WPS directory. Be sure to link the correct data type. My data was NAM, yours might be GFS, ECMWF, RAP, or even SST.

Edit the /WPS/namelist.wps file to match the input files.

  vim namelist.wps

Things to look out for:

• If you are doing more than one domain, the “max domain” needs to be greater than one. You also need to add a separate column to each variable in the namelist if the variable changes from domain to domain. The first column will be for domain one and so on. If the variable is constant then one column will work for all domains.
• Be sure the time/date is consistent with the data files you are trying to run through WRF.
• The “geog_data_path” should be the pathway to where you downloaded the “WPS_GEOG” library.

In the /WPS directory, type:

  ./geogrid.exe

Output: geo_em.d01.nc, geo_em.d02.nc, …, geo_em.d05.nc in the /WPS directory.

In the /WPS directory, type:

  ./ungrib.exe

Output: FILE:Year-Month-Day_Hour, FILE:Year-Month-Day_Hour+interval_seconds, …, FILE:Year-Month-Day_Hour+interval_seconds in the /WPS directory.

In the /WPS directory, type:

  ./metgrid.exe

Output: met_em.d01.Year-Month-Day_00:00:00.nc, …, met_em.d02.Year-Month-Day_00:00:00.nc, …, met_em.d05.Year-Month-Day_00:00:00.nc files in the /WPS directory. There should be a “met_em” file for each domain and for each output of WPS. For example, We ran a 24 hour case study with an interval time of 3 hours, this resulted in 8 files for each domain, 24 files in total.

If you get an error that is similar to,

  Processing domain 1 of 3
    Processing 2022-07-28_00
      FILE
  WARNING: Couldn't open file FILE:2022-07-28_00 for input.
  ERROR: The mandatory field TT was not found in any input data.

This is because you didn't properly “ungrib” your data. Be sure you didn't skip step 8.2!

In the /WRF/run directory, you can either symbolic link the WPS output files or copy the files to this directory.

  ln -sf /pathway-to-WPS/WPS/met_em.d0* .
                  OR
  cp /pathway-to-WPS/WPS/met_em.d0* /pathway-to-WRF/WRF/run/

Edit the /WRF/run/namelist.input file to match the input files.

  vim namelist.input

For this project, we used the exact same namelist from previous WRF versions used for the project. The only differences were date/time shifts and we added “do_radar_ref = 1” in the physics section in order to additionally output radar plots. We also had to omit the “afwa” section for version 4.0 because they did not allow WRF to run properly.

Things to look out for:

• All variables should be consistent with what the “namelist.wps” said as you ran the files through WPS. Do not change up the max_dom or interval_seconds or anything else like this because this will create inconsistencies for the data and WRF will not run properly.
• There needs to be a section header and commas after each variable.
• Be sure to check the start and end date/time before running WRF. WRF is not intuitive enough to assume that the input files in the folder are the ones that it needs to run.

In the /WRF/run directory, type:

  ./real.exe

Output: wrfinput_d01, wrfinput_d02, …, wrfinput_d05 AND wrfbdy_d01, wrfbdy_d02, …, wrfbdy_d05 files in the /WRF/run directory.

If you only get 'wrfinput' files for one or two of your domains, make sure your namelist has the correct 'max domain' and 'input_from_file' is 'true' for all your domains.

In the /WRF/run directory, type:

  ./wrf.exe

Output: wrfout_d01_Year-Month-Day_00:00:00, …, wrfout_d02_Year-Month-Day_00:00:00, …, wrfout_d05_Year-Month-Day_00:00:00 files in the /WRF/run directory.

If you receive an error after trying to run wrf.exe that reads,

  FATAL CALLED FROM FILE:  <stdin>  LINE:     186
  nesting requires either an MPI build or use of the -DSTUBMPI option

This means that you have compiled WRF for nesting data but your data isn't actually nesting. This will require you to recompile WRF.

• /WRF/README.md – Weblinks for more information about WRF-ARW Modeling System
• /WRF/doc/README.crtm – CRTM Coefficients
• /WRF/doc/README.CTSM – Using CTSM with WRF
• /WRF/doc/README.cygwin.md – Installing WRF on Cygwin
• /WRF/doc/README.DA – WRFDA Public Domain Notice. Line 192, V4.0 Release Notes
• /WRF/doc/README.DA_chem – Aerosol/Chemical Data Assimilation in WRFDA
• /WRF/doc/README.hybrid_vert_coord – Hybrid Vertical Coordinate
• /WRF/doc/README.hydro – Simplify the coupling of terrestrial hydrological models with the WRF model.
• /WRF/doc/README.io_config – Run time control of IO stream variable sets
• /WRF/doc/README.irr_diag – Outputting Integrated Reaction Rate diagnostics
• /WRF/doc/README.madwrf – Improving the solar irradiance nowcast
• /WRF/doc/README.netcdf4par – Using parallel IO through the netCDF-4 interface
• /WRF/doc/README.rsl_output – Combining rsl.output and rsl.error files into one
• /WRF/doc/README.SSIB – WRF coupled with SSiB v3.0 land-surface model
• /WRF/doc/README.test_cases – Directions for running a test case and list of available test cases.
• /WRF/doc/README.windturbine – Wind turbine drag parameterization schemes
• /WRF/doc/README.WRFPLUS – WRFPLUS Model Version 3.9.1
• /WRF/run/README.namelist – A complete list of the variables used in the WRF runs.
• /WPS/README – A brief overview of the WRF Pre-Processing System.