Atmoswiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
======Testing WRF v4.0 for NDARB======

Below, you will find how the ND Cloud Modification Project runs data through WRF 4.0 instead of WRF 3.7. This is a general instruction on downloading, coding, and running WRF 4.0 but will also include specific examples that are useful for this project in particular. The WRF 4.0 outputs will be compared to the WRF 3.7 original outputs and real-time observations to reveal the differences between the two versions.

**If you ever get stuck to the point of no repair ... turn it off and on again! Most of the errors I ran into were fixed by simply logging off, getting a snack, and logging back in.**

====Tips for building WRF====

    *Use the instructions found at https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php#STEP1 if this is your first time working with WRF.
    *To download files onto a server use the command 'wget' before the weblink.
    *Always make sure the environmental variables are defined everytime you try to run WRF!
    *Don't use the command, export PATH=$DIR/mpich/bin:$PATH, it will cause errors down the road.
    *I used the Linux x86_64 option '32. (Serial)' for this project's purposes and '1=basic' for the following nesting question.
    *A complete guide to WPS's namelist can be found here, https://www2.mmm.ucar.edu/wrf/users/namelist_best_prac_wps.html
    *Download WPS so it is in the same directory as WRF. Don't put WPS inside the WRF directory.
    *If you would like to learn more about WRF, check out this User's Guide, https://homepages.see.leeds.ac.uk/~lecrrb/wrf/aRWUsersGuide.pdf

===== NDARB servers =====

**ssh -X -A forecast@wopr.atmos.und.edu : In this directory you will find,**
   *00Z WSM6 WRF outputs at '/home/forecast/western_fwrf370'
   *00Z WDM6 WRF outputs at '/home/forecast/western_fwrf370_wdm6'
   *12Z WSM6 WRF outputs at '/home/forecast/western_fwrf370_12Z'



**ssh -X -A forecast@western.atmos.und.edu : In this directory you will find,**
   *00Z Thompson at '/home/forecast/western_fwrf370_MP_8_alt'
   *12Z Thompson at '/home/forecast/western_fwrf370_12Z_MP_8_alt'

__All data can be found at '/home/data/nam/' in the respective directories.__


//For this project specifically, I have downloaded WRF v4.0 in /wrf within the Calgary server. Calgary is accessible at 'ssh first.last@134.129.222.140', and the password is specific to each person with an account on Calgary.//

To move files from one server to another use 'scp  -r //original file server//://pathway to file in original server// .' Don't forget to include a period at the end!
//Example=scp  -r forecast@western.atmos.und.edu:/home/data/wrfout_west_370_MP_8_alt/20220914 .//

Find WRF namelist options at https://esrl.noaa.gov/gsd/wrfportal/namelist_input_options.html

To see the namelist specifications that a wrf output file was run with, type: ncdump -h wrfout_d0...

---------------------------------------

===== WRFV3.7 vs WRF4.0 vs. WRF4.2 =====

The following was taken from the README files for each version. Most of the specifics have remained the same but there are two primary differences that are used in 4.0 but not 3.7: the Hybrid-sigma vertical coordinate coordinates and flux-adjusting surface data nudging.

**Advanced Research WRF (ARW) solver: Eulerian mass, hydrostatic and non-hydrostatic**
    *Arakawa C-grid staggering
    *Runge-Kutta 2nd and 3rd order timestep options
    *scalar-conserving flux form for prognostic variables
    *2nd to 6th order advection options (horizontal and vertical)
    *time-split small step for acoustic modes
    *small step horizontally explicit, vertically implicit
    *divergence damping option and vertical time off-centering
    *external-mode filtering option for mass model
    *hydrostatic option via namelist option
    *positive-definite and monotonic advection for moisture, scalar, tke and chemical tracers
    *global modeling capability on latitude-longitude grid
    *digital filter initialization
    *WENO advection options
    ***WRF4.0/4.2 ONLY Hybrid sigma-pressure vertical coordinate (since V3.9)**

**Two-way nesting:**
    *multiple domains and multiple nest levels
    *supports integer nest grid ratio
    *feedback option for both odd and even nest grid ratios
    *smoothing options

**Moving nest:**
    *Specified move
    *Automatic move using a mid-level vortex-following algorithm

**Physics options:**
    *microphysics (Kessler/ WRF Single Moment 3, 5 and 6 classes / Lin et al./ Mibrandt 2-moment / Eta Ferrier / Thompson / Goddard / 2-moment Morrison / WRF Double Moment 5 and 6 classes / SBU-Lin 5-classes / NSSL 2-moment and 1-moment / CAM 5.1 ) / Thompson aerosol-aware / HUJI full and fast SBM
    *cumulus parameterization (Kain-Fritsch with shallow convection / Betts-Miller-Janjic / Grell-Devenyi ensemble / Grell 3D (with shallow convection option) / Grell-Freitas ensemble / Tiedtke (with shallow conv and momentum transport) / NSAS (with shallow conv and momentum transport) / SAS (with shallow conv for ARW) / Zhang-McFarlane (with momentum transport) ) / New Tiedtke (with shallow conv and momentum transport) / Multi-scale KF (with shallow convection)
    *UW shallow convection / GRIMS shallow convection
    *planetary boundary layer (Yosei University / Mellor-Yamada-Janjic / ACM2 / QNSE-EDMF / MYNN / BouLac / UW / TEMF / Grenier-Bretherton-McCaa ) / Shin-Hong
    *slab soil model (5-layer thermal diffusion / Noah land-surface model (4 levels) / RUC LSM (6 levels) / Pleim-Xu (2 levels / Noah-MP (4 levels) / SSiB (3 levels) / CLM4 (10 levels) )
    *Urban canopy model, BEP multi-layer and BEM  (works with Noah LSM, BEP and BEM also requires MYJ or BouLac)
    *longwave radiation (RRTM / CAM / RRTMG / new Goddard / FLG )
    *shortwave radiation (Dudhia / old Goddard / CAM / RRTMG / new Goddard / FLG ), terrain slope and shading effect
    *sub-grid turbulence (constant K diffusion/ 2-D Smagorinsky/ predicted TKE / 2-D, 6th order diffusion / Nonlinear Backscatter Anisotropic (NBA) sub-grid turbulence stress for LES )
    *Rayleigh damping for w at the upper boundary layer
    *gravity wave drag (including flow blocking)
    *land-use categories determine surface properties; support for 24 category USGS and 20 category MODIS
    *Options for modifying SST, sea ice, vegetation fraction, albedo, and deep soil temp for long simulations
    *fractional sea ice option for polar regions; modified snow/ice physics
    *single-column ocean mixed layer model / 3D Price-Weller-Pinkel (PWP) ocean model
    *drag and enthalpy flux formulation for hurricane applications
    *CLM lake model
    *windfarm drag

**Nudging:**
    *three-dimensional and surface analysis nudging
    *observation nudging
    *spectral nudging
    ***WRF4.0/4.2 ONLY flux-adjusting surface data nudging**
    
**Thompson Microphysics options:**
    ***WRF 4.2 ONLY significant alteration is introduced for diagnosing the Y-intercept parameter for the one- 
    moment graupel. A few adjustments made for rain number concentration tendencies, two of which are bug fixes. 
    Also a minor adjustment for consistency to minimum cloud ice size in event a balance of ice number is needed. 
    Lastly a bug fix for partly-melted snow. The most notable changes are: (1) a larger mean rain size from cloud 
    water to rain autoconversion that helps to produce a stronger leading edge of convective squall lines; (2) 
    partly-melted snow was falling too rapidly while this fix produces very realistic progression of melting snow 
    falling gradually toward rain velocity as it melts.**


    Hybrid Vertical Coordinate: The HVC option is a "hybrid" vertical coordinate, in that the eta levels 
    are terrain following near the surface, and then relax towards an isobaric surface aloft.  The 
    purpose of this coordinate option is to reduce the artificial influence of topography towards the top 
    of the model.

---------------------------------------

=====Steps for Compiling WRF v4.0=====

The following are steps for compiling WRF and WPS. Detailed instructions can be found at https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php and Linux instructions can be found at https://metclim.ucd.ie/2017/06/wrf-installation-on-a-linux-machine/.


====1. Downloading WRF and WPS Source Code====

Source code information can be found at https://www2.mmm.ucar.edu/wrf/users/download/get_sources.html //This site is for all versions of WRF so make sure you are downloading the correct version.// For this project, we are upgrading to v4.0 so I downloaded the tar files for WRF Version 4.0 and WPS Version 4.0.

---------------------------------------
====2. System Environment Tests====

**You can skip this step if you are compiling WRF on a supercomputer such as Cheyenne. It should already be done in a shared directory.**

In the ///WRF// directory, type:
    mkdir Build_WRF
    mkdir TESTS

In the ///WRF/TESTS// directory, type:
    wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/Fortran_C_tests.tar
    tar -xf Fortran_C_tests.tar
    gfortran TEST_1_fortran_only_fixed.f
    ./a.out
Output: SUCCESS test 1 fortran only fixed format
    gfortran TEST_2_fortran_only_free.f90
    ./a.out
Output: Assume Fortran 2003: has FLUSH, ALLOCATABLE, derived type, and ISO C Binding SUCCESS test 2 fortran only free format
    gcc TEST_3_c_only.c
    ./a.out
Output: SUCCESS test 3 c only
    gcc -c -m64 TEST_4_fortran+c_c.c
    gfortran -c -m64 TEST_4_fortran+c_f.f90
    gfortran -m64 TEST_4_fortran+c_f.o TEST_4_fortran+c_c.o
    ./a.out
Output: C function called by Fortran Values are xx = 2.00 and ii = 1 SUCCESS test 4 fortran calling c
    ./TEST_csh.csh
Output: SUCCESS csh test
    ./TEST_perl.pl
Output: SUCCESS perl test
    ./TEST_sh.sh
Output: SUCCESS sh test
---------------------------------------
====3. Building Libraries====

**You can skip this step if you are compiling WRF on a supercomputer such as Cheyenne. It should already be done in a shared directory.**

The WRF Libraries required for configuring and compiling are version specific and need to be 100% correct. If you download an incorrect version of a library or simply put the download in the wrong folder then WRF will not compile correctly. Environmental Variables seen below need to be defined every time you want to run WRF or permanently saved.

Type:
    export DIR=/wrf/WRF/Build_WRF/LIBRARIES
    export CC=gcc
    export CXX=g++
    export FC=gfortran
    export FCFLAGS=-m64
    export F77=gfortran
    export FFLAGS=-m64
    export JASPERLIB=$DIR/grib2/lib
    export JASPERINC=$DIR/grib2/include
    export LDFLAGS=-L$DIR/grib2/lib
    export CPPFLAGS=-I$DIR/grib2/include
    export PATH=$DIR/netcdf/bin:$PATH
    export NETCDF=$DIR/netcdf
    export LD_LIBRARY_PATH=$DIR/lib:$LD_LIBRARY_PATH
Include "export LD_LIBRARY_PATH=$DIR/grib2/lib:$LD_LIBRARY_PATH" for errors including "libpng12.so.0".

In the ///WRF/Build_WRF// directory, type:
    mkdir LIBRARIES

In the ///WRF/Build_WRF/LIBRARIES// directory, type:
    wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/jasper-1.900.1.tar.gz
    wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/libpng-1.2.50.tar.gz
    wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/zlib-1.2.7.tar.gz
    wget http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/netcdf-4.1.3.tar.gz
Find other Netcdf-c versions at https://github.com/Unidata/netcdf-c/releases. Find other Netcdf-fortran version at https://github.com/Unidata/netcdf-fortran/releases.

In the ///WRF/Build_WRF/LIBRARIES// directory, type:
    tar xzvf netcdf-4.1.3.tar.gz
    cd netcdf-4.1.3
    ./configure --prefix=$DIR/netcdf --disable-dap --disable-netcdf-4 --disable-shared
    make
    make install
    cd ..

    tar xzvf zlib-1.2.7.tar.gz
    cd zlib-1.2.7
    ./configure --prefix=$DIR/grib2
    make
    make install
    cd ..

    tar xzvf libpng-1.2.50.tar.gz
    cd libpng-1.2.50
    ./configure --prefix=$DIR/grib2
    make
    make install
    cd ..

    tar xzvf jasper-1.900.1.tar.gz
    cd jasper-1.900.1
    ./configure --prefix=$DIR/grib2
    make
    make install
    cd ..

---------------------------------------
====4. Library Compatibility Tests====

In the ///WRF/TESTS// directory, type:
    wget https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/Fortran_C_NETCDF_MPI_tests.tar
    tar -xf Fortran_C_NETCDF_MPI_tests.tar
    cp ${NETCDF}/include/netcdf.inc .
    gfortran -c 01_fortran+c+netcdf_f.f
    gcc -c 01_fortran+c+netcdf_c.c
    gfortran 01_fortran+c+netcdf_f.o 01_fortran+c+netcdf_c.o -L${NETCDF}/lib -lnetcdff -lnetcdf
    ./a.out
Output: C function called by Fortran Values are xx = 2.00 and ii = 1 SUCCESS test 1 fortran + c + netcdf    

---------------------------------------
====5. Building WRF====

In the ///WRF// directory, type:
    ./configure

Compilation Options:
    *em_real (3d real case)
    *em_quarter_ss (3d ideal case)
    *em_b_wave (3d ideal case)
    *em_les (3d ideal case)
    *em_heldsuarez (3d ideal case)
    *em_tropical_cyclone (3d ideal case)
    *em_hill2d_x (2d ideal case)
    *em_squall2d_x (2d ideal case)
    *em_squall2d_y (2d ideal case)
    *em_grav2d_x (2d ideal case)
    *em_seabreeze2d_x (2d ideal case)
    *em_scm_xy (1d ideal case)

Type:
    ./compile em_real >& log.compile
Replace //em_real// with the compilation option of your choice. Compilation might take 20-30 minutes.

Type:
    ls -ls main/*.exe

===No Executable files?===
**If the executable files, wrf.exe OR real.exe have not appeared in the /WRF/main directory then stop now!** There is an issue with your libraries and further progress is not possible. 

Type (if executable files are missing):
    ./clean -a
Then, **redo STEPS 3, 4, and 5** and ensure the libraries are matching what version of WRF you are using. The issue could also be a missing environmental variable. Check out the WRF & MPAS-A Forum https://forum.mmm.ucar.edu/.

---------------------------------------
====6. Building WPS====

In the ///WPS// directory, type:
    ./clean
    ./configure
    ./compile >& log.compile

===No Executable files?===
**If the executable files, geogrid.exe, ungrib.exe, OR metgrid.exe have not appeared in the /WPS directory then stop now!** There is an issue with your libraries or WRF compilation and further progress is not possible. My solution to this issue is deleting the untarred WPS directory and redoing the WRF compilation steps. Check out the WRF & MPAS-A Forum https://forum.mmm.ucar.edu/ for other solutions.

---------------------------------------
====7. Static Geography Data====
Find all Geographical Input Data Mandatory Field Downloads at https://www2.mmm.ucar.edu/wrf/users/download/get_sources_wps_geog.html

In the ///WRF/Build_WRF// directory, type:
    wget https://www2.mmm.ucar.edu/wrf/src/wps_files/geog_high_res_mandatory.tar.gz
    gunzip geog.tar.gz
    tar -xf geog.tar

Edit the /WPS/namelist.wps file:
    &geogrid
        geog_data_path = path_to_directory/Build_WRF/WPS_GEOG


-----------------------------------------------------
====8. Collecting Data====

At this point, we will need data to run through WRF.
To test if WRF was compiled correctly, you can use a Case Study available online at https://www2.mmm.ucar.edu/wrf/OnLineTutorial/CASES/SingleDomain/index.php. The following steps can be modified in order to accommodate the data set you are using.

---------------------------------------
===8.1. Setup the Model domain===

On the same level as the ///WPS// and ///WRF// directories, type:
    mkdir DATA
Put all data files within the ///DATA// directory.

In the ///WPS// directory, type:
    ln -sf ungrib/Variable_Tables/Vtable.NAM Vtable
    ./link_grib.csh /pathway-to-DATA/DATA/

Output: GRIBFILE.AAA, GRIBFILE.AAB, GRIBFILE.AAC, ..., GRIBFILE.AAZ files in the ///WPS// directory.
**Be sure to link the correct data type. My data was NAM, yours might be GFS, ECMWF, RAP, or even SST.**

Edit the /WPS/namelist.wps file to match the input files. 

    vim namelist.wps

**Things to look out for:**   
   
  * If you are doing more than one domain, the "max domain" needs to be greater than one. You also need to add a separate column to each variable in the namelist if the variable changes from domain to domain. The first column will be for domain one and so on. If the variable is constant then one column will work for all domains.
  * Be sure the time/date is consistent with the data files you are trying to run through WRF.
  * The "geog_data_path" should be the pathway to where you downloaded the "WPS_GEOG" library.

In the ///WPS// directory, type:
    ./geogrid.exe
Output: geo_em.d01.nc, geo_em.d02.nc, ..., geo_em.d05.nc in the ///WPS// directory.

---------------------------------------
===8.2. Unpack input GRIB data===

In the ///WPS// directory, type:
    ./ungrib.exe
Output: FILE://Year//-//Month//-//Day//_//Hour//, FILE://Year//-//Month//-//Day//_//Hour+interval_seconds//, ..., FILE://Year//-//Month//-//Day//_//Hour+interval_seconds// in the ///WPS// directory.

---------------------------------------
===8.3. Interpolate the input data onto out model domain===

In the ///WPS// directory, type:
    ./metgrid.exe
Output: met_em.d01.//Year//-//Month//-//Day//_00:00:00.nc, ..., met_em.d02.//Year//-//Month//-//Day//_00:00:00.nc, ..., met_em.d05.//Year//-//Month//-//Day//_00:00:00.nc files in the ///WPS// directory. There should be a "met_em" file for each domain and for each output of WPS. For example, We ran a 24 hour case study with an interval time of 3 hours, this resulted in 8 files for each domain, 24 files in total.

**If you get an error that is similar to,**
    Processing domain 1 of 3
      Processing 2022-07-28_00
        FILE
    WARNING: Couldn't open file FILE:2022-07-28_00 for input.
    ERROR: The mandatory field TT was not found in any input data.
**This is because you didn't properly "ungrib" your data. Be sure you didn't skip step 8.2!**

---------------------------------------
===8.4. Run the model===

In the ///WRF/run// directory, you can either symbolic link the WPS output files or copy the files to this directory.
    ln -sf /pathway-to-WPS/WPS/met_em.d0* .
                    OR
    cp /pathway-to-WPS/WPS/met_em.d0* /pathway-to-WRF/WRF/run/

Edit the /WRF/run/namelist.input file to match the input files.

    vim namelist.input

For this project, we used the exact same namelist from previous WRF versions used for the project. The only differences were date/time shifts and we added "do_radar_ref = 1" in the physics section in order to additionally output radar plots. We also had to omit the "afwa" section for version 4.0 because they did not allow WRF to run properly.

**Things to look out for:**   
   
  * All variables should be consistent with what the "namelist.wps" said as you ran the files through WPS. Do not change up the max_dom or interval_seconds or anything else like this because this will create inconsistencies for the data and WRF will not run properly.
  * There needs to be a section header and commas after each variable.
  * Be sure to check the start and end date/time before running WRF. WRF is not intuitive enough to assume that the input files in the folder are the ones that it needs to run.

In the ///WRF/run// directory, type:
    ./real.exe
Output: wrfinput_d01, wrfinput_d02, ..., wrfinput_d05 AND wrfbdy_d01, wrfbdy_d02, ..., wrfbdy_d05 files in the ///WRF/run// directory.

**If you only get 'wrfinput' files for one or two of your domains, make sure your namelist has the correct 'max domain' and 'input_from_file' is 'true' for all your domains.**

In the ///WRF/run// directory, type:
    ./wrf.exe
Output: wrfout_d01_//Year//-//Month//-//Day//_00:00:00, ..., wrfout_d02_//Year//-//Month//-//Day//_00:00:00, ..., wrfout_d05_//Year//-//Month//-//Day//_00:00:00 files in the ///WRF/run// directory.

**If you receive an error after trying to run wrf.exe that reads,**

    FATAL CALLED FROM FILE:  <stdin>  LINE:     186
    nesting requires either an MPI build or use of the -DSTUBMPI option

**This means that you have compiled WRF for nesting data but your data isn't actually nesting. This will require you to recompile WRF.**


---------------------------------------

=====Other Information=====

====README files in the WRF directory====
   *///WRF/README.md// -- Weblinks for more information about WRF-ARW Modeling System
   *///WRF/doc/README.crtm// -- CRTM Coefficients
   *///WRF/doc/README.CTSM// -- Using CTSM with WRF
   *///WRF/doc/README.cygwin.md// -- Installing WRF on Cygwin
   *///WRF/doc/README.DA// -- WRFDA Public Domain Notice. Line 192, V4.0 Release Notes
   *///WRF/doc/README.DA_chem// -- Aerosol/Chemical Data Assimilation in WRFDA
   *///WRF/doc/README.hybrid_vert_coord// -- Hybrid Vertical Coordinate
   *///WRF/doc/README.hydro// -- Simplify the coupling of terrestrial hydrological models with the WRF model.
   *///WRF/doc/README.io_config// -- Run time control of IO stream variable sets
   *///WRF/doc/README.irr_diag// -- Outputting Integrated Reaction Rate diagnostics
   *///WRF/doc/README.madwrf// -- Improving the solar irradiance nowcast
   *///WRF/doc/README.netcdf4par// -- Using parallel IO through the netCDF-4 interface
   *///WRF/doc/README.rsl_output// -- Combining rsl.output and rsl.error files into one
   *///WRF/doc/README.SSIB// -- WRF coupled with SSiB v3.0 land-surface model
   *///WRF/doc/README.test_cases// -- Directions for running a test case and list of available test cases.
   *///WRF/doc/README.windturbine// -- Wind turbine drag parameterization schemes
   *///WRF/doc/README.WRFPLUS// -- WRFPLUS Model Version 3.9.1
   *///WRF/run/README.namelist// -- A complete list of the variables used in the WRF runs.
   *///WPS/README// -- A brief overview of the WRF Pre-Processing System.