Skip to content
Success

Changes

Summary

  1. Clubb ghost exorcism (#1159) (details)
  2. Updated the README to explain the grid notations gr%nzm and gr%nzt, (details)
Commit fa3e59573b8c754e973edfb4a08117a6efa14268 by noreply
Clubb ghost exorcism (#1159)

* I am adding grid_class_new.F90 to the code as a copy of grid_class.F90.

The first step toward removing the ghost level from the model is to
modify the grid file. Having a second grid file in play will allow
functions and subroutines that are undergoing modification earlier in
this process to use the new file, while functions and subroutines that
have not yet been modified will continue to use the old grid file
until they can be modified.

At the end of this process, the old grid file will be able to be
removed and grid_class_new.F90 will simply be able to be renamed
grid_class.F90 at that point.

In this commit, I have altered the schematic and the comments
at the very start of the module to be "ghostless".

* Updated grid_class for the changes to the grid interpolation weighting
functions for the new "ghostless" ascending grid.

* Committing some updates to the new grid class file for ghostless
ascending.

* Updating to match recent changes made to grid_class.F90.

* I finished the removal of the ghost level from the entirety of the new
grid_class file. This standalone file compiles successfully.

* Removing the sources of compiler warnings in grid_class_new.F90 (these
were carried over from grid_class.F90).

* Committing progress in advance_wp2_wp3_module.F90 with regards to
ghostless grid.

* I am committing the updated version of grid_class_new.F90 that allows
all cases to run, regardless of grid_type.

* Fixed a bug in grid_class_new.F90

When grid_type = 2, the calculations of the indices begin_height and
end_height are based on the thermodynamic level grids. Those grid indices
should be used in the call to setup_grid_heights, where the array of
thermodynamic_heights, from index begin_height to index end_height,
should become gr%zt.

When grid_type - 3, the calculations of the indices begin_height and
end_height are based on the momentum grid levels.

Either way, momentum_heights and thermodynamic_heights should both
have the array from begin_height to end_height passed in. When
grid_type = 2, only the thermodynamic height array is relevant in
the setup. When grid_type = 3, only the momentum height array is
relevant in the setup.

* For thermodynamic_height, 1:nzmax-1 is the entire array, but I
entered this for a clarification, especially considering the
following check block.

* Updated grid_class_new to better handle the begin_height and end_height indices.

* Some modifications to advance_wp2_wp3

* Committed some temporary code.

* committed some temporary changes to clubb_api_module.

* Making a bunch of commits to clubb_driver that are mainly temporary for
interface and printing purposes.

* Deghosted another subroutine.

* Updated wp2_term_ta_lhs to use the new gridding.

* De-ghosting wp2_terms_ac_pr2_lhs

* De-ghosted wp2_term_pr3_rhs

* Deghosting wp3_term_tp_lhs.

* Deghosted wp3_terms_ac_pr2_lhs

* Updated the wp3_term_ta_ADG1_lhs subroutine to use boundary conditions
where wp3 (at the top level and bottom momentum levels of the model)
is assumed to have a value of 0.

* For single loop vectors, removing the "collapse(2)" from the acc
statements.

This commit is not bit changing, but the previous commit to this branch
was BIT_CHANGING:902d6caed8a4533f59f4165106276959b4f90895

* I updated wp3_term_ta_explicit_rhs for the ghostless grid.

When l_explicit_turbulent_adv_wp3 is turned on, the results from
this branch exactly match those found on the master branch.

* Updating diffusion_zm_lhs for "Ghostless" gridding.

* Fixed a typo in the comments that was introduced in the previous commit.

* Updated for "ghostless" diffusion.

* With wp3 now being set to a fixed-point boundary condition value
of 0 at the current thermodynamic level 1 (first t-lev above the
surface), it is not necessary to loop over level 1 within the code
for each wp3 term.

Also contains code to enforce a fixed-point boundary condition
of wp3=0 at the first thermodynamic level above the surface.

* Restoring diffusion to the way it should be for the exorcised ghost grid
prior to the most recent master merge. Lines for the ghost point were
accidentally added during the merge commit.

* Updated code to correct wp3_bp1 and wp3_pr2 budget terms so that
they still match bit-for-bit what is found in the master branch.

* Updating term_ma_zm_lhs for ghostless gridding. Results are bit-for-bit
identical.

* After debugging, I am now committing the ghostless version of
term_ma_zt_lhs.

* Updating the clubb_ghost_exorcism branch for ghostless wp23 rhs,
lhs, and solve.

This revision runs; however, there are still some issues tracking
down the source of problems in maintaining bit-for-bit results.

* Updated to maintain bit-for-bitness with the master code.

* Finished exorcising the ghost from advance_wp2_wp3_module.F90.

* Small openacc changes to fix GPU runs

* Adjusting new hybrid PDF code to continue to match the master branch
bit-for-bit while construction continues.

* Committing the first portion of a nomenclature change where the number
of momentum vertical grid levels is denoted as nzm rather than nz.

* Changing the notation of the number of grid levels on the momentum grid,
previously nz, to nzm in some files.

* Changed some comments and a little bit of code in the new grid_class
file to provide a better description.

* Updated sigma_sqd_w_module for ghostless gridding.

* I added the new "ghostless" discretization to turbulent_adv_pdf.F90,
and also appropriately altered the points in the code where the
turbulent advection subroutines are called from in
advance_xm_wpxp_module.F90 and advance_xp2_xpyp_module.F90.

The code produced bit-for-bit results in regards to the standard
configuration, the new hybrid PDF configuration, and also for the
configuration with the godunov flags turned on.

* Altered many of the lower-level subroutines in
advance_xp2_xpyp_module.F90 for ghostless gridding.

* Upgraded subroutine calc_xp2_xpyp_ta_terms for ghostless discretization.

This should alter the results when the new_hybrid PDF is used in
conjunction with upwind discretization, but it doesn't alter the
results for anything else.

* Completed making advance_xp2_xpyp ghostless.

* Completed the de-ghosting of advance_windm_edsclrm.

Results are BIT_CHANGING because the ghost level was interacting with
the results.

* The call to calc_xpwp from advance_microphys was not supposed to be
updated yet. It wound up being sort of "partially updated" for
ghostless gridding, which was a bug that led to some diagnostic output
fields not being bit-for-bit. In this commit, I returned it to its prior
form so that zm output files once again retain bit-for-bit status
between this branch and the master.

* The ghost point interface statements after the return from
advance_windm_edsclrm needed to be encased in the same conditional
statements that the calculations occurred in with the aforementioned
subroutine. This change fixes a bug where a handful of cases had results
that were not bit-for-bit from revisions before this code was added.

* Fixed bug where the wrong stats were being recorded to the stats
for coef_wpthlp2_implicit, etc.

* I am committing a bug fix to the implicit surface flux calculation
in the eddy-diffusivity wind code (which is used when
l_predict_upwp_vpwp is turned off).

I am also committing some temporary ghost point interface stats code
within advance_windm_edsclrm.

* I updated the diagrams in advance_windm_edsclrm to reflect the
new ghostless grid.

* I improved the description section of advance_windm_edsclrm.

* Made numerous low-level subroutines ghostless within
advance_xm_wpxp_module.F90. Maintained bit-for-bit results
with the master branch.

* I am updating the xm wpxp code that calls the wpxp turbulent advection
term based on your PDF and option for ghostless gridding.

* Updating some of the lower-level functions of the monotonic flux limiter
for ghostless gridding.

* I have updated the monotonic flux limiter for ghostless gridding!

* A couple of redundant declarations were brought in on the last merge.

* Committing further updates to ghostless gridding, including:

1) Further modifications to advance_xm_wpxp, which includes branching
out in Brunt-Vaisala frequency calculations in advance_helper;

2) Update in calculating the initial pressure;

and 3) updating grid class new to use the same "value drop down"
in the zt2zm function and "derivative drop down" in the ddzt function
that are now used in grid_class in the master.

Results are bit-for-bit identical with the master.

* Updated xm_wpxp_lhs for ghostless gridding.

* Made xm_wpxp_rhs ghostless.

* Updated xm_wpxp_solve for ghostless gridding!

* Updating pos_definite_module.F90 for ghostless gridding.

* Updated vertical hole filling and xm clipping and stats for
ghost point removal.

Results are still bit-for-bit with the master branch.

* Updated advance_xm_wpxp to remove the ghost point.

* Updating some zt stats to use a value of 0 for the "ghost" level.

* Updated subroutine compute_mixing_length for ghostless gridding.

* Removing the ghost point from calculate_Lscale_directly

* Added a "ghostbuster" option (-g) in the run_bindiff_all.py python
script that omits the "ghost" level from the comparison for _zt.nc
output files when envoked.

* I de-ghosted the l_diagnose_Lscale_from_tau code.

* Fixing a couple bugs with the merge.

* De-ghosted a couple of subroutines with advance_helper_module.F90.

* Code wasn't compiling because of too many characters on one line.

* De-ghosted precipitation_fraction.F90.

* De-ghosted more code, including sfc_varnce_module.F90.

* De-ghosted numerical_check.F90

* Committing a chunk of code that effectively "finishes off" the
deghosting of CLUBB core, although results aren't there yet and it still
needs more debugging.

* This corrects the bug where an error would occur when trying to compare
a run with a ghost level to a run without a ghost level.

* Commiting some necessary ghost point removal code.

* Updated the calculations of rcm_in_layer and cloud_cover for ghostless
gridding.

* I removed the ghost point from all the files within the Benchmark_cases
directory.

* I am committing code that removes the ghost level from
setup_clubb_pdf_params and most of the collection of CLUBB's
microphysics schemes.

* Fixing a bug in SILHS where sample point values were being overwritten
at level 1.

* I fixed some bugs in the ghostless version of the Morrison microphysics
driver.

* I fixed some array declaration mismatches in silhs_api_module.F90.

* I have deghosted the call to the code that calculates microphysics
tendencies and everything below it.

* Performed a de-ghosting, as well as an improvement that ensures
that mean sedimentation is always conservative (excluding the
amount of hydrometeor that leaves the domain at the surface),
to all subroutines at the level of microphys_lhs and lower
in advance_microphys_module.F90. This includes all
process-related subroutines like those that govern
sedimentation or turbulent sedimentation.

Gee, how is it possible that such are large amount of work
gets done when there's no office. It must have been magic
elves who did it. I mean, a physical office must be key
to every element of functioning in life, right?

* Reversing the accidental stats file commit I made in the previous
commit.

* Updating the "leftovers" from the previous merge to make it consistent
with the call syntax in the new, ghostless CLUBB.

* Updated the turbulent sedimentation code to handle upper and lower
boundary conditions in a better manner.

* I have finished de-ghosting the entirety of advance_microphys_module.F90
and all codes that are underneath its umbrella.

* I finished de-ghosting the radiation portion of CLUBB.

* I have removed the ghost level from hydrostatic_module.F90.

* I removed the ghost point from parameters_tunable and started
referencing the ghostless grid through the entirety of clubb_api_module
as well as in all of the G-unit tests.

* Deghosted another file. Not too many left.

* Deghosted sounding.F90

* I have deghosted clubb_driver.F90 and everything below its umbrella,
which is a huge benchmark in this process.

* It is not necessary to de-ghost the source code for the tuner, since
the information on what levels it is tuning for is fed in from input
files.

Within the tuner input files, I have adjusted the tuning ranges by 1
to compensate for the fact that the ghost level has now been entirely
removed from the model.

* Got rid of some unnecessary "use grid_class" statements that were
still referencing the old grid.

* I have found a bug in the ghostless SILHS code in the SILHS
code that calculates the variances and covariances of moisture
and heat (that are fed back into the model predictive equations).
In short, variables that were used to store the grid mean values
of thl, rt, and w were being set to 0 at level 1. This was fine
when level 1 was the ghost level, but level 1 is no longer the
ghost level. This was root cause of the issue.

* The line that sets rand_pool at level 1 that was merged in during
the previous merge commit needs to be removed.

* These lines also needed to be fixed because they were supposed to
be changed to nzt / 2 after the previous merge commit.

* Correcting a bug in coamps microphysics regarded the unnecessary
interpolation of hydrometeor sedimentation velocities to the zm grid.
This interpolation is a bug because CLUBB expects hydrometeor velocities
output on the zt grid.

* I am committing a fix to the _ta budget terms. It will fix the error
messages as well allow for budget term agreement between the master
and clubb_ghost_exorcism branches.

* Fixing an error in the ghostless budget stats for turbulent
sedimentation.

* After the merge, I am once again updating the merged code for
ghost level removal.

* Getting rid of leftover ghost removal junk that was still lying around.

* I needed to pass p_sfc inside CLUBB core to help set the value
of p_in_Pa_zm at level 1.

* C11_Skw_fnc is a zt variable, while Cx_fnc_Richardson is a zm variable.
The offset is to keep results the same between the current master
branch and the clubb_ghost_exorcism branch when the l_use_C11_Richardson
flag is enabled.

* Fixed a bug.

* I found a couple more instances where the api had dimensions switched
around.

* Eliminated diffusion_cloud_frac_zt_lhs, which is unused code that isn't
even hooked up (called from) anywhere in the model anymore.

* Changes that also needed to be made with the previous commit.

* The source of the array out-of-bounds error.

* Moving grid_class_new back to grid_class!

* Reminder notes in case input_fields gets used in the future.

* I accidentally committed input fields related changes to rico_model.in
that weren't meant to be committed in the previous commit. I am undoing
those now.

* Having clubb thermodynamic level 1 below the surface is no longer
the stanard scenario!

---------

Co-authored-by: Gunther Huebler <huebler@uwm.edu>
The file was modified src/Benchmark_cases/astex_a209.F90 (diff)
The file was modified src/sounding.F90 (diff)
The file was modified src/Benchmark_cases/bomex.F90 (diff)
The file was modified src/SILHS/latin_hypercube_driver_module.F90 (diff)
The file was modified src/clubb_driver.F90 (diff)
The file was modified input_misc/tuner/error_rico.in (diff)
The file was modified src/bugsrad_driver.F90 (diff)
The file was modified src/CLUBB_core/advance_xp3_module.F90 (diff)
The file was modified input_misc/tuner/error_gabls2.in (diff)
The file was modified src/G_unit_test_types/spurious_source_test.F90 (diff)
The file was modified src/CLUBB_core/advance_xm_wpxp_module.F90 (diff)
The file was modified src/CLUBB_core/setup_clubb_pdf_params.F90 (diff)
The file was modified input_misc/tuner/error_dycoms2_rf02_ds.in (diff)
The file was modified src/hydrostatic_module.F90 (diff)
The file was modified src/coamps_microphys_driver_module.F90 (diff)
The file was modified input_misc/tuner/error_arm.in (diff)
The file was modified input_misc/tuner/error_atex.in (diff)
The file was modified src/Benchmark_cases/atex.F90 (diff)
The file was modified src/CLUBB_core/numerical_check.F90 (diff)
The file was modified src/CLUBB_core/pos_definite_module.F90 (diff)
The file was modified src/extended_atmosphere_module.F90 (diff)
The file was modified src/CLUBB_core/clip_explicit.F90 (diff)
The file was modified input/grid/48_level_gfdl/bomex.grd (diff)
The file was modified src/Benchmark_cases/mpace_b.F90 (diff)
The file was modified src/CLUBB_core/adg1_adg2_3d_luhar_pdf.F90 (diff)
The file was modified input_misc/tuner/error_four_cases.in (diff)
The file was modified src/CLUBB_core/mean_adv.F90 (diff)
The file was modified src/Morrison_microphys/microphysics.F90 (diff)
The file was modified src/silhs_category_variance_module.F90 (diff)
The file was modified input/grid/48_level_gfdl/dycoms2_rf02_ds.grd (diff)
The file was modified input_misc/tuner/error_arm_97.in (diff)
The file was modified input_misc/tuner/error_bomex.in (diff)
The file was modified src/CLUBB_core/advance_xp2_xpyp_module.F90 (diff)
The file was modified src/CLUBB_core/stats_type.F90 (diff)
The file was modified src/CLUBB_core/precipitation_fraction.F90 (diff)
The file was modified src/input_fields.F90 (diff)
The file was modified input/grid/48_level_gfdl/mpace_b.grd (diff)
The file was modified src/Benchmark_cases/rico.F90 (diff)
The file was modified src/CLUBB_core/clubb_api_module.F90 (diff)
The file was modified src/CLUBB_core/grid_class.F90 (diff)
The file was modified src/lh_microphys_driver_module.F90 (diff)
The file was modified input_misc/tuner/error_bomex_fire.in (diff)
The file was modified input_misc/tuner/error_fire.in (diff)
The file was modified src/Benchmark_cases/dycoms2_rf02.F90 (diff)
The file was modified src/CLUBB_core/parameters_tunable.F90 (diff)
The file was modified src/G_unit_test_types/w_up_in_cloud_tests.F90 (diff)
The file was modified src/Benchmark_cases/wangara.F90 (diff)
The file was modified src/CLUBB_core/pdf_closure_module.F90 (diff)
The file was modified src/advance_microphys_module.F90 (diff)
The file was modified input/grid/deep_convection_128lev_27km_zt_grid.grd (diff)
The file was modified src/SILHS/output_2D_samples_module.F90 (diff)
The file was modified src/SILHS/silhs_api_module.F90 (diff)
The file was modified src/Benchmark_cases/gabls2.F90 (diff)
The file was modified src/mixed_moment_PDF_integrals.F90 (diff)
The file was modified src/CLUBB_core/advance_clubb_core_module.F90 (diff)
The file was modified src/CLUBB_core/hydromet_pdf_parameter_module.F90 (diff)
The file was modified src/morrison_microphys_module.F90 (diff)
The file was modified input/grid/48_level_gfdl/rico.grd (diff)
The file was modified src/CLUBB_core/sigma_sqd_w_module.F90 (diff)
The file was modified src/Benchmark_cases/cobra.F90 (diff)
The file was modified input/grid/dycoms2_rf02_97lev_grid.grd (diff)
The file was modified src/CLUBB_core/Skx_module.F90 (diff)
The file was modified input_misc/tuner/error_nov11_altocu.in (diff)
The file was modified src/CLUBB_core/sponge_layer_damping.F90 (diff)
The file was modified src/CLUBB_core/advance_windm_edsclrm_module.F90 (diff)
The file was modified src/KK_microphys_module.F90 (diff)
The file was modified src/SILHS/transform_to_pdf_module.F90 (diff)
The file was modified src/Benchmark_cases/cloud_sed_module.F90 (diff)
The file was modified src/G_unit_test_types/pdf_parameter_tests.F90 (diff)
The file was modified src/gfdl_activation.F90 (diff)
The file was modified src/CLUBB_core/advance_helper_module.F90 (diff)
The file was modified src/CLUBB_core/diffusion.F90 (diff)
The file was modified src/ice_dfsn_module.F90 (diff)
The file was modified src/variables_radiation_module.F90 (diff)
The file was modified run_scripts/run_bindiff_all.py (diff)
The file was modified src/CLUBB_core/advance_wp2_wp3_module.F90 (diff)
The file was modified src/time_dependent_input.F90 (diff)
The file was modified src/CLUBB_core/turbulent_adv_pdf.F90 (diff)
The file was modified src/CLUBB_core/fill_holes.F90 (diff)
The file was modified input_misc/tuner/error_ticket_756.in (diff)
The file was modified src/Benchmark_cases/lba.F90 (diff)
The file was modified input/grid/48_level_gfdl/atex.grd (diff)
The file was modified src/Benchmark_cases/mpace_a.F90 (diff)
The file was modified input_misc/tuner/error_dycoms2_rf02_do.in (diff)
The file was modified input_misc/tuner/error_all.in (diff)
The file was modified input_misc/tuner/error_dycoms2_rf01.in (diff)
The file was modified input/grid/coarse_30lev_32km_zt_grid.grd (diff)
The file was modified input/grid/ecmwf_90lev_zt_grid.grd (diff)
The file was modified input_misc/tuner/error_lba.in (diff)
The file was modified src/SILHS/est_kessler_microphys_module.F90 (diff)
The file was modified src/microphys_driver.F90 (diff)
The file was modified src/simple_rad_module.F90 (diff)
The file was modified input_misc/tuner/error_gabls3_night.in (diff)
The file was modified src/Benchmark_cases/dycoms2_rf01.F90 (diff)
The file was modified src/estimate_scm_microphys_module.F90 (diff)
The file was modified src/CLUBB_core/mono_flux_limiter.F90 (diff)
The file was modified src/CLUBB_core/mixing_length.F90 (diff)
The file was modified input/grid/dycoms2_rf02_sam_zt_grid.grd (diff)
The file was modified src/CLUBB_core/calc_pressure.F90 (diff)
The file was modified input/grid/48_level_gfdl/arm_97.grd (diff)
The file was modified src/CLUBB_core/sfc_varnce_module.F90 (diff)
The file was modified src/CLUBB_core/stats_clubb_utilities.F90 (diff)
The file was modified input/grid/48_level_gfdl/dycoms2_rf01.grd (diff)
The file was modified src/Benchmark_cases/nov11.F90 (diff)
The file was modified src/clubb_tuner.F90 (diff)
The file was modified src/SILHS/lh_microphys_var_covar_module.F90 (diff)
Commit ae5433c83c88388bbe3c6d73f2869767d7877891 by bmg2
Updated the README to explain the grid notations gr%nzm and gr%nzt,
as well as added a note about the use of the generic nz number of
grid levels.
The file was modified README (diff)