Changes

Lots of small loop pushes

Small bug fix. This was not a bit changing bug, but could cause crashes if we compile with floating point trapping in a multicolumn run.

Small refactoring to mfl_xm_lhs, just removing need to pass in gr.

Final loop pushes needed before the grid change.

Replacing use of gr with nz

Replacing use of gr with nz

added -heap-arrays tag to FFlags on Makefile to fix jenkins testing error

Makefile came with a few other accidental changes, these have been corrected.

The big grid change. Converting gr from being an array of types containing 1D arrays, to a type containing 2D arrays. All cases BFB, cam multicolumn+silhs BFB, and cam multicolumn (no silhs) with backwards compatible settings BFB.

The big grid change. Converting gr from being an array of types containing 1D arrays, to a type containing 2D arrays. All cases BFB, cam multicolumn+silhs BFB, and cam multicolumn (no silhs) with backwards compatible settings BFB.

Adding column index to gr arrays, hard coding to 1 since sam only uses single column calls.

Removing the zt2zm interface from clubb_api and making the api calls just redirect to the grid class interface for it.

Pushing column loop into mean advection procedures.

Little loop push

Moving above/below index of grid weights to last dimension.

Hardcoding 1 into uses of gr arrays where single column versions of gr is used.

Pushing loop into fill_holes.

Forgot small change.

Making compatible with latest clubb change.

Making linear_interpolated_azt_2D and linear_interpolated_azm_2D subroutines just to avoid a needless data copy.

Making update_xp2_mc 2D and creating interface for 1D calls.

Fixing units in stats_zt_module.F90

This change will be BIT_CHANGING because it will affect the descriptions of variables in the netcdf files, but will not change answers.

Fixing a bug in mono_flux_limiter.F90. (#1026)

* Fixing a bug in mono_flux_limiter.F90.  This code was outputting "xm" instead of "wpxp" in several places.

This commit is BIT_CHANGING since it will affect the netcdf output for these variables, even though there
is no real change to answers.

* Renaming wpthlp_entermfl to wpthlp_enter_mfl which is more consistent with other variable names.

This is BIT_CHANGING due to changing the name and description in the output netcdf files.

* Renaming a few more instances of wpthlp_entermfl to wpthlp_enter_mfl.

Fixing a bug.

I changed the w_up_in_cloud code in 2 ways:

1)  The denominator term is now the "cloudy updraft" frac in each
PDF component, rather than just the PDF component cloud_frac.  This
is more consistent with the quantity being integrated.

2)  I also added a w_down_in_cloud for cloudy downdraft velocity.

Since these fields are not output as part of standard stats, this
commit will be bit-for-bit for the normal CLUBB output files.

I added stats output for w_down_in_cloud to all_stats.in.

Since this stat was not added to standard_stats.in, official results
will still be bit-for-bit.

Modified for the latest changes to CLUBB.

adding line to fix mpi configuration in micro drizzle test

I altered the w_up_in_cloud and w_down_in_cloud code so that a
thresholding is used similar to what is seen in the cloud fraction
and cloud water code.  If the PDF component mean of w more than
the maximum allow number of standard deviations away from 0, the
PDF component is either all-updrafty or all-downdrafty, and the
code avoids expensive computations where large magnitude values
can potentially be fed into ERF or EXP.

Of course, the results are not bit-for-bit with the previous version,
meaning that there are some situations where these thresholds come
into effect.  However, plots of all cases show no visible differences
in w_up_in_cloud and w_down_in_cloud.

Since w_up_in_cloud and w_down_in_cloud are not included in normal
stats output, this commit does not change the bit-for-bitness of
CLUBB code.

I have optimized the new w_up_in_cloud and w_down_in_cloud code by
only doing repeated operations one time and then saving them as a
local variable.

Since the numerical order of operations changes for the argument
to the EXP term, the results will differ at the level of numerical
round-off, making this revision not bit-for-bit with the last revision.
This only affects the w_up_in_cloud and w_down_in_cloud statistical
output variables, which in turn are not output as part of
standard_stats.in.

Refactored fill_holes_vertical to make GPUization simple. This is BIT_CHANGING, but results are bit-for-bit when using -O0 optimization, thus it is not answer changing. The first pass over each grid column will not parallelize well, the k-loop needs to be done in serial. Maximum parallelization has been exposed for the global hole-filling though, at the cost of occasionally doing unneccesary calculations. larson-group/clubb#972.

Removing fill_holes_multiplicative and replacing magic numbers with parameters from constants_clubb. larson-group/clubb#972

Moving vertical_avg and vertical_integral to advance_helper_module. larson-group/clubb#972

Moving vertical_avg and vertical_integral to advance_helper_module. larson-group/clubb#972

Making compatible with latest clubb change.

Removing elementalness from sat_vapor_press_liq and making interal procedures subroutines rather than functions to prevent unneccesary data copies. Doing the same for thlm2T_in_K since it is often used in conjunction with sat_vapor_press_liq. Bit-for-bit confirmed with O0 using all single column cases with or without l_diag_Lscale_from_tau, and with cam_coarse_res. larson-group/clubb#972

Adding comments.

Adding initial subroutine calc_ice_cloud_frac_component.

Cleaning up new subroutine calc_liquid_cloud_frac_component, and making sat_mixrat_ice a subroutine that works the same way as sat_mixrat_liq. larson-group/clubb#972

Comment update

Removing these paratheses is BIT_CHANGING since it modifies the order of operations, but allows for the multiplication and subraction to be done in parallel for complex pipelines.

Improvements. Netcdf output is now functional and we can detect errors with multiple columns even when the standard output is identical.

Updating column num input in config flags. larson-group/clubb#1033

Adding comments, setting to num_standalone_columns=1 by default, and preventing multicolumn output when l_stats=false. larson-group/clubb#1033

Improvements. Netcdf output is now functional and we can detect errors with multiple columns even when the standard output is identical.

Namelists need to be closed with a '/', and omitting it will break many tests.

Making save variables in the multicol code threadprivate, this fixes the openmp test. Also changing the way we name and save the multicol output, this allows the parallel run to run with multiple columns. I've reflected the changes in comments where relevant. larson-group/clubb#1033

Adds commented-out line that prevents the monotonic flux

limiter from acting on rtm.  The comments are included to remind us of how to do this in case we choose to do so in the future.

Updating monotonic flux limiter code to remove spikes. (#1038)

Also putting a commented out code change in advance_xm_wpxp,
in case shutting of the flux limiter entirely for rtm is preferred.

See https://github.com/NCAR/amwg_dev/discussions/134 for discussion.

This commit will be BIT_CHANGING although the differences are only
obvious in sensitive cases like arm_97.

Creating new flags to control monotonic flux limiter (#1039)

* Creating new flags to control monotonic flux limiter and the IF statement fix for the spikes found in CAM.

* Removing no-longer-necessary logical l_mono_flux_lim.

* Adding requested changes from PR (intent statements and editing error statement).

* Accidentally set flag to false. Changing to true.

Creating new flags to control monotonic flux limiter (#1039)

* Creating new flags to control monotonic flux limiter and the IF statement fix for the spikes found in CAM.

* Removing no-longer-necessary logical l_mono_flux_lim.

* Adding requested changes from PR (intent statements and editing error statement).

* Accidentally set flag to false. Changing to true.

Updating for changes to CLUBB.  See https://github.com/larson-group/clubb/pull/1039.

I am adding cloudy_updraft_frac and cloudy_downdraft_frac as
outputs to the calculate_w_up_in_cloud code.

These fields are non-interactive for the standard set of cases.
Thus, all cases are bit-for-bit identical.

However, for the w_up_in_cloud and w_down_in_cloud results themselves,
it is possible that results might not be bit-for-bit since
the location of the max(eps, ...) clipping in the denominator
is changed.  However, results should not be appreciable different.

Updating SAM-CLUBB for the latest changes to the call to
advance_clubb_core_api.

Minor adjustment to eliminate spikes in thlm tendencies from the monotonic flux limiter. (#1043)

See https://github.com/NCAR/amwg_dev/discussions/134.

This commit will be BIT_CHANGING. Differences are mainly noticeable in the more sensitive cases.

Making interface for matrix solvers.

Making tridag tridiag everywhere, making sol soln, and adding intent comments.

A different way of dealing with monotonic flux limiter spikes (#1046)

* A different way of dealing with monotonic flux limiter spikes in CAM, by increasing the value
of thl_tol_mfl.  ALso reverts the earlier fix.  BIT_CHANGING.

See https://github.com/NCAR/amwg_dev/discussions/134#discussioncomment-4165447.

* Reducing value of thl_tol_mfl from 0.5 to 0.2.

This will be BIT_CHANGING.

Steffens tuner fixer branch (#1041)

* Clubb ticket #1025: Implemented changes dealing with pdf_params%thl1/2 and wp2 floating point errors occurring in tuning runs. BIT_CHANGING
- Added command-line option -t/--tuner to compile.bash which enables the -DTUNER compiler flag.
- Added line to gfortran compilation config file to easily disable openMP
- Added a couple error messages and cleaned up some instances of error handling in src/error.F90, src/clubb_driver.F90, and src/CLUBB_core/advance_clubb_core_module.F90
- Added global constant wp2_max in src/CLUBB_core/constants_clubb.F90 which sets the upper bound for wp2
- In pdf_closure, added sanity checks for pdf_params%thl1/2 (>=190K, <=1000K)
- Added debug warning in src/CLUBB_core/advance_wp2_wp3_module.F90 when wp2 is clipped.
- Added wp2_sfc clipping in src/CLUBB_core/sfc_varnce_module.F90
- Added debug_level_check to NaN check in clubb_driver.F90
- Added mention of the new compiler option to the README

Making compatible with latest clubb changes.

Adding capability to change matrix solving method via clubb_config_flags.

Fixing small bug, need to pass _copy arrays to prevent lapack mangling the real ones.

Adding capability to change matrix solving method via clubb_config_flags.

Making GPU and CPU versions of the penta_lu solver the same as discussed in larson-group/clubb#1024.

Replacing $ with & in namelist definition to make consistent.

I added "smooth" max clipping for invrs_tau_shear, which is a variable
that is supposed to be positive definite, yet was obtaining negative
values at the model lower boundary owing to linear extension at the
boundaries as part of the linear interpolation call.

This change is BIT_CHANGING.

ADG1_pdf_driver subroutine port with OpenACC

Added OpenACC related flags in linux_x86_64_nvhpc_casper.bash
You can enable/disable OpenACC compilation using OPENACC=true/false.
Added OpenACC directives in ADG1_pdf_driver subroutine.

Replacing old elemental ADG1_w_closure with new GPUized one. Making G_unit tests work with new version. Also making mixt_frac_max_mag a scalar since it was only being used as such. larson-group/clubb#1049

I can now safely remove all the "ifdef E3SM" statements from CLUBB's
parameters_tunable.F90.  This code is now located in the clubb_intr.F90
portion of E3SM.

Pushing column loop into lapack wrap.

Restructuring and Porting of Compute_mixing_length subroutine(Phase 1) (#1052)

* Restructuring and Porting of Compute_mixing_length subroutine(Phase 1)

Restructure: The compute_mixing_length is one of the top most routine taking 35-50%
of the total time in a single timestep. The subroutine has been restructured to
push the i-loop further down to extract vectorization and parallelization.
The restructuring also involves introduction of sat_mixrat_liq_acc routines to
extract parallelism when called inside a OpenACC parallel region.

Porting: OpenACC directives are inserted to port the restructured compute_mixing_length
code on to the GPUs. This port is currently unoptimized and there is still room for improvement.

NOTE: Currently, l_sat_mixrat_lookup = false and saturation_formula = saturation_flatau
(Earthworks config options) case is supported on OpenACC build. Any other options works
on CPUs as usual. OpenACC declare create directives are inserted in model_flags and
constants_clubb, as these module variables are used inside the saturation routines.

* Added debug message about only supporting
l_sat_mixrat_lookup = false and saturation_formula = saturation_flatau
 on GPUs
Answers are Bit for Bit with arm-multicolumn case + nvhpc compiler.

* Changing CLUBB debug level 1 to 0 for the saturation formula support
running on GPUs

* Changing indentation to make gfortran happy, it wants ifdefs to start at the beginning of the line.

* Adding use statements for error checks and printouts, also making the errors set err_code to clubb_fatal_error.

Co-authored-by: huebler <huebler@uwm.edu>

Restructuring and Porting of Compute_mixing_length subroutine(Phase 2) (#1054)

* Restructuring and Porting of Compute_mixing_length subroutine(Phase 2)

Restructure: sat_mixrat_liq_2D_acc is being directly called instead of
calling the 1D version inside the column loop. Changing sat_mixrat_liq_2D_acc
to a subroutine from a function and adding output array, start_index as
additional argument. This is a workaround for passing sub-arrays. The OpenACC
doen't like the sub-arrays being passed and fails the validation.

Porting: OpenACC directives are added inside sat_mixrat_liq_2D_acc for porting

Validation: Answers are Bit for Bit with arm-multicolumn case + nvhpc compiler

* Fix for compilation issues

Issue 1: Missed out declaring 'start_index' while intergrating the
change

Issue 2: The use of error_code module and the procedures inside it
causes OpenACC compilation issues when run on the device.

* Removing the sat_mixrat_liq_acc and sat_mixrat_liq_2D_acc, making the normal sat_mixrat_liq work for all current use cases, and making the other versions of sat_mixrat_liq (bolton,gfdl,lookup) functional with OPENACC.

Co-authored-by: huebler <huebler@uwm.edu>

Breaking up column loop in mono_flux_limiter. This may not be the final form for GPUization, but it's definitely a start, no longer do we have to copy single column variables to multicolumn ones anywhere. (#1051)

Adding tridiag_lu solver (#1056)

* Adding tridiag_lu solver

* Changing low to lower in tridiag_lu and penta_lu, also fixing up some spacing.

* Making variables that represent matrix solutions more consistently named.

Adding OpenACC data directives for mixing length and adg routines

OpenACC structured data regions are added to optimize the data transfers
between CPU and GPU. These data regions will converted to unstrucutred
data region in the later optimization phase.
Results are bit for bit.

Fixing error causing GPU code not to run. Some variables to be copied were labelled as (ngrdcol,nz) when it should be (:ngrdcol,:nz). I've just removed the data length specifiers completely since they are not neccesary in general.

GPUizing sat_mixrat_ice_2D.

Pushing column loop into calculate_thvm

Removing update_pressure since it is no longer called anywhere in clubb or host models. The addition of this subroutine was discussed in larson-group/e3sm#6 and the removal of the call to it was discussed in larson-group/clubb#926.

Removing update_pressure from public list. This was causing compilation crashes. RESOLVED:8c7230fecb877d04fb129ef5e143e0993b4b29b1

GPUizing some helping procedures.

Removing sigma_sqd_w from the acc data copyout statement. This is a bug which was causing the code to crash when not using managed memory.

Porting pdf_closure subroutine with OpenACC (#1059)

* Porting pdf_closure subroutine with OpenACC

OpenACC directives are added to pdf_closure subroutine. The necessary
structured data region is also added for optimzing data movement across
kernels. There is opportunity to task parallelize using streams and
will be explored in the future.

Answers are bit for bit.

* Fixing the copyin directives

* Making work without managed memory.

Adding wp3_on_wp2_cfl_num to CLUBB's output stats.

Removing usage of gr from pdf_closure. It was only ever used for nz, which is now fed in directly.

Clubb ticket #1025: Implemented way to make esa tuner reproducible, h… (#1068)

* Clubb ticket #1025: Implemented way to make esa tuner reproducible, hid error output of optional diagnostic variables behind check, fixed parallelization issue with tuner, esa max_iters parameter is now in stats namelist, fixed issue with TUNER compiler directive, some small fixes.
- New namelist variables prescribed_rand_seed and l_use_prescribed_rand_seed determine if the esa tuner will use a random or a fixed value as random seed. Added descriptions to README.
- Added max_iters to stats namelist to make it more modifiable.
- Renamed stp_adjst_intercept_in and stp_adjst_slope_in to stp_adjst_shift_in and stp_adjst_factor_in, respectively, to better reflect their influence on step size.
- The error output in src/CLUBB_core/pdf_closure_module.F90 for the diagnostic variables wprtp2, wpthlp2, wprtpthlp, and rcp2 is now hidden behind existence checks for these variables. And a clarification was added to "#ifdef TUNER" directive.
- NetCDF file access caused the tuner to crash in parallel mode (-fopenmp flag in config file and multiple cases). Adding an $OMP CRITICAL structure around the call to stats_init in clubb_driver.F90 fixed that.
- Fixed compile/README. Config files are specified with the -c option.
- The -t option in run_scripts/run_tuner.bash interfered with the previous usage of the TUNER compiler directive. Renamed the old TUNER directive to NR_SP, a short for "numerical recipes, single precision". TUNER now is the option to "turn on" code changes required to run the tuner.

Gpu updates (#1070)

* Fixing bug. This was only triggered when l_input_fields=.true., which I am only testing because it needs to be true so that I can test ADG2_driver.

* Removing usage of gr from pdf_closure. It was only ever used for nz, which is now fed in directly.

* Making openacc statements more consistent. Ensuring all statments on double loops have specified gang and vector, and that all parallel loops have an end parallel loop statment at the end of them. Everything BFB on CPUs and GPUs.

* Pushing acc data region to outermost parts of mixing_length.

* Removing pdf_implicit_coefs_terms from acc copyin and copyout. It is only used when iiPDF_type == iiPDF_new .or. iiPDF_type == iiPDF_new_hybrid, so we do not need to do any copying with it. The inclusion of it also caused the data statement to copy unallocated arrays, which are just garbage pointers, and that was causing random occasional crashes (either segfaults or gpu out of memory).

* The update device clauses for return variables seems to only be requried for arrays contained in types. See https://github.com/larson-group/clubb/issues/1049\#issuecomment-1440624778

* Moving acc end data to end of pdf_closure. This reuqired removing any conditional return statements that appear before the final return, since we're not allowed to branch out of an acc region early. I also moved a large printout statement outside of a loop. The only reason it was in the loop to begin with was because pdf_params used to be an array of types, but now is a type of arrays, allowing us to print the full arrays directly.

* Making loop an acc loop. If we weren't outputting w_[up/down]_in_cloud (iw_up_in_cloud <= 0 .or. iw_down_in_cloud <= 0, then these arrays were only being zerod out on the CPU and would've getting overwritten by the uninitialized GPU data at the end of the data statement. This change causes the arrays to get correctly zerod out on the GPU when we need.

* Update VariableGroupNondimMoments.py

Fixed a typo

* Merging new changes from master

* Removing need for -gpu=deepcopy, pushing some acc data statements up call tree, and replacing some acc data statements with acc delare statements so that return statements can be added back in.

* Adding back an acc loop that was accidentally removed during a merge.

---------

Co-authored-by: Brian Griffin <31553422+bmg929@users.noreply.github.com>

Implements Thomas Toniazzo's bug fix for spike removal

in the monotonic flux limiter.  See

https://github.com/larson-group/clubb_release/pull/11/files

This code change is
expected to be BIT_CHANGING for cases in which
`l_predict_upwp_vpwp = T`, `l_mono_flux_lim_um = T`
or `l_mono_flux_lim_vm = T`, and the monotonic
flux limiter is triggered.

This bug fix prevents non-conservation of momentum
when the vertical integral of either of the wind components `um` or `vm` is negative.

Making 'a_const' into a CLUBB tunable parameter. (#1072)

* Converting the sfc_varnce_module.F90 parameter 'a_const' into a CLUBB tunable parameter.

Results should be bit-for-bit since it will be set to 1.8 and with l_vary_convect_depth=F
this is the standard value.

Some discussion in ticket #994.

* Completing requested changes, see PR #1072.

Making 'a_const' into a CLUBB tunable parameter. (#1072)

* Converting the sfc_varnce_module.F90 parameter 'a_const' into a CLUBB tunable parameter.

Results should be bit-for-bit since it will be set to 1.8 and with l_vary_convect_depth=F
this is the standard value.

Some discussion in ticket #994.

* Completing requested changes, see PR #1072.

Updating for changes to CLUBB.  See PR #1072.

Pdf closure driver gpuization (#1071)

* GPUizing pdf_closure_driver, testing still needed, bug likely.

* Making data movement better. Also fixed some random things. Still need to return returns.

* Adding back a couple return statments. Moving a comment where it should be. GPUizing a missed loop.

* Making arguements list satisfy standards.

* Making arguements list satisfy standards. Also adding max(1,hydrometdim) to hydromet inputs.

* Capitalizing K

* Making the error case in compute_cloud_cover behavior a bit differently. This allows the error to be detected immediately by GPU runs as well.

* Calling compute_cloud_cover only needs to be done if l_use_cloud_cover = .true.

Moving compute_cloud_cover outside of if ( l_use_cloud_cover ) then statement, the cloud_cover and rcm_in_layer variables they compute aren't output in clubb_standalone, but are in cam, causing cam bit diff tests to break.

Adding Skthl_zm to the update host list, I missed this in the last PR. I noticed this by comparing results with and without managed memory, now I've checked BFBness with arm, mpace_b, mc3e, and gabls2.

Small GPU fixes (#1076)

* Fixing small things that I caught by adding the default(present) onto acc loops.

* Moving default(present) to the end because it looks nicer there.

Code changes to implement modificiations on wp3 clippings
This commit contains code changes to implement modifications of skewness
clippings on wp3 in src/CLUBB_core/clip_explicit.F90. The default method attempts
to apply smaller (larger) clippings below (above) 100m AGL level, which
can cause a discontinuities around 100m AGL level. This clippings is
found to trigger sawthooth oscillations in wp3 when linear diffusion is
used. Such swathooth oscillations are eleminated if a smoothed Heaviside
function is introduced to obtain a smooth transition of clippings at
around 100m AGL level. The changes is necessary to obtain the first
order convergence in CLUBB-SCM when lienar diffusion is used.

Previously committed by Shixuan.

Code changes to implement modificiations on wp3 clippings
This commit contains code changes to implement modifications of skewness
clippings on wp3 in src/CLUBB_core/clip_explicit.F90. The default method attempts
to apply smaller (larger) clippings below (above) 100m AGL level, which
can cause a discontinuities around 100m AGL level. This clippings is
found to trigger sawthooth oscillations in wp3 when linear diffusion is
used. Such swathooth oscillations are eleminated if a smoothed Heaviside
function is introduced to obtain a smooth transition of clippings at
around 100m AGL level. The changes is necessary to obtain the first
order convergence in CLUBB-SCM when lienar diffusion is used.

Previously committed by Shixuan.

This commit contains code changes to implement modifications on limiters in three places:

1. remove the limiters in denominator of equation for
   brunt_vaisala_freq_sqd_smth, which affects the computed
   eddy dissipation time scale in turbulent fluxes (wpxp).
   (in mixing_length.F90)

2. reduce the threshold values of limiters in the equation
   for richardson number (sqrt_Ri_zm)
   (in mixing_length.F90)

3. introduce the smoothed max/min function for limiters in
   equation of Cx_fnc_Richardson.
   (in advance_helper_module.F90).

After the modification, we also apply a zt2m(zm2zt) smoothing
on the calculated quantities. These modifications are found to
be benificial for improving solution convergence in CLUBB-SCM

The code changes are controlled by a newly introduced flag named
"l_modify_limiters_for_cnvg_test", which is set to .false. (meaning that
the modificaitons on limiters is turned off) by default.

Originally committed by Shixuan.

This commit contains code changes to implement modifications on limiters in three places:

1. remove the limiters in denominator of equation for
   brunt_vaisala_freq_sqd_smth, which affects the computed
   eddy dissipation time scale in turbulent fluxes (wpxp).
   (in mixing_length.F90)

2. reduce the threshold values of limiters in the equation
   for richardson number (sqrt_Ri_zm)
   (in mixing_length.F90)

3. introduce the smoothed max/min function for limiters in
   equation of Cx_fnc_Richardson.
   (in advance_helper_module.F90).

After the modification, we also apply a zt2m(zm2zt) smoothing
on the calculated quantities. These modifications are found to
be benificial for improving solution convergence in CLUBB-SCM

The code changes are controlled by a newly introduced flag named
"l_modify_limiters_for_cnvg_test", which is set to .false. (meaning that
the modificaitons on limiters is turned off) by default.

Originally committed by Shixuan.

Updated for recent changes to CLUBB.

changing tunable_parameters

changing tunable_parameters

changing tunable_parameters

Advance xm wpxp gpuization (#1077)

* Initial commit for GPUizing advance_xm_wpxp.

* GPUizing fill_holes_vertical

* Adding Skthl_zm to the update host list, I missed this in the last PR. I noticed this by comparing results with and without managed memory, now I've checked BFBness with arm, mpace_b, mc3e, and gabls2.

* Small GPU fixes (#1076)

* Fixing small things that I caught by adding the default(present) onto acc loops.

* Moving default(present) to the end because it looks nicer there.

* Adding default(present) to all acc loop statements. Also adding azt to a copyin statement, which was missed previously. All BFB.

* Incemental update, not well tested yet.

* Removing some copies and making the sclr_dim change.

* Fixing a bug that only seemed detectable with astex_a209. We need to pass only single arrays to functions, calling ddzt( nz, ngrdcol, gr, rho_ds_zt * K_zt_nu ) was resulting in rho_ds_zt * K_zt_nu being evluated on the CPU, but the values were only valid on the GPU. So we need to evaluate that expression on the GPU, save it into an array (currently K_zt_nu_tmp), then pass that to ddzt.

* GPUizing calc_turb_adv_range

* GPUizing mono_flux_limiter

* Cleaning up data statments and a couple other things.

* Updated for some different options.

* More updates needed for various options.

* Reverting accidental flag change

* Should be the final changes, all options tested now.

* Replacing some comments in monoflux limiter, and also modifying it to make it BFB on CPUs. Also changing incorrect error conditions on tridiag.

* Adding max_x_allowable to update host statement, missed previous.

* Properly naming tmp variables and variables calculated from ddzt and ddzm start with ddzt_ and ddzm_.

* Replacing constants with named ones from constants_clubb.

* Replacing hard coded numbers in lhs variables representing the number of bands they contain with fortran parameters.

Small tweaks to fix some GPU bugs. Some variables were uninitialized on the CPU while we were saving them. This could only have been caught by comparing consecutive runs and checking _zt and _zm files, even then few cases were having problems.

Fixing a labelling error in redirect_interpolated_azt_2D and similar procedures, since this interpolates to zt the input should be zm. I think this was my fault, so I cleaned all the zt2zm and zm2zt things up to make it a little nicer. Also ordered the routines _k _1D _2D to make it easier to jump around, it was a bit confusing as they were out of order and the typo really made it hard.

Making it so sclr_tol is set to 0 before the specified sclr_tol_in. This is so that it is initialized to 0 in the case that sclr_dim = 0, since now we are setting it to have a minimum allocation size of 1 and would otherwise have a garbage value. This is what broke the clubb_openmp_gfortran_test.

Lscale GPUization (#1079)

* Making 2 new functions zm2zt2zm and zt2zm2zt to handle smoothing by interpolation. Replaced the spots in clubb I know that uses this to smooth things. This is just a nice to have and could allow for easy optimizations in the future by inlining the interpolations. All cases BFB on CPU and GPU, checked all relevant options too.

* GPUizing diagnose_Lscale_from_tau

* Removing some unused variables.

* Moving acc data statements from calc_Lscale_directly up to advance_clubb_core.

* Removing an unused variable.

* GPUizing the l_smooth_min_max option.

* GPUizing l_avg_Lscale

* Changes to variable names to avoid gross long names only used once.

gfortran does not like intent(out) specifiers for functions.

GPUizing Various Small Routines (#1080)

* GPUizing compute_Cx_fnc_Richardson

* GPUizing term_wp2_splat and term_wp3_splat

* GPUizing xp3_LG_2005_ansatz

* GPUizing calculate_thvm

* GPUizing pvertinterp even though I don't think we care about the l_do_expldiff_rtm_thlm flag

* Fixing bug. Setting l_do_expldiff_rtm_thlm causes us to use edsclrm, so we need to also ensure that edsclrm > 1 (1 because it uses a edsclr_dim-1 index)

* Pushing some data statements up the call tree.

This commit is a commit that changes absolutely nothing. It is meant to trigger a change in the git update scripts, so that I can start the commit message logging in the autocommit updates larson-group/sys_admin#797

This commit is a commit that changes absolutely nothing. It is meant to trigger a change in the git update scripts, so that I can start the commit message logging in the autocommit updates larson-group/sys_admin#797

this is another commit that cahnges nothing that will trigger the gitUpdate scripts

adding an update that changes nothing and is just a test for the autoupdate script

Making CLUBB's splatting scheme implicit and smoother (#1075)

* Introducing rough implicit splatting scheme into CLUBB.  Not yet implicit for wp3.

* Implementing implicit splatting for wp3.

* Further improvements to implicit splatting code, including

- smoothing of the BV freq used for splatting
- adding a new variable 'bv_freq_sqd_splat' to stats
- add the new variable to pyplotgen standard output

See PR #1075 and issue #994.

* Some cleanup.

* More cleanup, and adding comments.

* More comments.

* Updating implicit splatting scheme.

* Updates to resolve PR requests.

Updates associated with implicit splatting will be BIT_CHANGING.

Calc sfc varnce GPUization (#1081)

* Initial loop push for calc_sfc_varnce

* BIT_CHANGING:3b086a40085284aa49c71d32c001d20153a8ddb4 the last commit is bit changing for only some cases and only when using higher than -02 optimization. uf min seems to be the first calculation that starting to differ bitwise. Using the check_multicol script confirms the differences are small.

* Adding a tweak to surface values in the extra columns. This helped me check calc_sfc_varance, since we were not changing any arrays that would've affected calculations there.

* Small optimization, making wstar and ustar2 scalars.

* GPUizing calc_sfc_varnce

* Removing conditional around some stats calls. Now we will always save sfc values to stats, because this will change stats files when gr%zm(i,1) > sfc_elevation, this is potentially BIT_CHANGING.

* Merging with latest clubb changes and making work on GPUs again.

This contained 2 commits that are BIT_CHANGING in some situations.

Reducing C_wp2_splat to 0.25 (from 2.0) for new implicit splatting code.

Also fixing bug that shows up when l_brunt_vaisala_freq_moist = T.  This
will hopefully help to get some failing CAM tests running again.

See #1074 and PR #1075 for info.

This is BIT_CHANGING for all cases due to the new tunable parameter value.

Reducing C_wp2_splat to 0.25 (from 2.0) for new implicit splatting code.

Also fixing bug that shows up when l_brunt_vaisala_freq_moist = T.  This
will hopefully help to get some failing CAM tests running again.

See #1074 and PR #1075 for info.

This is BIT_CHANGING for all cases due to the new tunable parameter value.

Removed Space From clip_explicit

This is another commit that changes absolutely nothing, and only exists to test autocommit changes

making changes to trigger autoudpate for test, these commits do nothing

here is a body to the commit as well

making changes to trigger autoudpate for test, these commits do nothing

here is a body to the commit as well

Here is another commit that changes nothing

Here is the body

Here is another commit that changes nothing

Here is the body

change to trigger updates

this is the last batch

change to calc pressure to trigger autoupdate

body

GPUizing Lscale_width_vert_avg. Loops have been restructured for simplicity, and algorithm has a different starting value to avoid k dependency. Results are BFB. (#1083)

GPUizing most of advance_clubb_core (#1084)

advance_wp2_wp3 with explicitly managed memory  (#1085)

* GPUizing most of advance_clubb_core

* GPUizing most of advance_wp2_wp3 based off Supreeths openacc branch

* Cleanup and small fixes.

* Converting data statement to declare statement to return the return.

* Small changes to prevent data copies

advance_xp2_xpyp with explicitly managed memory (#1086)

* GPUizing most of advance_clubb_core

* GPUizing most of advance_wp2_wp3 based off Supreeths openacc branch

* Cleanup and small fixes.

* Converting data statement to declare statement to return the return.

* Small changes to prevent data copies

* Initial advance_xp2_xpyp GPUization, adapted from Supreeth's branch, clubb_openacc, to work without managed memory.

* Small cleanup

* Tweaks to reduce data copies

* Small change

* More GPUization and some cleanup

* Final little fixes

* Removing magic logicals

advance_windm_edsclrm with explicitly managed memory (#1087)

* Initial commit for advance_windm_edsclrm adapted from Supreeth's branch

* Final fixes

* Accidentally undid fix

* Returning returns

Moving data statements to outter most parts of clubb and little fix in advance_wp2_wp3 (#1088)

Data movement improvement (#1089)

* Various reductions in data movement

* Bug fix

* Optimizations to fill_holes, this involves precalculating a multiplication an a division, so this is BIT_CHANGING.

* Missed a small part

* Small fix

CLUBB ticket #1069: (#1092)

- Added tunable parameter bv_efold
- Removed extraneous use statement in src/CLUBB_core/advance_xm_wpxp_module.F90
- Removed obsolete flag l_update_pressure from input/tunable_parameters_Lscale/configurable_model_flags.in
- Changed tmp_calc_min -> tmp_calc in src/CLUBB_core/advance_helper_module.F90

CLUBB ticket #1069: (#1092)

- Added tunable parameter bv_efold
- Removed extraneous use statement in src/CLUBB_core/advance_xm_wpxp_module.F90
- Removed obsolete flag l_update_pressure from input/tunable_parameters_Lscale/configurable_model_flags.in
- Changed tmp_calc_min -> tmp_calc in src/CLUBB_core/advance_helper_module.F90

Added bv_efold changes for CLUBB API calls

Changing "Fnc" to "fnc" for consistency and easy search.

Memory usage optimizations (#1091)

* Fixing merge conflict

* Fixing a bug.

* Restoring correct default flag value

* Undoing changes that push stats variables into pdf_closure. That change may be included in a small future refactoring.

* Making same sclr_dim changes to hydromet_dim

Making compatible with latest clubb changes.

Openacc tweaks and cleanup (#1097)

* Chaning acc declare statements to acc enter data statement

* Making acc statements more consistent

* Making lapack useable while using openacc. Lapack is still run on the CPU

* Updating setup_clubb_core to now accept clubb_config_flags as an input, and adding warning in case clubb is running with lapack but was compiled with openacc.

Loop to `nz-1` instead of `nz` because upper boundary condition

is set in the lines immediately below.

Openacc tweaks and cleanup 2 (#1099)

* Making all end parallel directives specify end loop

* Replacing last acc declare in a procedure with acc enter/exit data commands

* Removing pure declarations, turns out they dont really improve performance and openmp isn't allowed within them.

* Adding explicit directives to penta_lu and tridiag_lu, as opposed to the previously used kernels directive. Also splitting up a loop to improve GPU performance.

* Splitting up a couple loops for performance reasons.

* Missed one somehow

Bit changing changes (#1104)

* Moving functions called inside loops to their own 2D subroutines. This is for performance, but is also BIT_CHANGING.

* Correct range of values to calculate for term_dp1_lhs

* Comments and cleanup

* Slight change to Skx_func, this is mathematically equivalent but BIT_CHANGING. This is faster on GPUs, and doesn't seem to have a signficiant impact on CPU performance.

* Introducing wp_coef and wp_coef_zt to reduce needed computations. This changes order of operations, so it is BIT_CHANGING.

Adding wp_coef and wp_coef_zt to acc data statements.

Bv efold bit changing (#1095)

BIT_CHANGING! Added e-folding code for mixed Brunt Vaisala frequency - CLUBB ticket #1069
- Included usage for bv_efold in src/CLUBB_core/advance_helper_module.F90
- Added intent(out) for correction_stability in subroutine calc_correction_stability
- Removed brunt_vaisala_freq_sqd_plus
- Idiot proofed interaction between l_diag_Lscale_from_tau and l_use_invrs_tau_N2_iso
- Moved stat_update_var(invrs_tau_wp3_zm) out of if ( l_diag_Lscale_from_tau )
- Fixed some typos in src/CLUBB_core/stats_zt_module.F90
- Added comment about to input_misc/tuner/README about t_variables adn input/stats/tuning_stats.in

Fixed some oversights for the bv_efold commits and increased string length for tuning field names for invrs_tau variables - CLUBB ticket #1069 (#1106)

This commit is NOT bit changing
- Removed remaining mentions of brunt_vaisala_freq_sqd_plus
- Added assignment for invrs_tau_wp3_zm in Lscale mode in advance_clubb_core
- Increased length of string variables hoc_v, les_v, and t_variables in src/error.F90 to accomodate tuning for invrs_tau fields

BIT_CHANGING! brunt_vaisala_freq_sqd_mixed now replaces brunt_vaisala_freq_sqd for later calculations - CLUBB ticket #1102

Changes in src/CLUBB_core/mixing_length.F90:
- brunt_vaisala_freq_sqd_mixed is now used to calculate brunt_vaisala_freq_sqd_smth and brunt_vaisala_freq_sqd_clipped instead of brunt_vaisala_freq_sqd
- Removed 0.1 factor for bkgnd term in calculation of invrs_tau_xp2_zm
- Commented out clipping of invrs_tau_xp2_zm to 0.003 and lower out of cloud

BIT_CHANGING for lscale setup. Updated configurable_model_flags.in and model_flags.F90 - CLUBB ticket #1102 (#1108)

- Added new flags to inut/tunable_parameters_Lscale/configurable_model_flags.in
- Changed default setting for l_use_tke_in_wp3_pr_turb_term in model_flags.F90 from false to true

Likely BIT_CHANGING because of added output fields. Added code outputting C6_term and different Brunt Vaisala frequencies to disk - CLUBB ticket #1102 (#1109)

- Added C6_term and bv_freq fields to all_stats.F90 and standard_stats.F90
- Added new fields to VariableGroupTaus.py in pyplotgen
- Added calls to stat_update_var to advance_clubb_core_module.F90
- Modified stats_variables.F90 and stats_zm_module.F90 accordingly

Changed sqrt_Ri_zm to Ri_zm in preparation of changing the exponent for calculation of invrs_tau_wpxp_zm in mixing_length.F90 - CLUBB ticket #1102 (#1110)

BIT_CHANGING! Changed sqrt_Ri_zm to Ri_zm in preparation of changing the exponent for calculation of invrs_tau_wpxp_zm in mixing_length.F90 - CLUBB ticket #1102
Changes in input/stats/all_stats.in, src/CLUBB_core/advance_clubb_core_module.F90, src/CLUBB_core/mixing_length.F90, src/CLUBB_core/stats_variables.F90, and src/CLUBB_core/stats_zm_module.F90
Ri_zm was added to input/stats/standard_stats.in

BIT_CHANGING! Removed Richardson_num and all corresponding code - CLUBB ticket #1102 (#1111)

This commit is BIT_CHANGING because:
1. the output field Richardson_num was removed.
2. Since the flag l_use_shear_turb_freq_sqd was removed, the variable shear_sqd is now always written to disk. That means previously, the outputted field shear_sqd was 0 everywhere whereas now it is not anymore. The calculation of shear_sqd within CLUBB did not change!
- Removed mentions in input/stats/all_stats.in and standards_stats.in
- Removed mentions in src/CLUBB_core/stats_variables.F90 and stats_zm_module.F90
- In src/CLUBB_core/advance_helper_module.F90 removed mentions to [i]Richardson_num[_clipped], as well as l_use_shear_turb_freq_sqd, l_Richardson_vert_avg, and turb_freq_sqd sinc those are only used for calculation of Richardson_num.

Tunable Richardson number exponent - CLUBB ticket #1102 (#1114)

BIT_CHANGING! Added the exponent to the Richardson number in the calculation of the invrs_tau_wpxp threshold term in src/CLUBB_core/mixing_length.F90  to the list of tunable parameters.
This is bit changing due to the difference in Fortran between taking the sqrt and the .5th power.

Tunable Richardson number exponent - CLUBB ticket #1102 (#1114)

BIT_CHANGING! Added the exponent to the Richardson number in the calculation of the invrs_tau_wpxp threshold term in src/CLUBB_core/mixing_length.F90  to the list of tunable parameters.
This is bit changing due to the difference in Fortran between taking the sqrt and the .5th power.

Raw-copy updating SRC/CLUBB

Raw-copy updating CLUBB_PARAMETERS

Deletes svn id tag, will trigger host-model nightly tests.

Raw-copy updating SRC/CLUBB

Update clubb_sgs.F90

Added CLUBB parameter wpxp_Ri_exp

Add library calls to allow SAM to compile with new icx intel compiler (#154)

Remove svn id tag in order to trigger nightly tests.

Removed 3rd svn id tag in order to trigger nightly tests

Finally removing scratch variables. (#1116)

* Finally removing scratch variables.

* Zeroing out below ground levels to prevent NaN in stats.

Latest batch of non-bit changing code changes - CLUBB ticket #1115 (#1119)

- Added new folder input/tunable_parameters_tauMatchLscale/ for latest taus code setup reproducing the master Lscale output.
- Changes in src/CLUBB_core_mixing_length.F90:
   - Renamed ddzt_umvm -> ddzt_umvm_sqd
             ddzt_umvm_clipped -> ddzt_umvm_sqd_clipped
             sqrt_ddzt_umvm -> norm_ddzt_umvm
             smooth_sqrt_ddzt_umvm -> smooth_norm_ddzt_umvm
             Ri_zm_clipped -> Ri_zm_smooth in some places
   - Made some variable replacements for consistency's sake
- Fixed path of log files in run_scripts/run_bindiff_all.py

BIT_CHANGING! Fixed some issues with wpxp_Ri_exp, changed z_displace value - CLUBB ticket #1115 (#1120)

- Removed commented lines from tunable_parameters.in file in input/tunable_parameters_tauMatchLscale
- Increased z_displace value from 10 to 25. This value was chosen so that the taus code with the parameter and flag settings in input/tunable_parameters_tauMatchLscale match the master Lscale output below clouds
- Fixed some issues with the new parameter wpxp_Ri_exp:
   - Moved new exponentiation to a place so that it emulates the previous version with sqrt(Ri_zm)
   - Adjusted code for l_smooth_min_max=true to match the default version

Updated CLUBB so that adj_low_res_nu (which can be called on its own
or called through calling setup_parameters) no longer uses
momentum_heights and thermodynamic_heights, but rather makes use
of gr%zm and gr%zt, which are always set and available by the
time the calls are made.

Updated the CLUBB interface for recent changes to CLUBB.

Removing begin_height and end_height as outputs from setup_grid. They
are only necessary as local variables inside of setup_grid.

Updated for recent changes to CLUBB.

Z displace tunable (#1122)

CLUBB ticket #1115
Added the previously hard-coded z_displace as a tunable parameter.