Changes

Gunthers branch (#996)

* Pushing column loop into xm_wpxp_clipping_and_stats and monotonic_turbulent_flux_limit. This essentially completes advance_xm_wpxp for now. larson-group/clubb#972

* Replacing the i loop iterator for scalars with sclr.

* Pushing loop into advance_xp2_wpxp.

* Moving the lhs_dp1 calculation to outside of xp2_xpyp_lhs. This gives us more flexibility, since we want to pass lhs_dp1 into xp2_xpyp_implicit_stats.

* Moving the lmm_stepping and stats calls to immediately after calls to xp2_xpyp_solve. This is because xp2_xpyp_implicit_stats saves things that are saved by scratch variables, and these scratch variables are set by the lhs and rhs setup calls. So for the multiple lhs routine, we need to save the scratch variables immediately after these calls before overwriting them by calling lhs and rhs again for a different variable. Since the stats also saves the variable were solving for, we also have to move the lmm_stepping to before this call.

* Removing need for stats scratch variables by passing lhs terms to save through agument lists. This makes everything better in every way.

* Adding slicing for some lhs arrays being saved in stats. This was causing bit changing in a few lhs terms when run in multicolumn.

* Changing names of dp1 variables for up2 and vp2. Surrounding where they're set to be within a l_stats_samp.

* Breaking up column loop in advance_xp2_xpyp and pushing column loop into calc_xp2_xpyp_ta_terms. larson-group/cam#972

* Breaking up column loop in calc_xp2_xpyp_ta_terms.

* Pushing loop into solve_xp2_xpyp_with_single_lhs.

* Fixing mangled merge.

* Pushing loops down.

* Pushing loops and small cleanup.

Various simple changes to improve performance. All bit-for-bit.

Pushing loop into calc_stability_correction.

Small bug fix.

Another similar small bug fix.

Adding in comments to the Jenkinsfiles to clarify advanced syntax usage and commands with unclear purpose.

Gunthers branch (#999)

* Reworking the flow to make it good.

* Cleaning up.

* Pushing loop into advance_xp2_wpxp.

* Forgot to commit advance_clubb_core.

* Slight improvement to logic of lhs_diff_zm_crank calculation, and fixing stats but present when lhs_diff_zm_crank=true.

* Pushing lots of loops down.

* Changing dimension ordering.

* Cleaning up. This completes advance_wp2_wp3. larson-group/clubb#972

* Changing capitalization of intents.

Renamed smth_range to heaviside_smth_range

Added tot_vartn_normlzd statistics. Renamed sclr in advance_helper_module to scalar to be more consistent with clubb naming schemes.

Implemented three further normalized variation stats. Included a (most likely temporary) check because in a few cases, denominator for normalization would be 0.

Changed priorities of total normalized variation stats, included error handling in total normalized variation stats.

Renamed pdf_output_filename. Added grid_level constant to avoid magic numbers in stats_update_var_pt calls.

Gunthers branch 2 (#1002)

* Pushing loop down through advance_windm. Other files needed to be touched because they were using a simple function, xpwp_func, which has been replaced in relevenant places by the few lines of code it takes to do.

* Cleaned up intents

* Moving xpwp calcuations back to procedure, but making procedure a 1/2D interface in advance helper. Other various small tweaks as well.

Bugfix for merge #1000. stderr was not imported in stat_clubb_utilities.F90, causing compilation to fail.

Renaming the new case from SHEAR_DRIVEN to NEUTRAL, and associated mentions in files.

See https://github.com/larson-group/clubb/issues/991.

Added ability to apply smooth min max functions in mixing_length.F90

Updated deprecated documentation of smooth min and max functions in advance_helper_module.F90. https://github.com/larson-group/clubb/issues/965

Checked whether results really are identical even with round-off when we have l_smooth_min_max=T and smooth_min_max_mag=zero. Next commit rolls some of these changes back for merge into master. https://github.com/larson-group/clubb/issues/965

Added test cases to smooth_min_max_tests.F90 and updated documentation.

removed SAM_CLUBB_standards.tex, SAM_CLUBB.tex, SAM_code_notes.tex, and SAM6.10_guide.doc larson-group/sys_admin#622

COrrecting a couple of omissions from when I was renaming the shear_driven case to "neutral".

See https://github.com/larson-group/clubb/issues/991.

Constructing rcm within SILHS (as rcm_pdf) (#1011)

* Constructing rcm within SILHS (as rcm_pdf) to avoid having to pass rcm directly to SILHS. Relevant to possibly moving PDF call placement
into the "post" position in CAM.

See https://github.com/ESCOMP/CAM/issues/582.

* Reconstructing rcm inside of setup_pdf_parameters instead of passing it through
the argument lists.  Might help NCAR move the PDF call placement to "post" position
in CAM.

See https://github.com/larson-group/clubb/issues/997.

* Adding descriptive comment.

Constructing rcm within SILHS (as rcm_pdf) (#1011)

* Constructing rcm within SILHS (as rcm_pdf) to avoid having to pass rcm directly to SILHS. Relevant to possibly moving PDF call placement
into the "post" position in CAM.

See https://github.com/ESCOMP/CAM/issues/582.

* Reconstructing rcm inside of setup_pdf_parameters instead of passing it through
the argument lists.  Might help NCAR move the PDF call placement to "post" position
in CAM.

See https://github.com/larson-group/clubb/issues/997.

* Adding descriptive comment.

Updating for changes to CLUBB.  See https://github.com/larson-group/clubb/commit/e4f125ba067ba8083f917e0e06b6b2398483d3e4.

Almost final loop pushes (#1012)

* Pushing loop into clip_rcm

* Pushing column loop into trapezoid routines.

* Pushing column loop into clip_covar routines.

* Pushing column loop into clip_variance and clip_skewness.

* Replacing sgn_turbulent_velocity with built in fortran sign

* Removing gr from argument list of stats calls, replacing with just nz

* Replacing some usages of gr with just the number of vertical levels

* Replacing some usages of gr with just the number of vertical levels again

* Missed a couple files to commit

* Pushing some loops and removing need for gr to be passed into some routines.

* Pushing loop into advance_xp3

* Pushing loop into mixing length

* Some more loop pushing

* Pushing some loops into mono_flux limit, skipping the lhs routine since it's a bit tricky at the moment, will do after change to grid class.

* Removing gr from argument list of stats accumulate

* Precalculating zt2zm so that we can use 2d version

* Precalculating um/vm ddzt so that we can use 2d version

* Precalculating things to use 2d version of zt2zm

* Making 2d version of cubic interp for grid class

* Small modifications to sponge damping code argument lists

* Renaming some variables

Making compatible with latest clubb change.

Making pdf_implicit_coefs_terms a type containing 2D arrays. (#1014)

The G_unit tests never allocated pdf_implicit_coefs_terms. It's unclear to me how these were working before, but it seems that something about making these 2D allocatable arrays exposed the bug.

Making zt2zm calls with pdf_implicit_coefs_terms use the 2D version.

Removing setup_grid and setup_parameters functionality from setup_clubb_core. This is beacuse in host models the required grid information may not be known during the setup process, resulting in dummy arguments for setup_clubb_core and the grid and paramters being setup during runtime anyway. Now, in the host models we can call these subroutines immediately after setup_clubb_core to maintain identical functionality (sam, wrf), and in others we can wait to call these until the main timestepping procedure (cam,e3sm).

Making compatible with latest clubb change.

Making 2D versions of setup_grid and setup_parameters.

Making lmin a scalar again, twas a mistake to make it an array, it can only take on one value.

Making nu_vert_res_dep a type containing arrays, as opposed to being an array of types.

Lots of small loop pushes

Small bug fix. This was not a bit changing bug, but could cause crashes if we compile with floating point trapping in a multicolumn run.

Small refactoring to mfl_xm_lhs, just removing need to pass in gr.

Final loop pushes needed before the grid change.

Replacing use of gr with nz

Replacing use of gr with nz

added -heap-arrays tag to FFlags on Makefile to fix jenkins testing error

Makefile came with a few other accidental changes, these have been corrected.

The big grid change. Converting gr from being an array of types containing 1D arrays, to a type containing 2D arrays. All cases BFB, cam multicolumn+silhs BFB, and cam multicolumn (no silhs) with backwards compatible settings BFB.

The big grid change. Converting gr from being an array of types containing 1D arrays, to a type containing 2D arrays. All cases BFB, cam multicolumn+silhs BFB, and cam multicolumn (no silhs) with backwards compatible settings BFB.

Adding column index to gr arrays, hard coding to 1 since sam only uses single column calls.

Removing the zt2zm interface from clubb_api and making the api calls just redirect to the grid class interface for it.

Pushing column loop into mean advection procedures.

Little loop push

Moving above/below index of grid weights to last dimension.

Hardcoding 1 into uses of gr arrays where single column versions of gr is used.

Pushing loop into fill_holes.

Forgot small change.

Making compatible with latest clubb change.

Making linear_interpolated_azt_2D and linear_interpolated_azm_2D subroutines just to avoid a needless data copy.

Making update_xp2_mc 2D and creating interface for 1D calls.

Fixing units in stats_zt_module.F90

This change will be BIT_CHANGING because it will affect the descriptions of variables in the netcdf files, but will not change answers.

Fixing a bug in mono_flux_limiter.F90. (#1026)

* Fixing a bug in mono_flux_limiter.F90.  This code was outputting "xm" instead of "wpxp" in several places.

This commit is BIT_CHANGING since it will affect the netcdf output for these variables, even though there
is no real change to answers.

* Renaming wpthlp_entermfl to wpthlp_enter_mfl which is more consistent with other variable names.

This is BIT_CHANGING due to changing the name and description in the output netcdf files.

* Renaming a few more instances of wpthlp_entermfl to wpthlp_enter_mfl.

Fixing a bug.

I changed the w_up_in_cloud code in 2 ways:

1)  The denominator term is now the "cloudy updraft" frac in each
PDF component, rather than just the PDF component cloud_frac.  This
is more consistent with the quantity being integrated.

2)  I also added a w_down_in_cloud for cloudy downdraft velocity.

Since these fields are not output as part of standard stats, this
commit will be bit-for-bit for the normal CLUBB output files.

I added stats output for w_down_in_cloud to all_stats.in.

Since this stat was not added to standard_stats.in, official results
will still be bit-for-bit.

Modified for the latest changes to CLUBB.

adding line to fix mpi configuration in micro drizzle test

I altered the w_up_in_cloud and w_down_in_cloud code so that a
thresholding is used similar to what is seen in the cloud fraction
and cloud water code.  If the PDF component mean of w more than
the maximum allow number of standard deviations away from 0, the
PDF component is either all-updrafty or all-downdrafty, and the
code avoids expensive computations where large magnitude values
can potentially be fed into ERF or EXP.

Of course, the results are not bit-for-bit with the previous version,
meaning that there are some situations where these thresholds come
into effect.  However, plots of all cases show no visible differences
in w_up_in_cloud and w_down_in_cloud.

Since w_up_in_cloud and w_down_in_cloud are not included in normal
stats output, this commit does not change the bit-for-bitness of
CLUBB code.

I have optimized the new w_up_in_cloud and w_down_in_cloud code by
only doing repeated operations one time and then saving them as a
local variable.

Since the numerical order of operations changes for the argument
to the EXP term, the results will differ at the level of numerical
round-off, making this revision not bit-for-bit with the last revision.
This only affects the w_up_in_cloud and w_down_in_cloud statistical
output variables, which in turn are not output as part of
standard_stats.in.

Refactored fill_holes_vertical to make GPUization simple. This is BIT_CHANGING, but results are bit-for-bit when using -O0 optimization, thus it is not answer changing. The first pass over each grid column will not parallelize well, the k-loop needs to be done in serial. Maximum parallelization has been exposed for the global hole-filling though, at the cost of occasionally doing unneccesary calculations. larson-group/clubb#972.

Removing fill_holes_multiplicative and replacing magic numbers with parameters from constants_clubb. larson-group/clubb#972

Moving vertical_avg and vertical_integral to advance_helper_module. larson-group/clubb#972

Moving vertical_avg and vertical_integral to advance_helper_module. larson-group/clubb#972

Making compatible with latest clubb change.

Removing elementalness from sat_vapor_press_liq and making interal procedures subroutines rather than functions to prevent unneccesary data copies. Doing the same for thlm2T_in_K since it is often used in conjunction with sat_vapor_press_liq. Bit-for-bit confirmed with O0 using all single column cases with or without l_diag_Lscale_from_tau, and with cam_coarse_res. larson-group/clubb#972

Adding comments.

Adding initial subroutine calc_ice_cloud_frac_component.

Cleaning up new subroutine calc_liquid_cloud_frac_component, and making sat_mixrat_ice a subroutine that works the same way as sat_mixrat_liq. larson-group/clubb#972

Comment update

Removing these paratheses is BIT_CHANGING since it modifies the order of operations, but allows for the multiplication and subraction to be done in parallel for complex pipelines.

Improvements. Netcdf output is now functional and we can detect errors with multiple columns even when the standard output is identical.

Updating column num input in config flags. larson-group/clubb#1033

Adding comments, setting to num_standalone_columns=1 by default, and preventing multicolumn output when l_stats=false. larson-group/clubb#1033

Improvements. Netcdf output is now functional and we can detect errors with multiple columns even when the standard output is identical.

Namelists need to be closed with a '/', and omitting it will break many tests.

Making save variables in the multicol code threadprivate, this fixes the openmp test. Also changing the way we name and save the multicol output, this allows the parallel run to run with multiple columns. I've reflected the changes in comments where relevant. larson-group/clubb#1033

Adds commented-out line that prevents the monotonic flux

limiter from acting on rtm.  The comments are included to remind us of how to do this in case we choose to do so in the future.

Updating monotonic flux limiter code to remove spikes. (#1038)

Also putting a commented out code change in advance_xm_wpxp,
in case shutting of the flux limiter entirely for rtm is preferred.

See https://github.com/NCAR/amwg_dev/discussions/134 for discussion.

This commit will be BIT_CHANGING although the differences are only
obvious in sensitive cases like arm_97.

Creating new flags to control monotonic flux limiter (#1039)

* Creating new flags to control monotonic flux limiter and the IF statement fix for the spikes found in CAM.

* Removing no-longer-necessary logical l_mono_flux_lim.

* Adding requested changes from PR (intent statements and editing error statement).

* Accidentally set flag to false. Changing to true.

Creating new flags to control monotonic flux limiter (#1039)

* Creating new flags to control monotonic flux limiter and the IF statement fix for the spikes found in CAM.

* Removing no-longer-necessary logical l_mono_flux_lim.

* Adding requested changes from PR (intent statements and editing error statement).

* Accidentally set flag to false. Changing to true.

Updating for changes to CLUBB.  See https://github.com/larson-group/clubb/pull/1039.

I am adding cloudy_updraft_frac and cloudy_downdraft_frac as
outputs to the calculate_w_up_in_cloud code.

These fields are non-interactive for the standard set of cases.
Thus, all cases are bit-for-bit identical.

However, for the w_up_in_cloud and w_down_in_cloud results themselves,
it is possible that results might not be bit-for-bit since
the location of the max(eps, ...) clipping in the denominator
is changed.  However, results should not be appreciable different.

Updating SAM-CLUBB for the latest changes to the call to
advance_clubb_core_api.

Minor adjustment to eliminate spikes in thlm tendencies from the monotonic flux limiter. (#1043)

See https://github.com/NCAR/amwg_dev/discussions/134.

This commit will be BIT_CHANGING. Differences are mainly noticeable in the more sensitive cases.

Making interface for matrix solvers.

Making tridag tridiag everywhere, making sol soln, and adding intent comments.

A different way of dealing with monotonic flux limiter spikes (#1046)

* A different way of dealing with monotonic flux limiter spikes in CAM, by increasing the value
of thl_tol_mfl.  ALso reverts the earlier fix.  BIT_CHANGING.

See https://github.com/NCAR/amwg_dev/discussions/134#discussioncomment-4165447.

* Reducing value of thl_tol_mfl from 0.5 to 0.2.

This will be BIT_CHANGING.

Steffens tuner fixer branch (#1041)

* Clubb ticket #1025: Implemented changes dealing with pdf_params%thl1/2 and wp2 floating point errors occurring in tuning runs. BIT_CHANGING
- Added command-line option -t/--tuner to compile.bash which enables the -DTUNER compiler flag.
- Added line to gfortran compilation config file to easily disable openMP
- Added a couple error messages and cleaned up some instances of error handling in src/error.F90, src/clubb_driver.F90, and src/CLUBB_core/advance_clubb_core_module.F90
- Added global constant wp2_max in src/CLUBB_core/constants_clubb.F90 which sets the upper bound for wp2
- In pdf_closure, added sanity checks for pdf_params%thl1/2 (>=190K, <=1000K)
- Added debug warning in src/CLUBB_core/advance_wp2_wp3_module.F90 when wp2 is clipped.
- Added wp2_sfc clipping in src/CLUBB_core/sfc_varnce_module.F90
- Added debug_level_check to NaN check in clubb_driver.F90
- Added mention of the new compiler option to the README

Making compatible with latest clubb changes.

Adding capability to change matrix solving method via clubb_config_flags.

Fixing small bug, need to pass _copy arrays to prevent lapack mangling the real ones.

Adding capability to change matrix solving method via clubb_config_flags.

Making GPU and CPU versions of the penta_lu solver the same as discussed in larson-group/clubb#1024.

Replacing $ with & in namelist definition to make consistent.

I added "smooth" max clipping for invrs_tau_shear, which is a variable
that is supposed to be positive definite, yet was obtaining negative
values at the model lower boundary owing to linear extension at the
boundaries as part of the linear interpolation call.

This change is BIT_CHANGING.

ADG1_pdf_driver subroutine port with OpenACC

Added OpenACC related flags in linux_x86_64_nvhpc_casper.bash
You can enable/disable OpenACC compilation using OPENACC=true/false.
Added OpenACC directives in ADG1_pdf_driver subroutine.

Replacing old elemental ADG1_w_closure with new GPUized one. Making G_unit tests work with new version. Also making mixt_frac_max_mag a scalar since it was only being used as such. larson-group/clubb#1049

I can now safely remove all the "ifdef E3SM" statements from CLUBB's
parameters_tunable.F90.  This code is now located in the clubb_intr.F90
portion of E3SM.

Pushing column loop into lapack wrap.

Restructuring and Porting of Compute_mixing_length subroutine(Phase 1) (#1052)

* Restructuring and Porting of Compute_mixing_length subroutine(Phase 1)

Restructure: The compute_mixing_length is one of the top most routine taking 35-50%
of the total time in a single timestep. The subroutine has been restructured to
push the i-loop further down to extract vectorization and parallelization.
The restructuring also involves introduction of sat_mixrat_liq_acc routines to
extract parallelism when called inside a OpenACC parallel region.

Porting: OpenACC directives are inserted to port the restructured compute_mixing_length
code on to the GPUs. This port is currently unoptimized and there is still room for improvement.

NOTE: Currently, l_sat_mixrat_lookup = false and saturation_formula = saturation_flatau
(Earthworks config options) case is supported on OpenACC build. Any other options works
on CPUs as usual. OpenACC declare create directives are inserted in model_flags and
constants_clubb, as these module variables are used inside the saturation routines.

* Added debug message about only supporting
l_sat_mixrat_lookup = false and saturation_formula = saturation_flatau
 on GPUs
Answers are Bit for Bit with arm-multicolumn case + nvhpc compiler.

* Changing CLUBB debug level 1 to 0 for the saturation formula support
running on GPUs

* Changing indentation to make gfortran happy, it wants ifdefs to start at the beginning of the line.

* Adding use statements for error checks and printouts, also making the errors set err_code to clubb_fatal_error.

Co-authored-by: huebler <huebler@uwm.edu>

Restructuring and Porting of Compute_mixing_length subroutine(Phase 2) (#1054)

* Restructuring and Porting of Compute_mixing_length subroutine(Phase 2)

Restructure: sat_mixrat_liq_2D_acc is being directly called instead of
calling the 1D version inside the column loop. Changing sat_mixrat_liq_2D_acc
to a subroutine from a function and adding output array, start_index as
additional argument. This is a workaround for passing sub-arrays. The OpenACC
doen't like the sub-arrays being passed and fails the validation.

Porting: OpenACC directives are added inside sat_mixrat_liq_2D_acc for porting

Validation: Answers are Bit for Bit with arm-multicolumn case + nvhpc compiler

* Fix for compilation issues

Issue 1: Missed out declaring 'start_index' while intergrating the
change

Issue 2: The use of error_code module and the procedures inside it
causes OpenACC compilation issues when run on the device.

* Removing the sat_mixrat_liq_acc and sat_mixrat_liq_2D_acc, making the normal sat_mixrat_liq work for all current use cases, and making the other versions of sat_mixrat_liq (bolton,gfdl,lookup) functional with OPENACC.

Co-authored-by: huebler <huebler@uwm.edu>

Breaking up column loop in mono_flux_limiter. This may not be the final form for GPUization, but it's definitely a start, no longer do we have to copy single column variables to multicolumn ones anywhere. (#1051)

Adding tridiag_lu solver (#1056)

* Adding tridiag_lu solver

* Changing low to lower in tridiag_lu and penta_lu, also fixing up some spacing.

* Making variables that represent matrix solutions more consistently named.

Adding OpenACC data directives for mixing length and adg routines

OpenACC structured data regions are added to optimize the data transfers
between CPU and GPU. These data regions will converted to unstrucutred
data region in the later optimization phase.
Results are bit for bit.

Fixing error causing GPU code not to run. Some variables to be copied were labelled as (ngrdcol,nz) when it should be (:ngrdcol,:nz). I've just removed the data length specifiers completely since they are not neccesary in general.

GPUizing sat_mixrat_ice_2D.

Pushing column loop into calculate_thvm

Removing update_pressure since it is no longer called anywhere in clubb or host models. The addition of this subroutine was discussed in larson-group/e3sm#6 and the removal of the call to it was discussed in larson-group/clubb#926.

Removing update_pressure from public list. This was causing compilation crashes. RESOLVED:8c7230fecb877d04fb129ef5e143e0993b4b29b1

GPUizing some helping procedures.

Removing sigma_sqd_w from the acc data copyout statement. This is a bug which was causing the code to crash when not using managed memory.

Porting pdf_closure subroutine with OpenACC (#1059)

* Porting pdf_closure subroutine with OpenACC

OpenACC directives are added to pdf_closure subroutine. The necessary
structured data region is also added for optimzing data movement across
kernels. There is opportunity to task parallelize using streams and
will be explored in the future.

Answers are bit for bit.

* Fixing the copyin directives

* Making work without managed memory.

Adding wp3_on_wp2_cfl_num to CLUBB's output stats.

Removing usage of gr from pdf_closure. It was only ever used for nz, which is now fed in directly.

Clubb ticket #1025: Implemented way to make esa tuner reproducible, h… (#1068)

* Clubb ticket #1025: Implemented way to make esa tuner reproducible, hid error output of optional diagnostic variables behind check, fixed parallelization issue with tuner, esa max_iters parameter is now in stats namelist, fixed issue with TUNER compiler directive, some small fixes.
- New namelist variables prescribed_rand_seed and l_use_prescribed_rand_seed determine if the esa tuner will use a random or a fixed value as random seed. Added descriptions to README.
- Added max_iters to stats namelist to make it more modifiable.
- Renamed stp_adjst_intercept_in and stp_adjst_slope_in to stp_adjst_shift_in and stp_adjst_factor_in, respectively, to better reflect their influence on step size.
- The error output in src/CLUBB_core/pdf_closure_module.F90 for the diagnostic variables wprtp2, wpthlp2, wprtpthlp, and rcp2 is now hidden behind existence checks for these variables. And a clarification was added to "#ifdef TUNER" directive.
- NetCDF file access caused the tuner to crash in parallel mode (-fopenmp flag in config file and multiple cases). Adding an $OMP CRITICAL structure around the call to stats_init in clubb_driver.F90 fixed that.
- Fixed compile/README. Config files are specified with the -c option.
- The -t option in run_scripts/run_tuner.bash interfered with the previous usage of the TUNER compiler directive. Renamed the old TUNER directive to NR_SP, a short for "numerical recipes, single precision". TUNER now is the option to "turn on" code changes required to run the tuner.

Gpu updates (#1070)

* Fixing bug. This was only triggered when l_input_fields=.true., which I am only testing because it needs to be true so that I can test ADG2_driver.

* Removing usage of gr from pdf_closure. It was only ever used for nz, which is now fed in directly.

* Making openacc statements more consistent. Ensuring all statments on double loops have specified gang and vector, and that all parallel loops have an end parallel loop statment at the end of them. Everything BFB on CPUs and GPUs.

* Pushing acc data region to outermost parts of mixing_length.

* Removing pdf_implicit_coefs_terms from acc copyin and copyout. It is only used when iiPDF_type == iiPDF_new .or. iiPDF_type == iiPDF_new_hybrid, so we do not need to do any copying with it. The inclusion of it also caused the data statement to copy unallocated arrays, which are just garbage pointers, and that was causing random occasional crashes (either segfaults or gpu out of memory).

* The update device clauses for return variables seems to only be requried for arrays contained in types. See https://github.com/larson-group/clubb/issues/1049\#issuecomment-1440624778

* Moving acc end data to end of pdf_closure. This reuqired removing any conditional return statements that appear before the final return, since we're not allowed to branch out of an acc region early. I also moved a large printout statement outside of a loop. The only reason it was in the loop to begin with was because pdf_params used to be an array of types, but now is a type of arrays, allowing us to print the full arrays directly.

* Making loop an acc loop. If we weren't outputting w_[up/down]_in_cloud (iw_up_in_cloud <= 0 .or. iw_down_in_cloud <= 0, then these arrays were only being zerod out on the CPU and would've getting overwritten by the uninitialized GPU data at the end of the data statement. This change causes the arrays to get correctly zerod out on the GPU when we need.

* Update VariableGroupNondimMoments.py

Fixed a typo

* Merging new changes from master

* Removing need for -gpu=deepcopy, pushing some acc data statements up call tree, and replacing some acc data statements with acc delare statements so that return statements can be added back in.

* Adding back an acc loop that was accidentally removed during a merge.

---------

Co-authored-by: Brian Griffin <31553422+bmg929@users.noreply.github.com>

Implements Thomas Toniazzo's bug fix for spike removal

in the monotonic flux limiter.  See

https://github.com/larson-group/clubb_release/pull/11/files

This code change is
expected to be BIT_CHANGING for cases in which
`l_predict_upwp_vpwp = T`, `l_mono_flux_lim_um = T`
or `l_mono_flux_lim_vm = T`, and the monotonic
flux limiter is triggered.

This bug fix prevents non-conservation of momentum
when the vertical integral of either of the wind components `um` or `vm` is negative.

Making 'a_const' into a CLUBB tunable parameter. (#1072)

* Converting the sfc_varnce_module.F90 parameter 'a_const' into a CLUBB tunable parameter.

Results should be bit-for-bit since it will be set to 1.8 and with l_vary_convect_depth=F
this is the standard value.

Some discussion in ticket #994.

* Completing requested changes, see PR #1072.

Making 'a_const' into a CLUBB tunable parameter. (#1072)

* Converting the sfc_varnce_module.F90 parameter 'a_const' into a CLUBB tunable parameter.

Results should be bit-for-bit since it will be set to 1.8 and with l_vary_convect_depth=F
this is the standard value.

Some discussion in ticket #994.

* Completing requested changes, see PR #1072.

Updating for changes to CLUBB.  See PR #1072.

Pdf closure driver gpuization (#1071)

* GPUizing pdf_closure_driver, testing still needed, bug likely.

* Making data movement better. Also fixed some random things. Still need to return returns.

* Adding back a couple return statments. Moving a comment where it should be. GPUizing a missed loop.

* Making arguements list satisfy standards.

* Making arguements list satisfy standards. Also adding max(1,hydrometdim) to hydromet inputs.

* Capitalizing K

* Making the error case in compute_cloud_cover behavior a bit differently. This allows the error to be detected immediately by GPU runs as well.

* Calling compute_cloud_cover only needs to be done if l_use_cloud_cover = .true.

Moving compute_cloud_cover outside of if ( l_use_cloud_cover ) then statement, the cloud_cover and rcm_in_layer variables they compute aren't output in clubb_standalone, but are in cam, causing cam bit diff tests to break.

Adding Skthl_zm to the update host list, I missed this in the last PR. I noticed this by comparing results with and without managed memory, now I've checked BFBness with arm, mpace_b, mc3e, and gabls2.

Small GPU fixes (#1076)

* Fixing small things that I caught by adding the default(present) onto acc loops.

* Moving default(present) to the end because it looks nicer there.

Code changes to implement modificiations on wp3 clippings
This commit contains code changes to implement modifications of skewness
clippings on wp3 in src/CLUBB_core/clip_explicit.F90. The default method attempts
to apply smaller (larger) clippings below (above) 100m AGL level, which
can cause a discontinuities around 100m AGL level. This clippings is
found to trigger sawthooth oscillations in wp3 when linear diffusion is
used. Such swathooth oscillations are eleminated if a smoothed Heaviside
function is introduced to obtain a smooth transition of clippings at
around 100m AGL level. The changes is necessary to obtain the first
order convergence in CLUBB-SCM when lienar diffusion is used.

Previously committed by Shixuan.

Code changes to implement modificiations on wp3 clippings
This commit contains code changes to implement modifications of skewness
clippings on wp3 in src/CLUBB_core/clip_explicit.F90. The default method attempts
to apply smaller (larger) clippings below (above) 100m AGL level, which
can cause a discontinuities around 100m AGL level. This clippings is
found to trigger sawthooth oscillations in wp3 when linear diffusion is
used. Such swathooth oscillations are eleminated if a smoothed Heaviside
function is introduced to obtain a smooth transition of clippings at
around 100m AGL level. The changes is necessary to obtain the first
order convergence in CLUBB-SCM when lienar diffusion is used.

Previously committed by Shixuan.

This commit contains code changes to implement modifications on limiters in three places:

1. remove the limiters in denominator of equation for
   brunt_vaisala_freq_sqd_smth, which affects the computed
   eddy dissipation time scale in turbulent fluxes (wpxp).
   (in mixing_length.F90)

2. reduce the threshold values of limiters in the equation
   for richardson number (sqrt_Ri_zm)
   (in mixing_length.F90)

3. introduce the smoothed max/min function for limiters in
   equation of Cx_fnc_Richardson.
   (in advance_helper_module.F90).

After the modification, we also apply a zt2m(zm2zt) smoothing
on the calculated quantities. These modifications are found to
be benificial for improving solution convergence in CLUBB-SCM

The code changes are controlled by a newly introduced flag named
"l_modify_limiters_for_cnvg_test", which is set to .false. (meaning that
the modificaitons on limiters is turned off) by default.

Originally committed by Shixuan.

This commit contains code changes to implement modifications on limiters in three places:

1. remove the limiters in denominator of equation for
   brunt_vaisala_freq_sqd_smth, which affects the computed
   eddy dissipation time scale in turbulent fluxes (wpxp).
   (in mixing_length.F90)

2. reduce the threshold values of limiters in the equation
   for richardson number (sqrt_Ri_zm)
   (in mixing_length.F90)

3. introduce the smoothed max/min function for limiters in
   equation of Cx_fnc_Richardson.
   (in advance_helper_module.F90).

After the modification, we also apply a zt2m(zm2zt) smoothing
on the calculated quantities. These modifications are found to
be benificial for improving solution convergence in CLUBB-SCM

The code changes are controlled by a newly introduced flag named
"l_modify_limiters_for_cnvg_test", which is set to .false. (meaning that
the modificaitons on limiters is turned off) by default.

Originally committed by Shixuan.

Updated for recent changes to CLUBB.

changing tunable_parameters

changing tunable_parameters

changing tunable_parameters

Advance xm wpxp gpuization (#1077)

* Initial commit for GPUizing advance_xm_wpxp.

* GPUizing fill_holes_vertical

* Adding Skthl_zm to the update host list, I missed this in the last PR. I noticed this by comparing results with and without managed memory, now I've checked BFBness with arm, mpace_b, mc3e, and gabls2.

* Small GPU fixes (#1076)

* Fixing small things that I caught by adding the default(present) onto acc loops.

* Moving default(present) to the end because it looks nicer there.

* Adding default(present) to all acc loop statements. Also adding azt to a copyin statement, which was missed previously. All BFB.

* Incemental update, not well tested yet.

* Removing some copies and making the sclr_dim change.

* Fixing a bug that only seemed detectable with astex_a209. We need to pass only single arrays to functions, calling ddzt( nz, ngrdcol, gr, rho_ds_zt * K_zt_nu ) was resulting in rho_ds_zt * K_zt_nu being evluated on the CPU, but the values were only valid on the GPU. So we need to evaluate that expression on the GPU, save it into an array (currently K_zt_nu_tmp), then pass that to ddzt.

* GPUizing calc_turb_adv_range

* GPUizing mono_flux_limiter

* Cleaning up data statments and a couple other things.

* Updated for some different options.

* More updates needed for various options.

* Reverting accidental flag change

* Should be the final changes, all options tested now.

* Replacing some comments in monoflux limiter, and also modifying it to make it BFB on CPUs. Also changing incorrect error conditions on tridiag.

* Adding max_x_allowable to update host statement, missed previous.

* Properly naming tmp variables and variables calculated from ddzt and ddzm start with ddzt_ and ddzm_.

* Replacing constants with named ones from constants_clubb.

* Replacing hard coded numbers in lhs variables representing the number of bands they contain with fortran parameters.

Small tweaks to fix some GPU bugs. Some variables were uninitialized on the CPU while we were saving them. This could only have been caught by comparing consecutive runs and checking _zt and _zm files, even then few cases were having problems.

Fixing a labelling error in redirect_interpolated_azt_2D and similar procedures, since this interpolates to zt the input should be zm. I think this was my fault, so I cleaned all the zt2zm and zm2zt things up to make it a little nicer. Also ordered the routines _k _1D _2D to make it easier to jump around, it was a bit confusing as they were out of order and the typo really made it hard.

Making it so sclr_tol is set to 0 before the specified sclr_tol_in. This is so that it is initialized to 0 in the case that sclr_dim = 0, since now we are setting it to have a minimum allocation size of 1 and would otherwise have a garbage value. This is what broke the clubb_openmp_gfortran_test.

Lscale GPUization (#1079)

* Making 2 new functions zm2zt2zm and zt2zm2zt to handle smoothing by interpolation. Replaced the spots in clubb I know that uses this to smooth things. This is just a nice to have and could allow for easy optimizations in the future by inlining the interpolations. All cases BFB on CPU and GPU, checked all relevant options too.

* GPUizing diagnose_Lscale_from_tau

* Removing some unused variables.

* Moving acc data statements from calc_Lscale_directly up to advance_clubb_core.

* Removing an unused variable.

* GPUizing the l_smooth_min_max option.

* GPUizing l_avg_Lscale

* Changes to variable names to avoid gross long names only used once.

gfortran does not like intent(out) specifiers for functions.

GPUizing Various Small Routines (#1080)

* GPUizing compute_Cx_fnc_Richardson

* GPUizing term_wp2_splat and term_wp3_splat

* GPUizing xp3_LG_2005_ansatz

* GPUizing calculate_thvm

* GPUizing pvertinterp even though I don't think we care about the l_do_expldiff_rtm_thlm flag

* Fixing bug. Setting l_do_expldiff_rtm_thlm causes us to use edsclrm, so we need to also ensure that edsclrm > 1 (1 because it uses a edsclr_dim-1 index)

* Pushing some data statements up the call tree.

This commit is a commit that changes absolutely nothing. It is meant to trigger a change in the git update scripts, so that I can start the commit message logging in the autocommit updates larson-group/sys_admin#797

This commit is a commit that changes absolutely nothing. It is meant to trigger a change in the git update scripts, so that I can start the commit message logging in the autocommit updates larson-group/sys_admin#797

this is another commit that cahnges nothing that will trigger the gitUpdate scripts

adding an update that changes nothing and is just a test for the autoupdate script

Making CLUBB's splatting scheme implicit and smoother (#1075)

* Introducing rough implicit splatting scheme into CLUBB.  Not yet implicit for wp3.

* Implementing implicit splatting for wp3.

* Further improvements to implicit splatting code, including

- smoothing of the BV freq used for splatting
- adding a new variable 'bv_freq_sqd_splat' to stats
- add the new variable to pyplotgen standard output

See PR #1075 and issue #994.

* Some cleanup.

* More cleanup, and adding comments.

* More comments.

* Updating implicit splatting scheme.

* Updates to resolve PR requests.

Updates associated with implicit splatting will be BIT_CHANGING.

Calc sfc varnce GPUization (#1081)

* Initial loop push for calc_sfc_varnce

* BIT_CHANGING:3b086a40085284aa49c71d32c001d20153a8ddb4 the last commit is bit changing for only some cases and only when using higher than -02 optimization. uf min seems to be the first calculation that starting to differ bitwise. Using the check_multicol script confirms the differences are small.

* Adding a tweak to surface values in the extra columns. This helped me check calc_sfc_varance, since we were not changing any arrays that would've affected calculations there.

* Small optimization, making wstar and ustar2 scalars.

* GPUizing calc_sfc_varnce

* Removing conditional around some stats calls. Now we will always save sfc values to stats, because this will change stats files when gr%zm(i,1) > sfc_elevation, this is potentially BIT_CHANGING.

* Merging with latest clubb changes and making work on GPUs again.

This contained 2 commits that are BIT_CHANGING in some situations.

Reducing C_wp2_splat to 0.25 (from 2.0) for new implicit splatting code.

Also fixing bug that shows up when l_brunt_vaisala_freq_moist = T.  This
will hopefully help to get some failing CAM tests running again.

See #1074 and PR #1075 for info.

This is BIT_CHANGING for all cases due to the new tunable parameter value.

Reducing C_wp2_splat to 0.25 (from 2.0) for new implicit splatting code.

Also fixing bug that shows up when l_brunt_vaisala_freq_moist = T.  This
will hopefully help to get some failing CAM tests running again.

See #1074 and PR #1075 for info.

This is BIT_CHANGING for all cases due to the new tunable parameter value.

Removed Space From clip_explicit

This is another commit that changes absolutely nothing, and only exists to test autocommit changes

making changes to trigger autoudpate for test, these commits do nothing

here is a body to the commit as well

making changes to trigger autoudpate for test, these commits do nothing

here is a body to the commit as well

Here is another commit that changes nothing

Here is the body

Here is another commit that changes nothing

Here is the body

change to trigger updates

this is the last batch

change to calc pressure to trigger autoupdate

body

GPUizing Lscale_width_vert_avg. Loops have been restructured for simplicity, and algorithm has a different starting value to avoid k dependency. Results are BFB. (#1083)

GPUizing most of advance_clubb_core (#1084)

advance_wp2_wp3 with explicitly managed memory  (#1085)

* GPUizing most of advance_clubb_core

* GPUizing most of advance_wp2_wp3 based off Supreeths openacc branch

* Cleanup and small fixes.

* Converting data statement to declare statement to return the return.

* Small changes to prevent data copies

advance_xp2_xpyp with explicitly managed memory (#1086)

* GPUizing most of advance_clubb_core

* GPUizing most of advance_wp2_wp3 based off Supreeths openacc branch

* Cleanup and small fixes.

* Converting data statement to declare statement to return the return.

* Small changes to prevent data copies

* Initial advance_xp2_xpyp GPUization, adapted from Supreeth's branch, clubb_openacc, to work without managed memory.

* Small cleanup

* Tweaks to reduce data copies

* Small change

* More GPUization and some cleanup

* Final little fixes

* Removing magic logicals

advance_windm_edsclrm with explicitly managed memory (#1087)

* Initial commit for advance_windm_edsclrm adapted from Supreeth's branch

* Final fixes

* Accidentally undid fix

* Returning returns

Moving data statements to outter most parts of clubb and little fix in advance_wp2_wp3 (#1088)

Data movement improvement (#1089)

* Various reductions in data movement

* Bug fix

* Optimizations to fill_holes, this involves precalculating a multiplication an a division, so this is BIT_CHANGING.

* Missed a small part

* Small fix