Changes

Putting other log file copy commands in try catch to prevent it from failing the step before cleaning the directory

The big grid change. Converting gr from being an array of types containing 1D arrays, to a type containing 2D arrays. All cases BFB, cam multicolumn+silhs BFB, and cam multicolumn (no silhs) with backwards compatible settings BFB.

The big grid change. Converting gr from being an array of types containing 1D arrays, to a type containing 2D arrays. All cases BFB, cam multicolumn+silhs BFB, and cam multicolumn (no silhs) with backwards compatible settings BFB.

Making compatible with latest clubb change. update_xp2_mc will be broken by this, but all we need to do to fix it is push the column loop into it.

Fixing small bug preventing compilation.

Removing the zt2zm interface from clubb_api and making the api calls just redirect to the grid class interface for it.

Pushing column loop into mean advection procedures.

Little loop push

Moving above/below index of grid weights to last dimension.

Pushing loop into fill_holes.

Forgot small change.

Making linear_interpolated_azt_2D and linear_interpolated_azm_2D subroutines just to avoid a needless data copy.

Making update_xp2_mc 2D and creating interface for 1D calls.

Making use of new 2D call to update_xp2_mc

Fixing units in stats_zt_module.F90

This change will be BIT_CHANGING because it will affect the descriptions of variables in the netcdf files, but will not change answers.

Fixing a bug in mono_flux_limiter.F90. (#1026)

* Fixing a bug in mono_flux_limiter.F90.  This code was outputting "xm" instead of "wpxp" in several places.

This commit is BIT_CHANGING since it will affect the netcdf output for these variables, even though there
is no real change to answers.

* Renaming wpthlp_entermfl to wpthlp_enter_mfl which is more consistent with other variable names.

This is BIT_CHANGING due to changing the name and description in the output netcdf files.

* Renaming a few more instances of wpthlp_entermfl to wpthlp_enter_mfl.

Fixing a bug.

I changed the w_up_in_cloud code in 2 ways:

1)  The denominator term is now the "cloudy updraft" frac in each
PDF component, rather than just the PDF component cloud_frac.  This
is more consistent with the quantity being integrated.

2)  I also added a w_down_in_cloud for cloudy downdraft velocity.

Since these fields are not output as part of standard stats, this
commit will be bit-for-bit for the normal CLUBB output files.

I added stats output for w_down_in_cloud to all_stats.in.

Since this stat was not added to standard_stats.in, official results
will still be bit-for-bit.

Modifications for the latest CLUBB code changes.

Adding w_down_in_cl to the E3SM stats output, as well.

I altered the w_up_in_cloud and w_down_in_cloud code so that a
thresholding is used similar to what is seen in the cloud fraction
and cloud water code.  If the PDF component mean of w more than
the maximum allow number of standard deviations away from 0, the
PDF component is either all-updrafty or all-downdrafty, and the
code avoids expensive computations where large magnitude values
can potentially be fed into ERF or EXP.

Of course, the results are not bit-for-bit with the previous version,
meaning that there are some situations where these thresholds come
into effect.  However, plots of all cases show no visible differences
in w_up_in_cloud and w_down_in_cloud.

Since w_up_in_cloud and w_down_in_cloud are not included in normal
stats output, this commit does not change the bit-for-bitness of
CLUBB code.

I have optimized the new w_up_in_cloud and w_down_in_cloud code by
only doing repeated operations one time and then saving them as a
local variable.

Since the numerical order of operations changes for the argument
to the EXP term, the results will differ at the level of numerical
round-off, making this revision not bit-for-bit with the last revision.
This only affects the w_up_in_cloud and w_down_in_cloud statistical
output variables, which in turn are not output as part of
standard_stats.in.

Refactored fill_holes_vertical to make GPUization simple. This is BIT_CHANGING, but results are bit-for-bit when using -O0 optimization, thus it is not answer changing. The first pass over each grid column will not parallelize well, the k-loop needs to be done in serial. Maximum parallelization has been exposed for the global hole-filling though, at the cost of occasionally doing unneccesary calculations. larson-group/clubb#972.

Removing fill_holes_multiplicative and replacing magic numbers with parameters from constants_clubb. larson-group/clubb#972

Moving vertical_avg and vertical_integral to advance_helper_module. larson-group/clubb#972

Moving vertical_avg and vertical_integral to advance_helper_module. larson-group/clubb#972

Removing elementalness from sat_vapor_press_liq and making interal procedures subroutines rather than functions to prevent unneccesary data copies. Doing the same for thlm2T_in_K since it is often used in conjunction with sat_vapor_press_liq. Bit-for-bit confirmed with O0 using all single column cases with or without l_diag_Lscale_from_tau, and with cam_coarse_res. larson-group/clubb#972

Adding comments.

Adding initial subroutine calc_ice_cloud_frac_component.

Cleaning up new subroutine calc_liquid_cloud_frac_component, and making sat_mixrat_ice a subroutine that works the same way as sat_mixrat_liq. larson-group/clubb#972

Comment update

Removing these paratheses is BIT_CHANGING since it modifies the order of operations, but allows for the multiplication and subraction to be done in parallel for complex pipelines.

Improvements. Netcdf output is now functional and we can detect errors with multiple columns even when the standard output is identical.

I have updated the diagnostic. Main changes: 1. Merge Prof. Larson's tuner analyze_sensitivity_matrix. But domatrix is false by default, because the tuner is not the latest version.  2. Mergeing Kate's changes in function_pick_out.py. CAM's FV  dycore can be handled. 3. Plotting profiles below 700hPa. 4. Merge Brian's flag for DJF mean. 5. Creating new regional files for tuner, see draw_plots_hoz_2D.py.

Plotting file for large scale variables.

Adds commented-out line that prevents the monotonic flux

limiter from acting on rtm.  The comments are included to remind us of how to do this in case we choose to do so in the future.

Updating monotonic flux limiter code to remove spikes. (#1038)

Also putting a commented out code change in advance_xm_wpxp,
in case shutting of the flux limiter entirely for rtm is preferred.

See https://github.com/NCAR/amwg_dev/discussions/134 for discussion.

This commit will be BIT_CHANGING although the differences are only
obvious in sensitive cases like arm_97.

Creating new flags to control monotonic flux limiter (#1039)

* Creating new flags to control monotonic flux limiter and the IF statement fix for the spikes found in CAM.

* Removing no-longer-necessary logical l_mono_flux_lim.

* Adding requested changes from PR (intent statements and editing error statement).

* Accidentally set flag to false. Changing to true.

Updating for changes to CLUBB.  See https://github.com/larson-group/clubb/pull/1039.

I am adding cloudy_updraft_frac and cloudy_downdraft_frac as
outputs to the calculate_w_up_in_cloud code.

These fields are non-interactive for the standard set of cases.
Thus, all cases are bit-for-bit identical.

However, for the w_up_in_cloud and w_down_in_cloud results themselves,
it is possible that results might not be bit-for-bit since
the location of the max(eps, ...) clipping in the denominator
is changed.  However, results should not be appreciable different.

Modified for changes made to call to advance_clubb_core_api.

Minor adjustment to eliminate spikes in thlm tendencies from the monotonic flux limiter. (#1043)

See https://github.com/NCAR/amwg_dev/discussions/134.

This commit will be BIT_CHANGING. Differences are mainly noticeable in the more sensitive cases.

Making interface for matrix solvers.

Making tridag tridiag everywhere, making sol soln, and adding intent comments.

A different way of dealing with monotonic flux limiter spikes (#1046)

* A different way of dealing with monotonic flux limiter spikes in CAM, by increasing the value
of thl_tol_mfl.  ALso reverts the earlier fix.  BIT_CHANGING.

See https://github.com/NCAR/amwg_dev/discussions/134#discussioncomment-4165447.

* Reducing value of thl_tol_mfl from 0.5 to 0.2.

This will be BIT_CHANGING.

Steffens tuner fixer branch (#1041)

* Clubb ticket #1025: Implemented changes dealing with pdf_params%thl1/2 and wp2 floating point errors occurring in tuning runs. BIT_CHANGING
- Added command-line option -t/--tuner to compile.bash which enables the -DTUNER compiler flag.
- Added line to gfortran compilation config file to easily disable openMP
- Added a couple error messages and cleaned up some instances of error handling in src/error.F90, src/clubb_driver.F90, and src/CLUBB_core/advance_clubb_core_module.F90
- Added global constant wp2_max in src/CLUBB_core/constants_clubb.F90 which sets the upper bound for wp2
- In pdf_closure, added sanity checks for pdf_params%thl1/2 (>=190K, <=1000K)
- Added debug warning in src/CLUBB_core/advance_wp2_wp3_module.F90 when wp2 is clipped.
- Added wp2_sfc clipping in src/CLUBB_core/sfc_varnce_module.F90
- Added debug_level_check to NaN check in clubb_driver.F90
- Added mention of the new compiler option to the README

Making compatible with latest clubb changes.

Adding capability to change matrix solving method via clubb_config_flags.

Fixing small bug, need to pass _copy arrays to prevent lapack mangling the real ones.

Making GPU and CPU versions of the penta_lu solver the same as discussed in larson-group/clubb#1024.

I added "smooth" max clipping for invrs_tau_shear, which is a variable
that is supposed to be positive definite, yet was obtaining negative
values at the model lower boundary owing to linear extension at the
boundaries as part of the linear interpolation call.

This change is BIT_CHANGING.

ADG1_pdf_driver subroutine port with OpenACC

Added OpenACC related flags in linux_x86_64_nvhpc_casper.bash
You can enable/disable OpenACC compilation using OPENACC=true/false.
Added OpenACC directives in ADG1_pdf_driver subroutine.

Replacing old elemental ADG1_w_closure with new GPUized one. Making G_unit tests work with new version. Also making mixt_frac_max_mag a scalar since it was only being used as such. larson-group/clubb#1049

I updated the clubb_intr.F90 file in the clubb_silhs_devel branch to
match the new style of CLUBB parameter declaration that is found
in the v3_convection_candidate branches.

I can now safely remove all the "ifdef E3SM" statements from CLUBB's
parameters_tunable.F90.  This code is now located in the clubb_intr.F90
portion of E3SM.

Pushing column loop into lapack wrap.

Restructuring and Porting of Compute_mixing_length subroutine(Phase 1) (#1052)

* Restructuring and Porting of Compute_mixing_length subroutine(Phase 1)

Restructure: The compute_mixing_length is one of the top most routine taking 35-50%
of the total time in a single timestep. The subroutine has been restructured to
push the i-loop further down to extract vectorization and parallelization.
The restructuring also involves introduction of sat_mixrat_liq_acc routines to
extract parallelism when called inside a OpenACC parallel region.

Porting: OpenACC directives are inserted to port the restructured compute_mixing_length
code on to the GPUs. This port is currently unoptimized and there is still room for improvement.

NOTE: Currently, l_sat_mixrat_lookup = false and saturation_formula = saturation_flatau
(Earthworks config options) case is supported on OpenACC build. Any other options works
on CPUs as usual. OpenACC declare create directives are inserted in model_flags and
constants_clubb, as these module variables are used inside the saturation routines.

* Added debug message about only supporting
l_sat_mixrat_lookup = false and saturation_formula = saturation_flatau
 on GPUs
Answers are Bit for Bit with arm-multicolumn case + nvhpc compiler.

* Changing CLUBB debug level 1 to 0 for the saturation formula support
running on GPUs

* Changing indentation to make gfortran happy, it wants ifdefs to start at the beginning of the line.

* Adding use statements for error checks and printouts, also making the errors set err_code to clubb_fatal_error.

Co-authored-by: huebler <huebler@uwm.edu>

Restructuring and Porting of Compute_mixing_length subroutine(Phase 2) (#1054)

* Restructuring and Porting of Compute_mixing_length subroutine(Phase 2)

Restructure: sat_mixrat_liq_2D_acc is being directly called instead of
calling the 1D version inside the column loop. Changing sat_mixrat_liq_2D_acc
to a subroutine from a function and adding output array, start_index as
additional argument. This is a workaround for passing sub-arrays. The OpenACC
doen't like the sub-arrays being passed and fails the validation.

Porting: OpenACC directives are added inside sat_mixrat_liq_2D_acc for porting

Validation: Answers are Bit for Bit with arm-multicolumn case + nvhpc compiler

* Fix for compilation issues

Issue 1: Missed out declaring 'start_index' while intergrating the
change

Issue 2: The use of error_code module and the procedures inside it
causes OpenACC compilation issues when run on the device.

* Removing the sat_mixrat_liq_acc and sat_mixrat_liq_2D_acc, making the normal sat_mixrat_liq work for all current use cases, and making the other versions of sat_mixrat_liq (bolton,gfdl,lookup) functional with OPENACC.

Co-authored-by: huebler <huebler@uwm.edu>

Breaking up column loop in mono_flux_limiter. This may not be the final form for GPUization, but it's definitely a start, no longer do we have to copy single column variables to multicolumn ones anywhere. (#1051)

Adding tridiag_lu solver (#1056)

* Adding tridiag_lu solver

* Changing low to lower in tridiag_lu and penta_lu, also fixing up some spacing.

* Making variables that represent matrix solutions more consistently named.

Adding OpenACC data directives for mixing length and adg routines

OpenACC structured data regions are added to optimize the data transfers
between CPU and GPU. These data regions will converted to unstrucutred
data region in the later optimization phase.
Results are bit for bit.

Fixing error causing GPU code not to run. Some variables to be copied were labelled as (ngrdcol,nz) when it should be (:ngrdcol,:nz). I've just removed the data length specifiers completely since they are not neccesary in general.

GPUizing sat_mixrat_ice_2D.

Pushing column loop into calculate_thvm

Removing update_pressure since it is no longer called anywhere in clubb or host models. The addition of this subroutine was discussed in larson-group/e3sm#6 and the removal of the call to it was discussed in larson-group/clubb#926.

Removing update_pressure from public list. This was causing compilation crashes. RESOLVED:8c7230fecb877d04fb129ef5e143e0993b4b29b1

GPUizing some helping procedures.

Removing sigma_sqd_w from the acc data copyout statement. This is a bug which was causing the code to crash when not using managed memory.

Porting pdf_closure subroutine with OpenACC (#1059)

* Porting pdf_closure subroutine with OpenACC

OpenACC directives are added to pdf_closure subroutine. The necessary
structured data region is also added for optimzing data movement across
kernels. There is opportunity to task parallelize using streams and
will be explored in the future.

Answers are bit for bit.

* Fixing the copyin directives

* Making work without managed memory.

Adding wp3_on_wp2_cfl_num to CLUBB's output stats.

Removing usage of gr from pdf_closure. It was only ever used for nz, which is now fed in directly.

Clubb ticket #1025: Implemented way to make esa tuner reproducible, h… (#1068)

* Clubb ticket #1025: Implemented way to make esa tuner reproducible, hid error output of optional diagnostic variables behind check, fixed parallelization issue with tuner, esa max_iters parameter is now in stats namelist, fixed issue with TUNER compiler directive, some small fixes.
- New namelist variables prescribed_rand_seed and l_use_prescribed_rand_seed determine if the esa tuner will use a random or a fixed value as random seed. Added descriptions to README.
- Added max_iters to stats namelist to make it more modifiable.
- Renamed stp_adjst_intercept_in and stp_adjst_slope_in to stp_adjst_shift_in and stp_adjst_factor_in, respectively, to better reflect their influence on step size.
- The error output in src/CLUBB_core/pdf_closure_module.F90 for the diagnostic variables wprtp2, wpthlp2, wprtpthlp, and rcp2 is now hidden behind existence checks for these variables. And a clarification was added to "#ifdef TUNER" directive.
- NetCDF file access caused the tuner to crash in parallel mode (-fopenmp flag in config file and multiple cases). Adding an $OMP CRITICAL structure around the call to stats_init in clubb_driver.F90 fixed that.
- Fixed compile/README. Config files are specified with the -c option.
- The -t option in run_scripts/run_tuner.bash interfered with the previous usage of the TUNER compiler directive. Renamed the old TUNER directive to NR_SP, a short for "numerical recipes, single precision". TUNER now is the option to "turn on" code changes required to run the tuner.

Gpu updates (#1070)

* Fixing bug. This was only triggered when l_input_fields=.true., which I am only testing because it needs to be true so that I can test ADG2_driver.

* Removing usage of gr from pdf_closure. It was only ever used for nz, which is now fed in directly.

* Making openacc statements more consistent. Ensuring all statments on double loops have specified gang and vector, and that all parallel loops have an end parallel loop statment at the end of them. Everything BFB on CPUs and GPUs.

* Pushing acc data region to outermost parts of mixing_length.

* Removing pdf_implicit_coefs_terms from acc copyin and copyout. It is only used when iiPDF_type == iiPDF_new .or. iiPDF_type == iiPDF_new_hybrid, so we do not need to do any copying with it. The inclusion of it also caused the data statement to copy unallocated arrays, which are just garbage pointers, and that was causing random occasional crashes (either segfaults or gpu out of memory).

* The update device clauses for return variables seems to only be requried for arrays contained in types. See https://github.com/larson-group/clubb/issues/1049\#issuecomment-1440624778

* Moving acc end data to end of pdf_closure. This reuqired removing any conditional return statements that appear before the final return, since we're not allowed to branch out of an acc region early. I also moved a large printout statement outside of a loop. The only reason it was in the loop to begin with was because pdf_params used to be an array of types, but now is a type of arrays, allowing us to print the full arrays directly.

* Making loop an acc loop. If we weren't outputting w_[up/down]_in_cloud (iw_up_in_cloud <= 0 .or. iw_down_in_cloud <= 0, then these arrays were only being zerod out on the CPU and would've getting overwritten by the uninitialized GPU data at the end of the data statement. This change causes the arrays to get correctly zerod out on the GPU when we need.

* Update VariableGroupNondimMoments.py

Fixed a typo

* Merging new changes from master

* Removing need for -gpu=deepcopy, pushing some acc data statements up call tree, and replacing some acc data statements with acc delare statements so that return statements can be added back in.

* Adding back an acc loop that was accidentally removed during a merge.

---------

Co-authored-by: Brian Griffin <31553422+bmg929@users.noreply.github.com>

Implements Thomas Toniazzo's bug fix for spike removal

in the monotonic flux limiter.  See

https://github.com/larson-group/clubb_release/pull/11/files

This code change is
expected to be BIT_CHANGING for cases in which
`l_predict_upwp_vpwp = T`, `l_mono_flux_lim_um = T`
or `l_mono_flux_lim_vm = T`, and the monotonic
flux limiter is triggered.

This bug fix prevents non-conservation of momentum
when the vertical integral of either of the wind components `um` or `vm` is negative.

Making 'a_const' into a CLUBB tunable parameter. (#1072)

* Converting the sfc_varnce_module.F90 parameter 'a_const' into a CLUBB tunable parameter.

Results should be bit-for-bit since it will be set to 1.8 and with l_vary_convect_depth=F
this is the standard value.

Some discussion in ticket #994.

* Completing requested changes, see PR #1072.

Updating for changes to CLUBB.  See https://github.com/larson-group/clubb/pull/1072.

Pdf closure driver gpuization (#1071)

* GPUizing pdf_closure_driver, testing still needed, bug likely.

* Making data movement better. Also fixed some random things. Still need to return returns.

* Adding back a couple return statments. Moving a comment where it should be. GPUizing a missed loop.

* Making arguements list satisfy standards.

* Making arguements list satisfy standards. Also adding max(1,hydrometdim) to hydromet inputs.

* Capitalizing K

* Making the error case in compute_cloud_cover behavior a bit differently. This allows the error to be detected immediately by GPU runs as well.

* Calling compute_cloud_cover only needs to be done if l_use_cloud_cover = .true.

Moving compute_cloud_cover outside of if ( l_use_cloud_cover ) then statement, the cloud_cover and rcm_in_layer variables they compute aren't output in clubb_standalone, but are in cam, causing cam bit diff tests to break.

Adding Skthl_zm to the update host list, I missed this in the last PR. I noticed this by comparing results with and without managed memory, now I've checked BFBness with arm, mpace_b, mc3e, and gabls2.

Small GPU fixes (#1076)

* Fixing small things that I caught by adding the default(present) onto acc loops.

* Moving default(present) to the end because it looks nicer there.

Code changes to implement modificiations on wp3 clippings
This commit contains code changes to implement modifications of skewness
clippings on wp3 in src/CLUBB_core/clip_explicit.F90. The default method attempts
to apply smaller (larger) clippings below (above) 100m AGL level, which
can cause a discontinuities around 100m AGL level. This clippings is
found to trigger sawthooth oscillations in wp3 when linear diffusion is
used. Such swathooth oscillations are eleminated if a smoothed Heaviside
function is introduced to obtain a smooth transition of clippings at
around 100m AGL level. The changes is necessary to obtain the first
order convergence in CLUBB-SCM when lienar diffusion is used.

Previously committed by Shixuan.

This commit contains code changes to implement modifications on limiters in three places:

1. remove the limiters in denominator of equation for
   brunt_vaisala_freq_sqd_smth, which affects the computed
   eddy dissipation time scale in turbulent fluxes (wpxp).
   (in mixing_length.F90)

2. reduce the threshold values of limiters in the equation
   for richardson number (sqrt_Ri_zm)
   (in mixing_length.F90)

3. introduce the smoothed max/min function for limiters in
   equation of Cx_fnc_Richardson.
   (in advance_helper_module.F90).

After the modification, we also apply a zt2m(zm2zt) smoothing
on the calculated quantities. These modifications are found to
be benificial for improving solution convergence in CLUBB-SCM

The code changes are controlled by a newly introduced flag named
"l_modify_limiters_for_cnvg_test", which is set to .false. (meaning that
the modificaitons on limiters is turned off) by default.

Originally committed by Shixuan.

Updated for recent changes to CLUBB.

Advance xm wpxp gpuization (#1077)

* Initial commit for GPUizing advance_xm_wpxp.

* GPUizing fill_holes_vertical

* Adding Skthl_zm to the update host list, I missed this in the last PR. I noticed this by comparing results with and without managed memory, now I've checked BFBness with arm, mpace_b, mc3e, and gabls2.

* Small GPU fixes (#1076)

* Fixing small things that I caught by adding the default(present) onto acc loops.

* Moving default(present) to the end because it looks nicer there.

* Adding default(present) to all acc loop statements. Also adding azt to a copyin statement, which was missed previously. All BFB.

* Incemental update, not well tested yet.

* Removing some copies and making the sclr_dim change.

* Fixing a bug that only seemed detectable with astex_a209. We need to pass only single arrays to functions, calling ddzt( nz, ngrdcol, gr, rho_ds_zt * K_zt_nu ) was resulting in rho_ds_zt * K_zt_nu being evluated on the CPU, but the values were only valid on the GPU. So we need to evaluate that expression on the GPU, save it into an array (currently K_zt_nu_tmp), then pass that to ddzt.

* GPUizing calc_turb_adv_range

* GPUizing mono_flux_limiter

* Cleaning up data statments and a couple other things.

* Updated for some different options.

* More updates needed for various options.

* Reverting accidental flag change

* Should be the final changes, all options tested now.

* Replacing some comments in monoflux limiter, and also modifying it to make it BFB on CPUs. Also changing incorrect error conditions on tridiag.

* Adding max_x_allowable to update host statement, missed previous.

* Properly naming tmp variables and variables calculated from ddzt and ddzm start with ddzt_ and ddzm_.

* Replacing constants with named ones from constants_clubb.

* Replacing hard coded numbers in lhs variables representing the number of bands they contain with fortran parameters.

Small tweaks to fix some GPU bugs. Some variables were uninitialized on the CPU while we were saving them. This could only have been caught by comparing consecutive runs and checking _zt and _zm files, even then few cases were having problems.

Fixing a labelling error in redirect_interpolated_azt_2D and similar procedures, since this interpolates to zt the input should be zm. I think this was my fault, so I cleaned all the zt2zm and zm2zt things up to make it a little nicer. Also ordered the routines _k _1D _2D to make it easier to jump around, it was a bit confusing as they were out of order and the typo really made it hard.

Making it so sclr_tol is set to 0 before the specified sclr_tol_in. This is so that it is initialized to 0 in the case that sclr_dim = 0, since now we are setting it to have a minimum allocation size of 1 and would otherwise have a garbage value. This is what broke the clubb_openmp_gfortran_test.

Lscale GPUization (#1079)

* Making 2 new functions zm2zt2zm and zt2zm2zt to handle smoothing by interpolation. Replaced the spots in clubb I know that uses this to smooth things. This is just a nice to have and could allow for easy optimizations in the future by inlining the interpolations. All cases BFB on CPU and GPU, checked all relevant options too.

* GPUizing diagnose_Lscale_from_tau

* Removing some unused variables.

* Moving acc data statements from calc_Lscale_directly up to advance_clubb_core.

* Removing an unused variable.

* GPUizing the l_smooth_min_max option.

* GPUizing l_avg_Lscale

* Changes to variable names to avoid gross long names only used once.

gfortran does not like intent(out) specifiers for functions.

GPUizing Various Small Routines (#1080)

* GPUizing compute_Cx_fnc_Richardson

* GPUizing term_wp2_splat and term_wp3_splat

* GPUizing xp3_LG_2005_ansatz

* GPUizing calculate_thvm

* GPUizing pvertinterp even though I don't think we care about the l_do_expldiff_rtm_thlm flag

* Fixing bug. Setting l_do_expldiff_rtm_thlm causes us to use edsclrm, so we need to also ensure that edsclrm > 1 (1 because it uses a edsclr_dim-1 index)

* Pushing some data statements up the call tree.

This commit is a commit that changes absolutely nothing. It is meant to trigger a change in the git update scripts, so that I can start the commit message logging in the autocommit updates larson-group/sys_admin#797

This commit is a commit that changes absolutely nothing. It is meant to trigger a change in the git update scripts, so that I can start the commit message logging in the autocommit updates larson-group/sys_admin#797

this is another commit that cahnges nothing that will trigger the gitUpdate scripts

adding an update that changes nothing and is just a test for the autoupdate script

Making CLUBB's splatting scheme implicit and smoother (#1075)

* Introducing rough implicit splatting scheme into CLUBB.  Not yet implicit for wp3.

* Implementing implicit splatting for wp3.

* Further improvements to implicit splatting code, including

- smoothing of the BV freq used for splatting
- adding a new variable 'bv_freq_sqd_splat' to stats
- add the new variable to pyplotgen standard output

See PR #1075 and issue #994.

* Some cleanup.

* More cleanup, and adding comments.

* More comments.

* Updating implicit splatting scheme.

* Updates to resolve PR requests.

Updates associated with implicit splatting will be BIT_CHANGING.

Calc sfc varnce GPUization (#1081)

* Initial loop push for calc_sfc_varnce

* BIT_CHANGING:3b086a40085284aa49c71d32c001d20153a8ddb4 the last commit is bit changing for only some cases and only when using higher than -02 optimization. uf min seems to be the first calculation that starting to differ bitwise. Using the check_multicol script confirms the differences are small.

* Adding a tweak to surface values in the extra columns. This helped me check calc_sfc_varance, since we were not changing any arrays that would've affected calculations there.

* Small optimization, making wstar and ustar2 scalars.

* GPUizing calc_sfc_varnce

* Removing conditional around some stats calls. Now we will always save sfc values to stats, because this will change stats files when gr%zm(i,1) > sfc_elevation, this is potentially BIT_CHANGING.

* Merging with latest clubb changes and making work on GPUs again.

This contained 2 commits that are BIT_CHANGING in some situations.

Reducing C_wp2_splat to 0.25 (from 2.0) for new implicit splatting code.

Also fixing bug that shows up when l_brunt_vaisala_freq_moist = T.  This
will hopefully help to get some failing CAM tests running again.

See #1074 and PR #1075 for info.

This is BIT_CHANGING for all cases due to the new tunable parameter value.

Removed Space From clip_explicit

This is another commit that changes absolutely nothing, and only exists to test autocommit changes

making changes to trigger autoudpate for test, these commits do nothing

here is a body to the commit as well

making changes to trigger autoudpate for test, these commits do nothing

here is a body to the commit as well

Here is another commit that changes nothing

Here is the body

Here is another commit that changes nothing

Here is the body

change to trigger updates

this is the last batch

change to calc pressure to trigger autoupdate

body