Changes

For RegTune, adds quadratic curves to three-dot plot.

For #910.

For RegTune, defaults to polynomial fit for sensitivity matrix
and adds observations of sea-level pressure (PSL).

For #910.

In RegTune, normalizes metrics by a user-specified number
rather than the observed value of the metric.

For #910.

For RegTune, adds color-coded matrix of correlations among parameter vectors.
Also, this commit separates the three-dot panel plot into its own function.

For #910.

For clarity, renames figures in RegTune.

For #910.

Creates new function, setUpPreliminaries, in RegTune.
setUpPreliminaries calculates some basic quantities
that are used later.
As part of this refactoring, a number of functions
were moved from analyze_sensitivity_matrix to
set_up_dashboard_inputs.

For #910.

For RegTune, adds capability to prescribe (not tune) a parameter
to a value that is different from the default.
The code has not been checked, but it does run and produce plots.

For #910.

Hides print statements in RegTune by using beVerbose=False.

For #910.

Fixes bug in QuadTune's code that prescribes parameter values.
Also fixes divide-by-zero warning message in dpMin2Pt plotting code.

For #910.

Adds basic cost function printouts to QuadTune.
Also adds SWCF_RMSE metric.

For #910.

QuadTune: refactored to use more subroutines
and added a bar chart of column vectors (one per tuning parameter.)

For #910.

Fixed some oversights for the bv_efold commits and increased string length for tuning field names for invrs_tau variables - CLUBB ticket #1069 (#1106)

This commit is NOT bit changing
- Removed remaining mentions of brunt_vaisala_freq_sqd_plus
- Added assignment for invrs_tau_wp3_zm in Lscale mode in advance_clubb_core
- Increased length of string variables hoc_v, les_v, and t_variables in src/error.F90 to accomodate tuning for invrs_tau fields

BIT_CHANGING! brunt_vaisala_freq_sqd_mixed now replaces brunt_vaisala_freq_sqd for later calculations - CLUBB ticket #1102

Changes in src/CLUBB_core/mixing_length.F90:
- brunt_vaisala_freq_sqd_mixed is now used to calculate brunt_vaisala_freq_sqd_smth and brunt_vaisala_freq_sqd_clipped instead of brunt_vaisala_freq_sqd
- Removed 0.1 factor for bkgnd term in calculation of invrs_tau_xp2_zm
- Commented out clipping of invrs_tau_xp2_zm to 0.003 and lower out of cloud

Added option to plot "Lumped Buoyance Budgets" (wpxp_bp combined with
wpxp_pr3) to the pyplotgen plotter for CLUBB.

Added a statement that I had previously omitted. The new combined budget
works great!

Updating pyplotgen to include a "contour" plot of rcm in the background.
This will be made optional with the next commit.

I added the --background-rcm optional flag to pyplotgen.

Small additions that allow all cases to run with pyplotgen.

Needed to remove a print statement.

In the previous commits for the --background-rcm option of pyplotgen,
I had previously thought that the code to extract values of rcm from
a file wanted the case name as input, but in actuality it required
the directory name instead of the case name. I am correcting that
error in this commitment.

I am updating the --background-rcm option of pyplotgen to make two
changes:

1) Use a grayscale color map.
2) Set the maximum of the colorbar to be twice the largest value in
the rcm plot so that the colors aren't so dark.

BIT_CHANGING for lscale setup. Updated configurable_model_flags.in and model_flags.F90 - CLUBB ticket #1102 (#1108)

- Added new flags to inut/tunable_parameters_Lscale/configurable_model_flags.in
- Changed default setting for l_use_tke_in_wp3_pr_turb_term in model_flags.F90 from false to true

QuadTune: Adds chisqd metric and bar chart for a global run.
linSolnBiasesCol has been redefined to mean simulation - obs.

For #910.

QuadTune: adds commented-out attempt to vary the bar base
but it's not working yet.

For #910.

Likely BIT_CHANGING because of added output fields. Added code outputting C6_term and different Brunt Vaisala frequencies to disk - CLUBB ticket #1102 (#1109)

- Added C6_term and bv_freq fields to all_stats.F90 and standard_stats.F90
- Added new fields to VariableGroupTaus.py in pyplotgen
- Added calls to stat_update_var to advance_clubb_core_module.F90
- Modified stats_variables.F90 and stats_zm_module.F90 accordingly

Changed sqrt_Ri_zm to Ri_zm in preparation of changing the exponent for calculation of invrs_tau_wpxp_zm in mixing_length.F90 - CLUBB ticket #1102 (#1110)

BIT_CHANGING! Changed sqrt_Ri_zm to Ri_zm in preparation of changing the exponent for calculation of invrs_tau_wpxp_zm in mixing_length.F90 - CLUBB ticket #1102
Changes in input/stats/all_stats.in, src/CLUBB_core/advance_clubb_core_module.F90, src/CLUBB_core/mixing_length.F90, src/CLUBB_core/stats_variables.F90, and src/CLUBB_core/stats_zm_module.F90
Ri_zm was added to input/stats/standard_stats.in

BIT_CHANGING! Removed Richardson_num and all corresponding code - CLUBB ticket #1102 (#1111)

This commit is BIT_CHANGING because:
1. the output field Richardson_num was removed.
2. Since the flag l_use_shear_turb_freq_sqd was removed, the variable shear_sqd is now always written to disk. That means previously, the outputted field shear_sqd was 0 everywhere whereas now it is not anymore. The calculation of shear_sqd within CLUBB did not change!
- Removed mentions in input/stats/all_stats.in and standards_stats.in
- Removed mentions in src/CLUBB_core/stats_variables.F90 and stats_zm_module.F90
- In src/CLUBB_core/advance_helper_module.F90 removed mentions to [i]Richardson_num[_clipped], as well as l_use_shear_turb_freq_sqd, l_Richardson_vert_avg, and turb_freq_sqd sinc those are only used for calculation of Richardson_num.

QuadTune: Adds fnc to plot a horizonal bar chart for metrics.

For #910

QuadTune: Includes full perturbation in params bar chart

QuadTune: Adds new parameter correlation matrix plot.

For #910.

Major overhaul of run_bindiff_all.py

CLUBB ticket #1102 - pull request #1113
- Changed the outputs to be more evocative and differentiating between types of differences: Now we can distinguish between differences in the set of variables or differences in specific variables
- Fixed the issue where a variable that appears in only one of the compared files would either not be detected or cause the script to crash, depending on the order of input.
- Changed file output structure
- Changed case handling depending on existing files:
    - Case is still treated even if it is commented in RUN_CASES
    - Case is still treated even if not all files exist and messages are printed accordingly

Tunable Richardson number exponent - CLUBB ticket #1102 (#1114)

BIT_CHANGING! Added the exponent to the Richardson number in the calculation of the invrs_tau_wpxp threshold term in src/CLUBB_core/mixing_length.F90  to the list of tunable parameters.
This is bit changing due to the difference in Fortran between taking the sqrt and the .5th power.

Removes blank line in order to trigger nightly tests.

Deletes svn id tag, will trigger host-model nightly tests.

Remove svn id tag in order to trigger nightly tests.

Removed 3rd svn id tag in order to trigger nightly tests

Finally removing scratch variables. (#1116)

* Finally removing scratch variables.

* Zeroing out below ground levels to prevent NaN in stats.

Making run_bindiff_all.py an executable.

QuadTune: Imposes lower bound of 0 on tuned parameter values.

For #910.

QuadTune: Adds marker for residual bias on metrics bar chart.

For #910.

QuadTune: Updates setup to reproduce results from sens0707 sample 34.

For #910.

Latest batch of non-bit changing code changes - CLUBB ticket #1115 (#1119)

- Added new folder input/tunable_parameters_tauMatchLscale/ for latest taus code setup reproducing the master Lscale output.
- Changes in src/CLUBB_core_mixing_length.F90:
   - Renamed ddzt_umvm -> ddzt_umvm_sqd
             ddzt_umvm_clipped -> ddzt_umvm_sqd_clipped
             sqrt_ddzt_umvm -> norm_ddzt_umvm
             smooth_sqrt_ddzt_umvm -> smooth_norm_ddzt_umvm
             Ri_zm_clipped -> Ri_zm_smooth in some places
   - Made some variable replacements for consistency's sake
- Fixed path of log files in run_scripts/run_bindiff_all.py

BIT_CHANGING! Fixed some issues with wpxp_Ri_exp, changed z_displace value - CLUBB ticket #1115 (#1120)

- Removed commented lines from tunable_parameters.in file in input/tunable_parameters_tauMatchLscale
- Increased z_displace value from 10 to 25. This value was chosen so that the taus code with the parameter and flag settings in input/tunable_parameters_tauMatchLscale match the master Lscale output below clouds
- Fixed some issues with the new parameter wpxp_Ri_exp:
   - Moved new exponentiation to a place so that it emulates the previous version with sqrt(Ri_zm)
   - Adjusted code for l_smooth_min_max=true to match the default version

Updated CLUBB so that adj_low_res_nu (which can be called on its own
or called through calling setup_parameters) no longer uses
momentum_heights and thermodynamic_heights, but rather makes use
of gr%zm and gr%zt, which are always set and available by the
time the calls are made.

Removing begin_height and end_height as outputs from setup_grid. They
are only necessary as local variables inside of setup_grid.

Z displace tunable (#1122)

CLUBB ticket #1115
Added the previously hard-coded z_displace as a tunable parameter.

Quick fix for AnimationPanel so it works with the background-rcm changes (#1123)

- Modified super call in AnimationPanel constructor
- Removed first frame from consideration when determining x-axis scaling since the starting values are sometimes large compared to the other time steps

New Stats Indices Type (#1118)

* Grouping stats indices into a derived type and making that accessible globally, for now. This does not run yet, but a checkpoint I wish to save.

* Checkpoint to save

* Checkpoint

* It actually works now and is BFB with every case.

* Starting to push the new derived type through the argument list.

* The new dervied type, now stats_metadata, has been fed through the argument list.

* Moved l_stats_samp into stats_metadata

* Moved all the other stats_variables into stats_metadata

* Small cleanup

* Fixes a stats bug when calling compute_sample_mean in silhs.

* Arrays in stats_metadata need only be allocated once, but we may want to call the stats init routines to set up more stats variables. So now we check for allocation before allocating. Also made the stat_indices type publicly available through clubb_api.

* This should be initialized to 0. The new changes in cam were able to tease this out, which is a good sign.

* A couple more to initialize to zero.

* Moving stats_metadata to before stats_ variables in argument lists, just for consistency.

* Changing stat_indices to stats_metadata_type

stats_metadata needs to be threadprivate to make our standalone thread test work.

Changing order of sqrt and **3 to improve stability, see clubb#1126. Since this is a change to the order of operations, it is BIT_CHANGING.

Setting the minimum tolerance to be epsilon, this is BIT_CHANGING but only for single precision. See larson-group/clubb#1126

BIT_CHANGING! Updated taus code calculations - CLUBB ticket #1115 (#1125)

- Changed invrs_tau_xp2/wp2 calculations to match the wpxp calculation:
   Those now have a brunt_freq_pos and a brunt_freq_out_cloud term
- Changed tmp_calc magic number from .007 to .001
- Changed upper clipping bound for Ri_zm from 12 to 2 in invrs_tau_wpxp calculation
- Reversed accidental changes to input_misc/tuner/error_four_cases.in

Added non-trapezoidal zm interpolation for cloud_frac (#1127)

- Added a calculation for `cloud_frac_zm` when `l_call_pdf_closure_twice=false` and `l_trapezoidal_rule_zt=false`
This is BIT_CHANGING if `l_call_pdf_closure_twice=false` and `l_trapezoidal_rule_zt=false` since, in that case, `cloud_frac_zm` is now nonzero.
- Some minor formatting fixes
- Some minor fixes to pyplotgen's source code
- Added `a3_coef_min` to Lscale tunable_parameters.in

Added new utility script print_tunable_parameters_table.py (#1130)

Re-adjusted cap for invrs_tau_wpxp in src/CLUBB_core/mixing_length.F90
BIT_CHANGING! CLUBB ticket #1115

Adding boundary conditions for the wp3_term_ta_ADG1_lhs subroutine.
The value of wp3_zm at the model lower and upper boundaries is now
assumed to be 0 for simplicity and conservation.

For single loop vectors, removing the "collapse(2)" from the acc statements.

This commit is not bit changing, but the previous commit to this branch
was BIT_CHANGING:902d6caed8a4533f59f4165106276959b4f90895

Added new Brunt-Väisälä frequency plots to pyplotgen's VariableGroupTaus
CLUBB ticket #1132, BIT_CHANGING because new output fields were added
- Added 3 new Brunt-Väisälä output fields:
    - bv_freq_sqd_smth: The smoothed bv_mixed which is used to calculate the invrs_taus
    - bv_freq_pos: The positive part of the sqrt of bv_freq_sqd_smth
    - bv_freq_out_cloud: The out-of-cloud part of bv_freq_pos
- Added the plots of those fields to VariableGroupTaus

Adds commas to fix bug

QuadTune: updates setup in order to read in 30x30
regional files automatically.

For #910.

Updating the code to set the bottom of the thermodynamic level grid for
diffusion at thermodynamic level 2 -- the first level above the surface.

This is being done in preparation for ghostless gridding.

Commit to enforce a fixed-point boundary condition of wp3=0 at
thermodynamic level k = 2, the first above-ground thermodynamic level.

This is being done in preparation for ghostless gridding.

This commit is BIT_CHANGING.

Results are much more similar to the previous results than using
using zero-flux as the lower boundary condition for wp3.

With wp3 now being set to a fixed-point boundary condition value
of 0 at the current thermodynamic level 2 (first t-lev above the
surface), it is not necessary to loop over level 2 within the code
for each wp3 term.

This commit is BIT_CHANGING, but only for the wp3 budget terms
(e.g. wp3_ta) in the zt output file at level k = 2. It doesn't change
the actual answer. All other variables are bit-for-bit identical.

Threadpublic (#1133)

* Initial threadprivate removals and simple changes.

* model flags update, incomplete

* Some updates, almost working

* Added saturation_formula and l_host_applies_sfc_fluxes to clubb_config_flags

* Accompanying changes to flags lists

* Changes to corrrelation array setup

* Adding more flag checking

* Forgot to set l_host_applies_sfc_fluxes in initialize_clubb_config_flags_type and commit adding l_implemented to the argument loit

* Adding saturation parameters to api

* Most of the way through passing sclr_dim, hydromet_dim, edsclr_dim, and sclr_tol through arguemnt lists. Moved first_write breaking silhs stats aparently, and bit changing due to the way we were writing sclr_tol for cases that have sclr_dim=0 (arm different, gabls2 BFB).

* Removing accidentally commited files.

* Fixed silhs output, all cases BFB but sclr_tol is saved differently to stats files in cases where sclr_dim = 0, so this is bc also bc:fb20458d5212b007c79c17613201af637bbc3d7e since that introduced the difference.

* Changing stats logicals back to false, only set to true for testing. So technically BIT_CHANGING

* Starting array_vars changes.

* No more global scalar indices

* Starting to remove hydromet indices

* Finished the annoying part with the microphysics interfaces.

* No more global variables in array_index. All cases BFB relative to last commit.

* No more global pdf_dim, Ncnp2_on_Ncnm2 now set via argument list, and added temporary hydromet_dim_glob because I thought it would fix a bug but it didn't, it might in the future though, so I left it in for now, it will be removed soon.

* Added nan checking before overriding default value for Ncnp2_on_Ncnm2

* Finished hydromet_dim argument listing and removed global versions of it as well as sclr_dim, edsclr_dim, and sclr_tol.

* open_netcdf_for_writing does nothing if nvars=0 for a stats file, which is understandable, but now that we call first_write immediately after we need that also to check for nvar=0

* Adding better error messages to stats_init

* The nvidia compiler does not like zero size arrays.

* Adding defaults for scalar arrays values in clubb driver, and adding an error check after calling stats_init.

* Reverting accidental changes to clubb_thread_test

* Changing name of scarr/scalar_array to sclr_idx and hydromet_array to hm_metadata

* Adding new types to _api

* Fixing erroneous name changes.

* Fixing bug introduced in 65987c8a77496f7bb10bfd28717345b1f63de1b7

* Removing print

* QuadTune: updates setup in order to read in 30x30
regional files automatically.

For #910.

* Updating the code to set the bottom of the thermodynamic level grid for
diffusion at thermodynamic level 2 -- the first level above the surface.

This is being done in preparation for ghostless gridding.

* Commit to enforce a fixed-point boundary condition of wp3=0 at
thermodynamic level k = 2, the first above-ground thermodynamic level.

This is being done in preparation for ghostless gridding.

This commit is bc.

Results are much more similar to the previous results than using
using zero-flux as the lower boundary condition for wp3.

* With wp3 now being set to a fixed-point boundary condition value
of 0 at the current thermodynamic level 2 (first t-lev above the
surface), it is not necessary to loop over level 2 within the code
for each wp3 term.

This commit is BIT_CHANGING, but only for the wp3 budget terms
(e.g. wp3_ta) in the zt output file at level k = 2. It doesn't change
the actual answer. All other variables are bit-for-bit identical.

* Removing omp parallel test line

* Removing parameter declarion around l_quintic_poly_interp. Having this be a parameter causes bit changes when using -O2

---------

Co-authored-by: vlarson <vlarson@uwm.edu>
Co-authored-by: bmg929 <bmg2@uwm.edu>

Removing awkward nan check in favor of a default for Ncnp2_on_Ncnm2

Initial creation of gpu vs cpu output comparison jenkins test

Adding clean step before compiling GPU to new jenkins test

Making script check the specific multicore files

Adding more cases to test

Adding new argument list variables to openacc copyin statements

Limiting gabls2 to 500 timesteps in new jenkins test

Changing sclr test case to cobra, keeping 500 timestpes, gabls2 appears less stable as a result of commit 77ce77409e9da770515f66bbb50a869da74aca7d

Setting l_smooth_Heaviside_tau_wpxp to .true. by default

Undoing accidental changes to compiler config

BIT_CHANGING:1ce764cad4ea41e739b6ce9d95d8bec7db72097f

Making nightly cpu vs gpu test use a new config script for the cpu compile. This is the minimize the differences to be only whether or not the openacc GPU code is enabled.

Limiting mpace_b to 250 timestpes for gpu vs cpu test. The mpace_b test passed on my home GPU (RTX 3080), but not when run in jenkens on nelson (P4000). So either the tolerance is too strict, or we're running too many timesteps for mpace_b. Currently the largest difference on mpace_b multicol check is 3.31551e-05 on jenkins, so if this fails also I'll up the tolerance for mpace_b instead of further reducing timesteps run.

I am altering the mean advection code in preparation for ghost point
removal, since the mean advection code incorporates the ghost point
in its current calculations.

The lower boundary is now treated in the same manner as the
upper boundary, where a zero derivative over the boundary is assumed.
Any interaction of the current ghost level with the model code is
now removed from the mean advection calculation.

The results are bit-for-bit for most cases, but not all of them.
They are not necessarily expected to be bit-for-bit. There are
no discernable differences on plots of priority cases. However,
this commit is technically BIT_CHANGING.

Fixing bug. See https://github.com/larson-group/clubb/issues/1139

Fixing tridiag solver bug. See https://github.com/larson-group/clubb/issues/1139

Slight rework to acc data statements. This allows us to use return calls in advance_clubb_core again. See https://github.com/larson-group/clubb/issues/1139

Fixing small GPU stats bug

Changing xm to out in mono flux solver. There is no need for it to be inout, and it confused me for a bit so I'm fixing it.

Adding clubb specific compiler flag -DCLUBB_GPU, intended for use with openmp or openacc running on the GPU. Also adding a compiler flags, -DCUDA, and case statment for generating random numbers in silhs.

Adding new compiler flags to nvfortran GPU compile script

Adding openacc to openmp conversion script, and making some new compiler config files.

Adds comments to openmp conversion script

Now that the value of wp3 is being set to a boundary value of 0
at the first thermodynamic grid level above the surface, it doesn't
make sense to loop over budget terms anymore at that particular level,
since they don't factor in the solution and give a false impression.

With this commit, wp3 budget terms now start their loop at level 3.

This commit is BIT_CHANGING, but the only thing that changes are
budget stats for a couple of wp3 budget terms. The solution
does not actually change.

This is (amazingly) BIT_CHANGING. The differences are due to some compiler optimizations, see https://github.com/larson-group/clubb/pull/1133\#issuecomment-1865156979.

Removing git log print before logging to file. On some systems/configs the command would cause a busy wait until the person running hit q, and it is superfluous since we are already saving the git log to a file.

Removing backslashes. These should be slashes, but gnu,intel,nvidia generate code that treats slashes as whitespace and backslashes as a character, while cray treats backslashes as whitespace as well. This is easily fixed by using quotes, and in the meantime we will change the backslashes to forward slashes. See https://github.com/larson-group/clubb/issues/1138\#issuecomment-1910844013

Adding ifdefs around multicol netcdf outputting.

These should've been single quotes. I didn't tested a files with single quotes, then for some reason did all the changes as double quotes and only retested with gfortran. RESOLVED:9b495acd26a0b009ec84845aa289c36befff0c41