Changes

Clubb Driver Semi GPUization Part4 (#1209)

* Initial GPUization, and testing multi_col output method as single precision.

* advance_clubb_core tweaks

* Adding radht acc update

* Updates

* Adding new flags to the mono_flux test to prevent CPU and GPU divergence which breaks it.

* Small changes and improvements.

* Making new monoflux test lines

* Reworking some SILHS GPUization to make it more similar to the GPU code in the rest of clubb. This adds some extra parts that run on GPUs, so it is BIT_CHANGING

* Cleanup and comment update

* Cleanup

* Removing DCUDA flag from compile config scripts to help GPU and CPU results match for silhs cases

* Removing accidentally added file.

* Updating script

* Updating script

* Updating script

* Updating tolerance in script to handle rico_silhs differences, and hopefully final GPU updates

* Small cleanup

* Adding option to multi_col diff check script to scale the differences by the field avg. This is only needed (so far) for thlm differences that are slightly too large

* Testing gpu lock method

* Adding GPU locking to rest of GPU tests.

* Small fixes

QuadTune: Plot a map of the change in the loss function for each region.

This map tells us which regions are improved and which are degraded.  I.e., it displays the tuning trade-offs.

For #910.

Fixed time-height plot colormaps (#1210)

CLUBB ticket #1164

Added a "quick" fix for the generation of the time-height plot colormaps when variables are not correlations (i.e. values outside of [-1;1])
Updated VariableGroupCorrelations to include all correlations that are usually output in CLUBB.

Multi col output update (#1211)

* Multi_col output update to enable averaging over timesteps and outputting with single precision. This replaces the timestep batched output method used before, which was only noticably faster at low column numbers. This is technically BC (edited during merge, PR no longer BC), because the single precision output will be interpretted as different from the current double precision output.

* Small cleanup

* Script updates

* Script update

* Script update

* Adding capability to output multi_col as double or single precision using new flag l_output_double_prec.

* Removing accidentally added file.

* Making new jenkins script run single and double precision tests.

* Update to jenkins test

QuadTune: Added colorbars to regional map plots.

For #910.

QuadTune: 1) Change coloring of scatterpoints to represent either loss changes or residuals. 2) Add map of residuals to complement map of bias.

For #910.

Likely non-bit-changing housekeeping (#1213)

Non-bit-changing housekeeping regarding wp2 clipping
CLUBB ticket #1165

Renamed threshold_array -> wp2_min_array
Moved upper clipping of wp2 into clip_variance
Added warning outputs where wp2 is hard-clipped
Added debugging checks to hole-filling and wp2_min_array clipping
Some necessary OpenACC-modifications

Fix pyplotgen timeheight cmaps (#1212)

* Fixed time-height plot colormaps
CLUBB ticket #1199

Added a "quick" fix for time-height plot colormaps when variables are not correlations (i.e. values in [-1;1])
Updated VariableGroupCorrelations to include all correlations that are usually output in CLUBB.

* Fixed time-height plot colormaps
CLUBB ticket #1199

Fixed distinction for colormaps in time-height plots and added tick formatter.
We now check the labels for the string "corr" to find correlation variables.
Otherwise we just use one-sided or two-sided normed colormaps.

QuadTune: 1) Add bar chart that shows |metric_changes| due to each parameter. 2) Fix colorbars on maps.  3) Allow download of hi-res figures.  4) Make background of scatterplots grey.

For #910.

Reduced the upper bound for wp2_min_array to 1.0 (#1214)

CLUBB ticket #1165

This is BIT_CHANGING for at least the cases ARM_97 and TWP_ICE,
and likely any other more non-standard cases.
This change will prevent the minimum clipping value for wp2 from exploding when l_min_wp2_from_corr_wx=.true.

Added up2 and vp2 to calculation of wp2_min_array (#1215)

CLUBB ticket #1165

This is BIT_CHANGING for the cases ARM_97, ATEX, CGILS_S6, GABLS3_NIGHT, LBA, MPACE_B, and TWP_ICE out of the priority cases.
These lines of code were added since the fields `u`, `v`, `rt`, and `thl`, and their correlations with `w` should be treated equally.

QuadTune: Add bar chart showing linear vs. nonlinear contributions of each parameter.

For #910.

Changed default setting of flag l_min_wp2_from_corr_wx to false (#1216)

CLUBB ticket #1165

BIT_CHANGING
Changed default setting in configurable_model_flags.in and model_flags.F90
to decrease clipping contribution to `wp2` and increase clipping to `rtp2` and `thl2`.

QuadTune: Add map plots of "linSoln" output from global E3SM runs.

For #910.

QuadTune: Include special regions (DYCOMS, HAWAII, etc.), in addition to 20x20 regions.  However, the weights are set to epsilon, which wrecks the loss calculation.  This will have to be remedied later.

For #910.

Recoded a loop within the mono flux limiter so that it's written
in a more convenient manner to allow for easier coding for
generalized gridding.

Results are bit-for-bit.

Changed some variable names so that it is clear that, while they contain
the word wpxp_, they are actually thermodynamic-level variables.

Results are bit-for-bit.

Within the mono flux limiter, a thermodynamic-level variable needed to
be interpolated to momentum levels so it could be directly compared
to wpxp.

The results of this change are BIT_CHANGING for the ARM 97, MC3E,
and TWP-ICE cases.

I forgot to add the new variable to the Atlantic Coast Conference
statements in the previous commit.

Changing a couple loop limits in mono flux limiter.

Results do not change.

QuadTune: Remedy loss function and facilitate plotting of special, non-20x20 regions (DYCOMS, HAWAII, etc.).

For #910.

Fix binormal correlation clipping (#1199)

Fixed clipping in binormal correlation calc
CLUBB ticket #1164

BIT_CHANGING

* Implemented new denominator with smooth_max clipping
* Created subroutine `smooth_corr_quotient` in pdf_utilities.F90, outsourcing smooth correlation division
* This subroutine is used `calc_comp_corrs_binormal` and `transform_pdf_chi_eta_component`

Added new hole-filling method which takes TKE from up2 and vp2 (#1217)

Added new hole-filling method which takes TKE from up2 and vp2
CLUBB ticket #1165

BIT_CHANGING if flag `l_wp2_fill_holes_tke` is set to true
Introduced new flag `l_wp2_fill_holes_tke` to turn on new wp2 hole-filling subroutine
that takes TKE from up2 and vp2.

Finally fixing gpu_diffness test (#1219)

* Modifying diffness test to handle fields with < 0 averages and accept an argument as the threshold.

* Overwriting correct variable.

Slightly relaxing the convergence criteria to fix convergence test.

QuadTune: On matrix-eqn bar chart, plot only extraMetricsToPlot.  Add sens-bias scatterplot of only extraMetricsToPlot.  In parameter increments bar chart, change absolute values to squares.

For #910.

QuadTune: Add useLongTitle option, which adds extra information on the plotted quantity, if desired.

For #910.

QuadTune: Plot bias instead of negative bias on metricsBarChart and bias-vs-sensitivity plots.

For #910.

Monoflux GPU optimization (#1221)

* Making a fast version of the serial loop to check if we even need to perform the slow version at all.

* Updates

* Final touches to make new version logically the same.

* Improving comments and variables names

QuadTune: Made plot_PcSensMap work when len(varPrefixes) > 1.

For #910.

QuadTune: Merge in bootstrap sampling

@LuisHasenauer added bootstrap sampling code to QuadTune.  Bootstrap calculations are performed if `useBootstrap=T`.  Luis and I checked that the best-fit parameter values and weighted chiSqd are unchanged either with `useBootstrap=T` or `useBootstrap=F`. 

Swapping the loop order on a few kernels, I found this faster for the nvhpc compiler and it's BFB

BIT_CHANGING bug fix that only affects l_t_dependent cases running with multiple columns, I detected this with cgils cases, but I'm not sure how it slipped through initially.

Set l_wp2_fill_holes_tke to true by default (#1226)

This code change is BIT_CHANGING for extreme cases (e.g. TWP_ICE, LBA, ARM_97).
Changed configurable_model_flags.in files and default value in model_flags.F90.

For CLUBB ticket #1165.

Sneaky commit

Fixing ordering of parameter lists in clubb_driver