Started by an SCM change00004438284417261.0959095911453418958795875839429ba2f18a79e7b17ef53bccc8f17af6649c3fd9429ba2f18a79e7b17ef53bccc8f17af6649c3fdrefs/remotes/origin/master9429ba2f18a79e7b17ef53bccc8f17af6649c3fd9429ba2f18a79e7b17ef53bccc8f17af6649c3fdrefs/remotes/origin/master9429ba2f18a79e7b17ef53bccc8f17af6649c3fd9429ba2f18a79e7b17ef53bccc8f17af6649c3fdrefs/remotes/origin/masterhttps://github.com/larson-group/clubb.githttp://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/583/artifacthttp://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/changeshttp://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/583/pipeline-overviewhttp://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/583/testReportfalseUpdate sys_admin RepoRun Python Testfalse#583443828486675clubb_BFB_e3sm_flags_gfortran_test #583583false58310293SUCCESS1791533770837http://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/583/src/clubb_driver.F909429ba2f18a79e7b17ef53bccc8f17af6649c3fd1791488049000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comAlways do idiot-proofing of flags in check_clubb_settings_api Changed debug level check from 1 to 0 for parameter sanity checks. 2026-10-08 14:34:09 -05009429ba2f18a79e7b17ef53bccc8f17af6649c3fdAlways do idiot-proofing of flags in check_clubb_settings_apieditsrc/clubb_driver.F90githttp://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreplyfalseStarted by an SCM change00006152176140761.01521152115844292a70f53a6b6c4eab864fe27b8fd9af5d869094292a70f53a6b6c4eab864fe27b8fd9af5d86909refs/remotes/origin/master4292a70f53a6b6c4eab864fe27b8fd9af5d869094292a70f53a6b6c4eab864fe27b8fd9af5d86909refs/remotes/origin/master4292a70f53a6b6c4eab864fe27b8fd9af5d869094292a70f53a6b6c4eab864fe27b8fd9af5d86909refs/remotes/origin/masterhttps://github.com/larson-group/clubb.gitfalse#584615217486675clubb_BFB_e3sm_flags_gfortran_test #584584false58410388SUCCESS1791879366475http://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/584/.agents/skills/clubb-change-messages/SKILL.mdLLM_prompts/SHORTCUTS.mdAGENTS.mdLLM_prompts/change_messages.mdfc9087d1a7a4a8f71c74f5763ece34f18328d2001791830025000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comAdd CLUBB commit and PR writing guidance (#1391) * Add CLUBB commit and PR writing guidance Add a canonical workflow and historical examples for writing specific titles and explaining purpose, motivation, results and testing. Scope BFB claims to their comparisons and distinguish tolerance matches from bitwise equality. A thin repo skill, shortcut and AGENTS link make the workflow discoverable. This addresses generated messages that bury the reason for a change or overstate what testing establishes. Six historical PR/commit drafting pairs passed independent review. Skill format, 20 local links, canonical-text consistency and git diff --check passed. BFB is not applicable to these documentation-only changes. * Keep only the final commit and PR writing workflow Remove the worked historical examples and their references. The repository should contain the final reusable guidance; the research and evaluation artifacts stay outside it. The shortcut and repo skill still point to one canonical workflow. Skill-format validation, all 19 local guidance links and git diff --check passed. BFB is not applicable to these documentation-only changes. 2026-10-12 13:33:45 -0500fc9087d1a7a4a8f71c74f5763ece34f18328d200Add CLUBB commit and PR writing guidance (#1391)editAGENTS.mdadd.agents/skills/clubb-change-messages/SKILL.mdaddLLM_prompts/change_messages.mdeditLLM_prompts/SHORTCUTS.mdclubb_python_api/pytests/auto_llm_generated_pytests/test_advance_core_subs.pyclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_xp2_xpyp.pyclubb_jax/src/Benchmark_cases/cobra.pyclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_wp2_wp3.pyclubb_python_api/clubb_python/prescribe_forcings.pyclubb_jax/src/CLUBB_core/advance_windm_edsclrm_module.pysrc/Benchmark_cases/time_dependent_input.F90clubb_python_api/clubb_python/CLUBB_core/parameters_tunable.pyclubb_python_api/clubb_python/CLUBB_core/numerical_check.pysrc/CLUBB_core/advance_windm_edsclrm_module.F90clubb_jax/pytests/auto_llm_generated_pytests/test_driver_lifecycle.pyclubb_python_driver/advance_clubb_core.pyclubb_jax/src/Benchmark_cases/time_dependent_input.pyutilities/pytests/auto_llm_generated_pytests/test_expldiff_settings_rules.pyclubb_python_api/f2py_fortran_wrappers/advance_windm_edsclrm_module_wrapper.F90clubb_jax/src/CLUBB_core/advance_clubb_core_module.pyclubb_jax/pytests/auto_llm_generated_pytests/test_expldiff_edsclrm_forcing.pyclubb_python_api/clubb_python/CLUBB_core/advance_windm_edsclrm.pyclubb_jax/pytests/auto_llm_generated_pytests/test_pressure_coord_forcing.pyclubb_python_api/pytests/auto_llm_generated_pytests/test_init_sequence.pyclubb_jax/src/clubb_case_initalization.pyclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_windm_edsclrm.pyclubb_jax/pytests/auto_llm_generated_pytests/test_windm_edsclrm_scalar_indices.pyexperimental/experimental_pdfs/python/dash_additions/misc_labs/sam_lscale.pysrc/CLUBB_core/numerical_check.F90clubb_python_api/clubb_python/derived_types/nu_vert_res_dep.pyclubb_python_driver/advance_clubb_to_end.pyclubb_python_driver/clubb_case_initalization.pysrc/CLUBB_core/advance_clubb_core_module.F90clubb_jax/pytests/auto_llm_generated_pytests/test_cobra_sfclyr.pyclubb_jax/src/Benchmark_cases/prescribe_forcings.pyclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_core.pysrc/clubb_driver.F90utilities/clubb_settings_validation.pytests/run_clubb_settings_validation_test.pysrc/Benchmark_cases/prescribe_forcings.F90clubb_python_api/clubb_f2py.pyfclubb_jax/src/CLUBB_core/numerical_check.pyclubb_jax/src/advance_clubb_to_end.pysrc/Benchmark_cases/cobra.F90clubb_python_api/f2py_fortran_wrappers/numerical_check_wrapper.F90clubb_jax/pytests/auto_llm_generated_pytests/test_forcing_port_structure.pyclubb_python_api/f2py_fortran_wrappers/prescribe_forcings_module_wrapper.F90clubb_jax/src/CLUBB_core/err_info_codes.pyclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_xm_wpxp.py4292a70f53a6b6c4eab864fe27b8fd9af5d869091791843053000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comFix arm97+l do expldiff rtm thlm+set edsclrm f early (#1386) * Do not force the thlm/rtm copies in the expldiff eddy-scalar solve With l_do_expldiff_rtm_thlm, advance_windm_edsclrm copies thlm into edsclrm(:,:,edsclr_dim-1) and rtm into edsclrm(:,:,edsclr_dim), advances all eddy scalars, and copies those two slots back into thlm and rtm. The solve added edsclrm_forcing of each slot to the copies, but thlm and rtm are already advanced with thlm_forcing and rtm_forcing in advance_xm_wpxp. In the standalone driver the slots are filled by iiedsclr_thl/iiedsclr_rt: - arm_97 (iiedsclr_rt = 1, iiedsclr_thl = 2) put thlm_forcing [K/s] on rtm, so rtm < 0 at ~5.5-8.8 km within the first steps (flag11 / expldiff). - With the slots in expldiff order, rtm_forcing was applied twice and arm_97 still hit rtm < 0 near 12 km after 1379 of 5580 steps. The copies now get no forcing. Mirrored in the JAX port. Only changes runs with l_do_expldiff_rtm_thlm = .true. and non-zero edsclrm_forcing in the last two eddy scalars, therefore BIT_CHANGING for those. Default runs are bit-for-bit identical (arm_97, 1076 variables). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * not hardwire the thlm and rtm edsclrm variables to be the last two; instead use sclr_idx%iiedsclr_thlm and sclr_idx%iiedsclr_rtm Previously l_do_expldiff_rtm_thlm = true forced the last slot to be rtm and the one before that to be thlm. If in the ..._model.in file iiedsclr_rt = 1 ! Location in the edsclrm array to place total water and iiedsclr_thl = 2 ! Location in the edsclrm array to place a scalar like thetal a switcheroo happens. But if we use a not hardcoded indexing we need to check for out of bounds and distinctness of the 2 indices. * Zero edsclrm_forcings early in apply_time_dependent_forcings_from_array Check edsclrm_forcings != 0 instead of zeroing in advance_windm_edsclrm BIT_CHANGING for l_do_expldiff_rtm_thlm = .true. for arm_97 and for cobra. Python changes include: clubb_jax, clubb_python (+f2py) + tests * Check l_do_expldiff_rtm_thlm eddy scalars in check_clubb_settings_api 1. Move the eddy-scalar checks into check_clubb_settings_api check_clubb_settings_api now takes edsclr_dim and sclr_idx. With l_do_expldiff_rtm_thlm = .true. it stops the run if - edsclr_dim <= 1, since the copies of thlm and rtm need two eddy scalars, or - iiedsclr_thl and iiedsclr_rt are not distinct indices in 1:edsclr_dim. The index check moved to check_clubb_settings_api from advance_windm_edsclrm. The check that edsclrm_forcing is zero for the thlm/rtm copies stays in advance_windm_edsclrm, because prescribe_forcings sets edsclrm_forcing every time step, after the settings check has run. Callers in Fortran and mirrors (clubb-jax, python api) are updated Behavior change: l_do_expldiff_rtm_thlm = .true. with fewer than two eddy scalars used to skip explicit diffusion silently and now stops at setup. This affects flag sets that turn the flag on for every case, e.g. flag11 in input/flag_sets/run_bindiff_w_flags_config_core_flags.json. Host models must pass the new arguments; CAM must also set sclr_idx%iiedsclr_thl/iiedsclr_rt when clubb_l_do_expldiff_rtm_thlm is on, since it leaves them at -1. 2. Fix the Python API - calc_derived_params now requires nu_vert_res_dep and always pushes it to the F2PY storage (new helper NuVertResDep.zeros(ngrdcol)). Before, nu_vert_res_dep=None kept the previous call's stored arrays, and calc_derived_params only allocates them when unallocated. So a call with more columns than an earlier one in the same process wrote out of bounds. This crashed the in-process Python API pytest suite once the windm tests used two columns. - clubb_f2py.pyf was missing l_do_expldiff_rtm_thlm for f2py_prescribe_forcings, so the Python driver failed on every case. 3. Gate the JAX rerun settings check at debug level 0 set_case_initial_conditions in the JAX driver now runs check_clubb_settings at debug level >= 0, matching clubb_driver.F90 since 9429ba2f1. Also: put l_do_expldiff_rtm_thlm on its own line in the time_dependent_input/prescribe_forcings argument lists. Bit-for-bit: default-flag output is unchanged, and arm_97 with l_do_expldiff_rtm_thlm = .true. matches the previous commit exactly. Tested: debug + Python build and source checks; settings parity test; Python API pytests (237 passed in one process); JAX pytests (791 passed); utilities settings-rule pytests; python-vs-fortran and jax-vs-fortran bomex/atex; bomex and cobra with the flag stop at setup in the Fortran, Python and JAX drivers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Do not apply surface fluxes to the expldiff thlm/rtm eddy-scalar copies With l_do_expldiff_rtm_thlm, advance_windm_edsclrm copies thlm and rtm into the iiedsclr_thl and iiedsclr_rt eddy scalars, solves all eddy scalars with their surface flux wpedsclrp(:,gr%k_lb_zm,:) (from wpedsclrp_sfc) and copies the two slots back. thlm and rtm already get wpthlp_sfc and wprtp_sfc in advance_xm_wpxp, so a non-zero surface flux on the copies applies it twice. Same approach as the edsclrm_forcing fix: 1. cobra_sfclyr (the only case that sets these fluxes) takes l_do_expldiff_rtm_thlm and sets wpedsclrp_sfc of the thlm/rtm copies to zero when it is on. set_sclr_sfc_rtm_thlm already zeroes all eddy-scalar surface fluxes. 2. advance_windm_edsclrm stops with a fatal error if the surface flux of either copy is non-zero, after the edsclrm_forcing check. Mirrored in the JAX port (new reason code ERR_WINDM_EDSCLRM_SFC_FLUX_EXPLDIFF_COPY; marks the columns fatal without returning early, like the forcing check). No Python API wrapper changes. CAM zeroes all eddy-scalar surface fluxes, so it is unaffected. Default cobra output is unchanged. But technically BIT_CHANGING for cobra + l_use_expldiff_rtm_thlm * Compute cobra's CO2 surface flux per column cobra_sfclyr converted the CO2 surface flux with each column's surface density inside the column loop, but stored it in the scalar CO2_flx2, and the later loops gave every column that one value: - CPU: every column got the last column's CO2 flux. Standalone columns currently share one density profile, so results were unaffected. - OpenACC: the scalar was private to the device loop, and the later loops copied the never-set host value (implicit firstprivate). In the GPU build cobra's CO2 surface flux was 0 at every step. CO2_flx2 is now a per-column array (enter/exit data create/delete like other case routines). The JAX port already computes it per column. Bit-for-bit: CPU cobra output is unchanged (1 and 2 columns, standard stats incl. the CO2 scalars). GPU cobra output changes: before, the CO2 surface flux was 0 and CO2 differed from the CPU run by up to 10%; now the CO2 scalars match the CPU run to ~1e-11 relative, like the other fields. Tested: gfortran debug build and source checks; standalone check of cobra_sfclyr with two columns of different density (before: column 1 got column 2's flux); nvhpc 24.3 OpenACC build and cobra on a Quadro P4000. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * removed the idiot proofing * fully removed idiot proofing for l_do_expldiff_rtm_thlm+esclrdim --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> 2026-10-12 17:10:53 -05004292a70f53a6b6c4eab864fe27b8fd9af5d86909Fix arm97+l do expldiff rtm thlm+set edsclrm f early (#1386)editclubb_python_api/clubb_python/CLUBB_core/advance_windm_edsclrm.pyeditclubb_jax/src/Benchmark_cases/cobra.pyeditclubb_jax/src/Benchmark_cases/prescribe_forcings.pyeditclubb_jax/pytests/auto_llm_generated_pytests/test_driver_lifecycle.pyeditclubb_python_api/clubb_f2py.pyfeditclubb_jax/pytests/auto_llm_generated_pytests/test_cobra_sfclyr.pyeditclubb_jax/src/CLUBB_core/err_info_codes.pyeditclubb_python_api/clubb_python/prescribe_forcings.pyeditclubb_python_api/f2py_fortran_wrappers/advance_windm_edsclrm_module_wrapper.F90editclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_xp2_xpyp.pyeditutilities/clubb_settings_validation.pyeditclubb_python_api/f2py_fortran_wrappers/prescribe_forcings_module_wrapper.F90editclubb_python_driver/advance_clubb_to_end.pyeditsrc/Benchmark_cases/cobra.F90edittests/run_clubb_settings_validation_test.pyeditclubb_jax/src/CLUBB_core/numerical_check.pyeditclubb_jax/src/clubb_case_initalization.pyeditsrc/Benchmark_cases/time_dependent_input.F90editclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_core.pyeditclubb_jax/pytests/auto_llm_generated_pytests/test_pressure_coord_forcing.pyeditclubb_python_api/pytests/auto_llm_generated_pytests/test_init_sequence.pyeditclubb_python_driver/advance_clubb_core.pyeditsrc/Benchmark_cases/prescribe_forcings.F90editclubb_jax/src/advance_clubb_to_end.pyeditclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_xm_wpxp.pyeditclubb_python_driver/clubb_case_initalization.pyaddutilities/pytests/auto_llm_generated_pytests/test_expldiff_settings_rules.pyeditclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_windm_edsclrm.pyeditclubb_jax/src/Benchmark_cases/time_dependent_input.pyeditsrc/CLUBB_core/numerical_check.F90editsrc/clubb_driver.F90editsrc/CLUBB_core/advance_clubb_core_module.F90editclubb_python_api/clubb_python/derived_types/nu_vert_res_dep.pyaddclubb_jax/pytests/auto_llm_generated_pytests/test_expldiff_edsclrm_forcing.pyeditclubb_jax/pytests/auto_llm_generated_pytests/test_forcing_port_structure.pyeditclubb_jax/src/CLUBB_core/advance_windm_edsclrm_module.pyaddclubb_jax/pytests/auto_llm_generated_pytests/test_windm_edsclrm_scalar_indices.pyeditclubb_python_api/pytests/auto_llm_generated_pytests/test_call_tree_advance_wp2_wp3.pyeditsrc/CLUBB_core/advance_windm_edsclrm_module.F90editclubb_python_api/f2py_fortran_wrappers/numerical_check_wrapper.F90editexperimental/experimental_pdfs/python/dash_additions/misc_labs/sam_lscale.pyeditclubb_python_api/clubb_python/CLUBB_core/numerical_check.pyeditclubb_python_api/clubb_python/CLUBB_core/parameters_tunable.pyeditclubb_python_api/pytests/auto_llm_generated_pytests/test_advance_core_subs.pyeditclubb_jax/src/CLUBB_core/advance_clubb_core_module.pygithttp://carson.math.uwm.edu/jenkins/user/noreplynoreplyfalse585http://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/585/583http://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/583/Started by an SCM change00014223084201770.99192559255582e92c2f24486a32610935ff4eede3512c67e313eae92c2f24486a32610935ff4eede3512c67e313earefs/remotes/origin/mastere92c2f24486a32610935ff4eede3512c67e313eae92c2f24486a32610935ff4eede3512c67e313earefs/remotes/origin/mastere92c2f24486a32610935ff4eede3512c67e313eae92c2f24486a32610935ff4eede3512c67e313earefs/remotes/origin/masterhttps://github.com/larson-group/clubb.gitfalse#582422308486675clubb_BFB_e3sm_flags_gfortran_test #582582false58210213SUCCESS1791447370624http://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/582/tuner/loss_backend.pytuner/job_runtime.pytuner/pytests/auto_llm_generated_pytests/test_tuner_top_overrides.pyjenkins_tests/clubb_tuner/Jenkinsfileclubb_jax/pytests/auto_llm_generated_pytests/test_jax_cli_options.pydash_app/pytests/auto_llm_generated_pytests/test_tune_replay_completion.pyrun_scripts/README.mddash_app/shared/activity.pydash_app/pytests/test_jax_device.pyclubb_jax/run_jax.pydash_app/README.mdtuner/pytests/auto_llm_generated_pytests/test_jax_launch_errors.pytuner/request.pydash_app/compile_tab/callbacks.pydash_app/shared/actions.pydash_app/shared/jax_device.pyutilities/create_case_namelist.pydash_app/profile_tab/runtime.pydash_app/tune_tab/runtime.pytuner/tune_clubb.pyclubb_jax/pytests/auto_llm_generated_pytests/test_device_selection.pydash_app/run_tab/runtime.pytuner/pytests/auto_llm_generated_pytests/test_runtime_device_settings.pytuner/tuning_worker.pytuner/status.pyclubb_jax/backends/cuda.pydash_app/services/models.pydash_app/pytests/test_agent_services.pydash_app/pytests/auto_llm_generated_pytests/test_jax_tuning_runtime.pydash_app/tune_tab/callbacks_runs.pydash_app/DEVELOPMENT.mdtuner/tuning_scheduler.pydash_app/pytests/test_run_broker_simplification.pydash_app/compile_tab/build_selector.pytuner/README.mdtuner/pytests/auto_llm_generated_pytests/test_jax_backend.pyclubb_jax/README.mdrun_scripts/run_tuner_job.py9a1b8b23408d3a5abe78ebae7dd1390755fa4fec1791407526000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comUse JAX in tuner (#1383) This makes the JAX work through the new tuner, same sort of way the fortran does. Probably needs a little more testing but seems to be working properly now. * Support reusable JAX driver state and runtime batches * Add standalone JAX loss evaluation and consistency coverage * Keep standalone loss CLI coverage separate from tuning * Integrate JAX with managed tuner jobs and add bounded Jenkins coverage * Remove trailing whitespace from extracted loss CLI tests * Run JAX loss help checks in the JAX pytest suite * Keep JAX lifecycle coverage with native driver tests and document its origins * Document loss test scenarios, native counterparts and JAX extensions * Keep the native loss executable path intact in the test description * Explain managed tuning test ownership and native backend contracts * Mirror the native driver lifecycle test and compare complete JAX statistics * Run JAX driver extensions as an independent step in the lifecycle stage * Update the C8 comparison mutation for indexed parameter loading * Refresh JAX tuner integration for merged loss frontends Use the relocated standalone loss entry point and retain CLI checks only in their shared owner. Give each Jenkins tuning run a separate job directory so repeated builds can succeed. * Record invalid direct tuner requests as terminal errors Let the existing request and launcher error handlers report malformed JSON and non-string JAX options instead of crashing during bootstrap before writing job results. * Preserve winning parameter columns in tuner reruns Retain request physics overrides while excluding initialized values for tuned parameters. Window and full-case reruns now use the winning candidate values, including multiple columns and grouped override names. * Use the selected JAX runtime throughout Dash tuning Persist backend and per-job device settings through native and typed broker requests, saved revisions, continuation and result reruns. Reuse the launcher, shared device controls and rerun override resolver; supervise browser replay processes in the broker. * Allow run IDs in broker replay completion metadata Avoid a selector argument collision that killed the replay watcher before recording terminal state. Preserve run IDs and cancellation metadata while reporting completed jobs accurately. * Keep JAX device setup inside the launcher and backends Expose a standard-library selection interface and forward per-job device and preallocation settings through the SCM and tuner launch paths. Use the same backend configuration for discovery, provenance, and execution; remove GPU environment handling from Dash and the tuner. Validate with Dash/shared suites, focused launcher checks, a CPU tuning and replay workflow, and synthetic two-GPU discovery. * Forward device settings through the existing JAX selection option Extend the launcher-owned -options grammar with device=UUID and prealloc_gpu_mem=true|false, while retaining xla_prealloc compatibility. Remove the transient device-specific options from SCM and tuner scripts so they forward one opaque -jax value. Validate saved settings, discovery and replays through the same public interface; verify CPU tuning metrics remain identical. 2026-10-07 16:12:06 -05009a1b8b23408d3a5abe78ebae7dd1390755fa4fecUse JAX in tuner (#1383)editdash_app/shared/actions.pyeditdash_app/README.mdadddash_app/pytests/auto_llm_generated_pytests/test_jax_tuning_runtime.pyedittuner/README.mdaddclubb_jax/pytests/auto_llm_generated_pytests/test_device_selection.pyeditdash_app/tune_tab/runtime.pyedittuner/job_runtime.pyeditclubb_jax/README.mdeditclubb_jax/run_jax.pydeletedash_app/shared/jax_device.pyaddtuner/pytests/auto_llm_generated_pytests/test_jax_launch_errors.pyeditdash_app/run_tab/runtime.pyeditclubb_jax/backends/cuda.pyeditdash_app/tune_tab/callbacks_runs.pyaddtuner/pytests/auto_llm_generated_pytests/test_runtime_device_settings.pyeditdash_app/DEVELOPMENT.mdeditrun_scripts/run_tuner_job.pyadddash_app/pytests/auto_llm_generated_pytests/test_tune_replay_completion.pyeditdash_app/profile_tab/runtime.pyeditdash_app/compile_tab/build_selector.pyedittuner/status.pyaddtuner/pytests/auto_llm_generated_pytests/test_tuner_top_overrides.pyeditutilities/create_case_namelist.pyeditdash_app/services/models.pyeditrun_scripts/README.mdeditdash_app/pytests/test_run_broker_simplification.pyeditdash_app/compile_tab/callbacks.pyeditjenkins_tests/clubb_tuner/Jenkinsfileedittuner/tuning_scheduler.pyedittuner/tune_clubb.pyaddtuner/loss_backend.pyedittuner/request.pyeditclubb_jax/pytests/auto_llm_generated_pytests/test_jax_cli_options.pyeditdash_app/pytests/test_agent_services.pyaddtuner/pytests/auto_llm_generated_pytests/test_jax_backend.pyeditdash_app/shared/activity.pyedittuner/tuning_worker.pyeditdash_app/pytests/test_jax_device.pyrun_scripts/run_tuner.pydash_app/run_tab/tab.pydash_app/persistence.pyutilities/README.md.gitignoredash_app/run_tab/callbacks_selection.pydash_app/run_tab/layout.pyutilities/case_json_to_namelist.pydash_app/pytests/auto_llm_generated_pytests/test_case_editor.pydash_app/README.mdinput/case_setups/case_definitions.jsondash_app/assets/34_case_editor.jsdash_app/shared/actions.pyutilities/pytests/auto_llm_generated_pytests/test_case_json_to_namelist.pyrun_scripts/run_scm_loss.pydash_app/run_tab/discovery.pydash_app/assets/12_case_editor.cssdash_app/assets/11_tab_run_theme.cssutilities/create_case_namelist.pytuner/case_defaults.pydash_app/profile_tab/runtime.pyinput/case_setups/READMEdash_app/run_tab/cases.py5f8b57f69c01c59fc3c4457178ae85d57894c1d91791415324000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comAdd JSON case definitions and Dash case editor (#1389) This is mainly a dash update right now, but it adds json versions of our model files. Basically just 2 new files for that - each json stores default values then one entry for each case with overrides it is not default yet and the current system of defining cases is unchanged. * Add JSON case definitions and Dash case editor * Simplify named case lookup and remove queued snapshots * Remove model validation from JSON case conversion 2026-10-07 18:22:04 -05005f8b57f69c01c59fc3c4457178ae85d57894c1d9Add JSON case definitions and Dash case editor (#1389)editdash_app/run_tab/tab.pyeditrun_scripts/run_tuner.pyadddash_app/assets/12_case_editor.cssadddash_app/pytests/auto_llm_generated_pytests/test_case_editor.pyeditdash_app/shared/actions.pyaddinput/case_setups/case_definitions.jsoneditdash_app/run_tab/layout.pyadddash_app/assets/34_case_editor.jseditdash_app/run_tab/callbacks_selection.pyeditdash_app/README.mdedittuner/case_defaults.pyaddutilities/pytests/auto_llm_generated_pytests/test_case_json_to_namelist.pyeditdash_app/profile_tab/runtime.pyeditdash_app/run_tab/discovery.pyeditdash_app/assets/11_tab_run_theme.csseditdash_app/persistence.pyaddutilities/case_json_to_namelist.pyeditutilities/create_case_namelist.pyeditinput/case_setups/READMEeditrun_scripts/run_scm_loss.pyadddash_app/run_tab/cases.pyeditutilities/README.mdedit.gitignoretests/README.mdtests/pytests/auto_llm_generated_pytests/test_run_stats_output_consistency.pyjenkins_tests/clubb_stats_output_consistency/Jenkinsfiletests/run_stats_output_consistency.py166ec211468128c37d8bb4b917ffac337bf802471791416548000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comRun JAX statistics consistency with an explicit rounding tolerance (#1385) This adds a jax jenkins stage for the statistics consistency test. This is the test that runs clubb with different batches sizes and averaging methods, e.g - start by: run once with 60s output intervals all the way through - test: running with 300s output intervals should give the same stats as manually averaging 5 steps of 60s output - test: outputing only a "window" of stats data (like from timesteps 100-200 instead of whole run), should produce a window idendical to the 60s full output run - test: running 16 columns at once should produce the same as running 16 columns in batches of 4 at a time In fortran these comparisons should be BFB (other than the manually averaging one, which is only close), but in jax, changing things like the batch size can cause optimizations to happen differently, which breaks BFBness and seems to cause differences larger than expected by fortran. For example, the threshold for the manual averaging test in fortran is 1e-12, but for the jax we need 1e-9 even for the batch test. I added a tolerance, `jax_tolerance`, that I left named after jax for now since that is the only current reason that flag exists. This difficulty with jax wasn't unexpected - we also have to compile the fortran with `-O0` to get this test working, also to disable optimizations. There just seems to be less control over how jax does that, so I went with a "relax the tolerance for this specific instance" approach. * Support reusable JAX driver state and runtime batches * Expose JAX in the strict statistics consistency workflow * Keep JAX lifecycle coverage with native driver tests and document its origins * Reference native driver and statistics owners in the JAX test workflow * Keep full Fortran statistics source paths in the test description * Mirror the native driver lifecycle test and compare complete JAX statistics * Run JAX driver extensions as an independent step in the lifecycle stage * Update the C8 comparison mutation for indexed parameter loading * Keep statistics windows within the capped native case duration * Allow explicit JAX statistics tolerance for JIT rounding differences 2026-10-07 18:42:28 -0500166ec211468128c37d8bb4b917ffac337bf80247Run JAX statistics consistency with an explicit rounding tolerance (#1385)addtests/pytests/auto_llm_generated_pytests/test_run_stats_output_consistency.pyeditjenkins_tests/clubb_stats_output_consistency/Jenkinsfileedittests/run_stats_output_consistency.pyedittests/README.mdtests/run_timestep_tests.pytests/README.mdjenkins_tests/clubb_timestep/Jenkinsfileclubb_jax/README.mdclubb_jax/run_jax.pyclubb_jax/pytests/auto_llm_generated_pytests/test_compilation_cache.pytests/run_timestep_sweep.pytests/pytests/auto_llm_generated_pytests/test_run_timestep_sweep.py341cac040a0d352fd5a4da6a5938a86d7d06e3321791417082000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comRun timestep sweep tests with JAX (#1384) Adding a jax run to the timestep test (now renamed timestep sweep because it's more descriptive). This was really slow on JAX at first - one trick was to add a compilation cache, so that different runs that only use different timesteps didn't need to be recompiled. From my undestanding this is super easy with jax, basically just set `JAX_COMPILATION_CACHE_DIR` to a directory the process can save compiled parts is all it takes, then the compiled parts can live there even after the processes dies, and future ones can reuse it. * Run existing timestep sweeps with the JAX runner * Explain the shared native and JAX timestep sweep behavior * Keep timestep checks aligned with current launcher and CLI conventions Select CPU explicitly in the bounded JAX stage and resolve optional output roots through the shared owner. Support standard single-dash help, reject ambiguous option abbreviations, and cover opaque runtime forwarding while preserving native defaults and exploratory status. Validate with 373 shared checks and four eight-step BOMEX/ATEX runs per backend at 60 and 600 seconds. * Rename the timestep sweep and run shared JAX cases at native defaults Select the 18 cases shared by the native sweep and JAX/Fortran comparison list. Remove Jenkins timestep and iteration overrides, update stage names and documentation, and preserve the sweep's stability-reporting behavior. Validate the selection and rename with 374 shared checks before live branch testing. * Reuse JAX compilations and parallelize case timestep sweeps Keep a backend-specific persistent cache outside the checkout and preserve explicit cache overrides. Run four isolated case sweeps, prioritize measured expensive cases, and print each completed report as a block while preserving timestep order and first-failure semantics. Validate cache reuse across fresh processes and cache clearing, scheduling/overlap/interruption contracts, 379 shared checks, and full 90-run cold-cache JAX/native matrices. 2026-10-07 18:51:22 -0500341cac040a0d352fd5a4da6a5938a86d7d06e332Run timestep sweep tests with JAX (#1384)addclubb_jax/pytests/auto_llm_generated_pytests/test_compilation_cache.pyeditclubb_jax/README.mdeditjenkins_tests/clubb_timestep/Jenkinsfiledeletetests/run_timestep_tests.pyaddtests/run_timestep_sweep.pyaddtests/pytests/auto_llm_generated_pytests/test_run_timestep_sweep.pyeditclubb_jax/run_jax.pyedittests/README.mdjenkins_tests/clubb_plots/JenkinsfileLLM_prompts/jenkins_workflow.mde92c2f24486a32610935ff4eede3512c67e313ea1791427833000http://carson.math.uwm.edu/jenkins/user/hueblerGunther Hueblerhuebler@uwm.eduAdd JAX comparison plots 2026-10-07 21:50:33 -0500e92c2f24486a32610935ff4eede3512c67e313eaAdd JAX comparison plotseditLLM_prompts/jenkins_workflow.mdeditjenkins_tests/clubb_plots/Jenkinsfilegithttp://carson.math.uwm.edu/jenkins/user/noreplynoreplyhttp://carson.math.uwm.edu/jenkins/user/hueblerGunther Hueblerfalse583http://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/583/581http://carson.math.uwm.edu/jenkins/job/clubb_BFB_e3sm_flags_gfortran_test/581/