Started by an SCM change000090056891050.997021702319707770197019599ec5e8070d3173282a6317b067f8673a657a1eb6fec5e8070d3173282a6317b067f8673a657a1eb6frefs/remotes/origin/masterec5e8070d3173282a6317b067f8673a657a1eb6fec5e8070d3173282a6317b067f8673a657a1eb6frefs/remotes/origin/masterec5e8070d3173282a6317b067f8673a657a1eb6fec5e8070d3173282a6317b067f8673a657a1eb6frefs/remotes/origin/masterhttps://github.com/larson-group/clubb.githttp://carson.math.uwm.edu/jenkins/job/clubb_BFB_r8029_flags_gfortran_test/lastStableBuild/artifacthttp://carson.math.uwm.edu/jenkins/job/clubb_BFB_r8029_flags_gfortran_test/changeshttp://carson.math.uwm.edu/jenkins/job/clubb_BFB_r8029_flags_gfortran_test/lastStableBuild/pipeline-overviewhttp://carson.math.uwm.edu/jenkins/job/clubb_BFB_r8029_flags_gfortran_test/lastStableBuild/testReportfalseUpdate sys_admin RepoRun Python Testfalse#59990056114814clubb_BFB_r8029_flags_gfortran_test #599599false59911028SUCCESS1792398428292http://carson.math.uwm.edu/jenkins/job/clubb_BFB_r8029_flags_gfortran_test/599/.agents/skills/clubb-bit-changing/SKILL.md.agents/skills/clubb-jax-port/SKILL.md.agents/skills/clubb-finalization/SKILL.mdAGENTS.md.agents/skills/clubb-documentation/change-messages/SKILL.md.agents/skills/clubb-testing-and-reporting/SKILL.mdccafe4f166f956eed26b91f7fdacd5600f60f5b21792344464000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comDefine when to declare bit-changing commits (#1403) * Define conservative bit-change declarations Centralize the policy around intentional Fortran numerical answer changes, with named affected BFB tests or precise untested conditions. Preserve standalone coverage and the current JAX parity boundary, keep declarations sparse, and separate answer-changing edits from preserving cleanup. Route message writing, validation and branch finalization to one owner. Skill schemas, local links, heading anchors, parser examples and manual decision scenarios were checked; no model or CI code changes. * Define the optimization level for bitwise preservation Use -O0 as the normal preservation standard. Differences confined to optimized builds do not require an answer-change declaration when -O0 is verified BFB; report known optimized-only differences and their configuration in the PR. The changed skill schema and whitespace checks pass. Inspected current debug compile flags and manually reviewed the optimized-only decision scenario; documentation only, no numerical rerun. 2026-10-18 12:27:44 -0500ccafe4f166f956eed26b91f7fdacd5600f60f5b2Define when to declare bit-changing commits (#1403)edit.agents/skills/clubb-finalization/SKILL.mdedit.agents/skills/clubb-jax-port/SKILL.mdedit.agents/skills/clubb-documentation/change-messages/SKILL.mdadd.agents/skills/clubb-bit-changing/SKILL.mdeditAGENTS.mdedit.agents/skills/clubb-testing-and-reporting/SKILL.mdclubb_jax/src/clubb_standalone.pyclubb_jax/pytests/auto_llm_generated_pytests/test_jax_cli_options.pyrun_scripts/README.mdutilities/clubb_case_runner.pyutilities/pytests/test_time_clubb.pytests/run_timestep_sweep.pyclubb_jax/pytests/auto_llm_generated_pytests/test_case_cache_reuse.pyclubb_python_driver/clubb_standalone.pytests/README.mdsrc/clubb_standalone.F90run_scripts/run_scm_loss.pyutilities/time_clubb.py.agents/skills/clubb-testing-and-reporting/SKILL.mdutilities/create_case_namelist.pyutilities/pytests/test_create_case_namelist.pytests/run_stats_output_consistency.pytests/run_jax_vs_fortran_cases.py.agents/skills/clubb-fortran-port/SKILL.mdclubb_jax/src/clubb_case_initalization.pyutilities/README.mdutilities/pytests/auto_llm_generated_pytests/test_timing_workers.pytests/case_runner_lifecycle_test.pyclubb_jax/pytests/auto_llm_generated_pytests/test_device_selection.py.agents/skills/clubb-numerical-diagnosis/SKILL.mdutilities/pytests/auto_llm_generated_pytests/test_loss_cli.pytests/pytests/auto_llm_generated_pytests/test_scm_jax_driver_test.pyrun_scripts/run_scm.py.agents/skills/clubb-fortran-format/SKILL.md.agents/skills/clubb-finalization/SKILL.mdclubb_jax/src/Microphys/microphys_init_cleanup.pyAGENTS.mdjenkins_tests/clubb_driver/Jenkinsfileutilities/timing_profiles.pyclubb_jax/README.mdrun_scripts/run_tuner_job.pyutilities/pytests/auto_llm_generated_pytests/test_clubb_case_runner.pytests/run_restart_test.pyutilities/pytests/timing_test_inputs.pyclubb_jax/src/CLUBB_core/calc_pressure.pytests/run_loss_output_consistency.py4f8368a377989b649fc0ed53c26957e193d210671792350419000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comAdd reusable case runners for timing (#1400) We have multiple entry points now for clubb standalone (fortran,python,jax), and there's additional logic that chooses which fortran installation to use (gfortran,ifort,nvhpc,etc). All of this has historically (~1 year) been done by run_scm.py, but we have a number of other scripts that need this same kind of logic, so instead of making run_scm.py more complicated and letting other scripts import it's pieces, we've made a `clubb_case_runner`. `clubb_case_runner` is a python object that abstracts the backend decisions. The "flow" is roughly: - high level script wants to run a clubb case some how - generates one or more case namelists (arm.in, bomex.in, etc) - creates a `clubb_case_runner` by passing it whatever runtime options (-jax/-python/-exe) the user requested - runs a case by passing this `clubb_case_runner` whatever namelists it wants to run. In addition to deduplicating runtime setting logic, this allows python scripts to create these high level `clubb_case_runner` objects and use those to do things that usually requires duplicate multiprocessesing and error code parsing logic. 2026-10-18 14:06:59 -05004f8368a377989b649fc0ed53c26957e193d21067Add reusable case runners for timing (#1400)edittests/pytests/auto_llm_generated_pytests/test_scm_jax_driver_test.pyedittests/run_jax_vs_fortran_cases.pyedittests/README.mdaddclubb_jax/pytests/auto_llm_generated_pytests/test_case_cache_reuse.pyaddutilities/pytests/timing_test_inputs.pyeditutilities/pytests/test_time_clubb.pyeditutilities/timing_profiles.pyedit.agents/skills/clubb-finalization/SKILL.mdeditclubb_jax/pytests/auto_llm_generated_pytests/test_jax_cli_options.pyeditrun_scripts/run_tuner_job.pyeditutilities/README.mdeditclubb_jax/src/CLUBB_core/calc_pressure.pyeditclubb_jax/src/clubb_case_initalization.pyedittests/run_restart_test.pyeditutilities/pytests/test_create_case_namelist.pyeditclubb_jax/README.mdeditclubb_jax/pytests/auto_llm_generated_pytests/test_device_selection.pyeditclubb_jax/src/Microphys/microphys_init_cleanup.pyedit.agents/skills/clubb-numerical-diagnosis/SKILL.mdedittests/run_stats_output_consistency.pyeditutilities/pytests/auto_llm_generated_pytests/test_loss_cli.pyaddutilities/pytests/auto_llm_generated_pytests/test_timing_workers.pyaddtests/case_runner_lifecycle_test.pyeditclubb_jax/src/clubb_standalone.pyedittests/run_timestep_sweep.pyeditsrc/clubb_standalone.F90editAGENTS.mdeditclubb_python_driver/clubb_standalone.pyeditrun_scripts/run_scm_loss.pyedit.agents/skills/clubb-testing-and-reporting/SKILL.mdedittests/run_loss_output_consistency.pyeditjenkins_tests/clubb_driver/Jenkinsfileeditrun_scripts/README.mdeditrun_scripts/run_scm.pyeditutilities/time_clubb.pyedit.agents/skills/clubb-fortran-format/SKILL.mdaddutilities/clubb_case_runner.pyaddutilities/pytests/auto_llm_generated_pytests/test_clubb_case_runner.pyedit.agents/skills/clubb-fortran-port/SKILL.mdeditutilities/create_case_namelist.pyclubb_jax/src/clubb_case_initalization.pyclubb_jax/src/clubb_standalone.pyutilities/README.mdclubb_jax/pytests/auto_llm_generated_pytests/test_jax_cli_options.pyclubb_jax/pytests/auto_llm_generated_pytests/test_device_selection.pyclubb_jax/src/clubb_loss_driver.pyclubb_jax/pytests/auto_llm_generated_pytests/test_code_timer_module.pyrun_scripts/README.mdclubb_jax/view_profile.pyclubb_jax/run_jax.pyclubb_jax/pytests/auto_llm_generated_pytests/test_profile_cli.pyclubb_jax/src/CLUBB_core/code_timer_module.pyutilities/time_clubb.pyclubb_jax/src/advance_clubb_to_end.pyclubb_jax/README.mdclubb_jax/src/clubb_driver_test.pyec5e8070d3173282a6317b067f8673a657a1eb6f1792362852000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comAdd JAX timing and profiling (#1393) * Add JAX timing and persistent benchmark workers Write Fortran-format CPU timing summaries and optional XProf/Perfetto traces from JAX driver phases. Use synchronized persistent workers for full-case warmups and repetitions across Fortran, Python and JAX. Retain compiled JAX programs when initialization settings are unchanged. Validation: 243 focused checks and CPU case workflows passed. Repeated five-step BOMEX/RICO_SILHS numeric outputs matched fresh controls exactly. The two-column BOMEX JAX/Fortran comparison passed 1e-7 absolute and percent thresholds. BFB against the parent is not established by these comparisons. GPU and GPTL execution remain untested. * Making jax timer use wall clock time instead of cpu/process time * Keep JAX timing separate from reusable runners Move process persistence, cache reuse and Python timing adapters to the case-runner branch. Keep JAX wall-clock summaries, optional deep profiles, driver instrumentation and the parser change that reads their clock label. The timing-only shared suite and a three-step CPU BOMEX run pass. BFB against master is not established; physics expressions are unchanged. * Add reusable case runners for timing and tuning Share runtime selection and persistent standalone sessions between SCM, timing and tuning. Support namelist replacement and empty model reruns, plus parameter matrices for loss sessions. Synchronize timing cohorts and record retained-model initialization separately. Use executable loss workers for native tuning instead of requiring F2PY. Restore microphysics defaults between cases (BIT_CHANGING for sessions that previously inherited settings). Short native/CPU JAX sessions match fresh same-backend stats values exactly; broader BFB is not established. Native build, focused suites and repeated loss/tuner/timing workflows pass. * Use F2PY for reusable Fortran loss runs Run persistent Fortran loss sessions through the existing Python API. Restore the native loss standalone and remove its parameter namelist reader and result writers. Python workers pass arrays directly into F2PY; ordinary standalone persistence and JAX execution remain available. Fifteen native loss metric arrays match saved results exactly. Native and CPU JAX repeated loss/tuning workflows, tuner shutdown, the native build, 454 shared checks and four API checks pass. Broader BFB is not established. * Keep case runners persistent without retaining model state Run each ordinary request through full initialization, advancement and cleanup while keeping the process alive for in-memory JIT reuse. Remove retained-model commands, transport, timing/UI options and extra bindings. Restore tuning and loss frontends to their existing allocation-reusing APIs. The native build, 445 shared checks, 121 focused JAX checks and 85 Dash checks pass. Short native/JAX/Python reused-process stats match fresh controls exactly; tuning and synchronized timing sweeps pass. Two full 16-column JAX repeats log no new compilations. These comparisons do not establish blanket BFB. * Separate microphysics initialization fixes from case runners Move native default resets to their own branch/PR and keep the runner focused on shared runtime selection and full-case process reuse. Record the mixed-case native test's dependency on that fix. Refresh the timing base and master, and clarify runner, driver and test documentation under the current skills. The shared checks pass, including output drainage, cancellation, synchronized warmup/repetition barriers and CLI forwarding. Python AST checks confirm the remaining audit edits change only documentation. Jenkins validation follows this published checkpoint. * Complete case-runner procedure descriptions Describe the remaining changed CLI, test and profile routines under the current documentation standards. Executable Python ASTs are unchanged from f66ccf2ff, the revision running in Jenkins; existing validation still applies. * Separate JAX timing from case runner changes Keep runtime selection and process reuse independent so the case runner can merge into master before JAX timing. Shared runner/CLI/timing checks pass (445 tests, including provisional coverage). * Add JAX timing after the case runner Restore the audited wall-clock summaries and optional execution tracing on top of the independent case runner. Keep the existing implementation and review history while reversing the PR merge order. * Add a JAX profiling option and viewer commands Enable captures with -jax=profile and print commands after cleanup. Keep capture export in the timer and viewing in view_profile.py. 181 focused checks pass. CPU BOMEX captures preserve numeric stats; both viewers were verified in-browser. Large full-Python Perfetto exports can truncate; XProf retained all five tested timesteps. * Clarify JAX timing usage and cleanup Document the timer lifecycle, failure handling and Dash wall-time comparison metric. Executable Python AST is unchanged. 2026-10-18 17:34:12 -0500ec5e8070d3173282a6317b067f8673a657a1eb6fAdd JAX timing and profiling (#1393)addclubb_jax/pytests/auto_llm_generated_pytests/test_profile_cli.pyeditclubb_jax/src/clubb_loss_driver.pyaddclubb_jax/src/CLUBB_core/code_timer_module.pyeditclubb_jax/pytests/auto_llm_generated_pytests/test_jax_cli_options.pyeditutilities/README.mdeditclubb_jax/README.mdeditutilities/time_clubb.pyeditclubb_jax/pytests/auto_llm_generated_pytests/test_device_selection.pyeditclubb_jax/src/advance_clubb_to_end.pyeditclubb_jax/src/clubb_standalone.pyaddclubb_jax/view_profile.pyeditclubb_jax/src/clubb_driver_test.pyaddclubb_jax/pytests/auto_llm_generated_pytests/test_code_timer_module.pyeditrun_scripts/README.mdeditclubb_jax/run_jax.pyeditclubb_jax/src/clubb_case_initalization.pygithttp://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreplyfalseStarted by an SCM change00001714591704400.991641964195986e48f97c2a96bc78fb498c82f1ed0bb6902353496e48f97c2a96bc78fb498c82f1ed0bb690235349refs/remotes/origin/master6e48f97c2a96bc78fb498c82f1ed0bb6902353496e48f97c2a96bc78fb498c82f1ed0bb690235349refs/remotes/origin/master6e48f97c2a96bc78fb498c82f1ed0bb6902353496e48f97c2a96bc78fb498c82f1ed0bb690235349refs/remotes/origin/masterhttps://github.com/larson-group/clubb.gitfalse#598171459114814clubb_BFB_r8029_flags_gfortran_test #598598false59810907SUCCESS1792312027640http://carson.math.uwm.edu/jenkins/job/clubb_BFB_r8029_flags_gfortran_test/598/clubb_jax/pytests/auto_llm_generated_pytests/test_windm_edsclrm_scalar_indices.pyclubb_jax/pytests/auto_llm_generated_pytests/test_xp2_error_diagnostics.pyclubb_jax/pytests/auto_llm_generated_pytests/test_radiation_error_diagnostics.pyclubb_jax/src/CLUBB_core/err_info.pyclubb_jax/src/CLUBB_core/advance_windm_edsclrm_module.pyclubb_jax/pytests/auto_llm_generated_pytests/test_jit_compile.pyclubb_jax/src/CLUBB_core/__init__.pyclubb_jax/src/CLUBB_core/numerical_check.pyclubb_jax/src/advance_clubb_to_end.pyclubb_jax/src/CLUBB_core/advance_xp2_xpyp_module.pyclubb_python_api/JAX_INTEROPERABILITY_PROPOSAL.mdclubb_jax/src/CLUBB_core/err_info_codes.pyclubb_jax/src/Radiation/radiation_module.pyaa8c2aa4037dda69d686ca97bcbbe1a8208401fd1792259885000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comRemove the JAX error-message code registry (#1401) * Remove the JAX error-message code registry Print failure details at the runtime checks that detect them. Remove the extra reason_code field and its setters, decoder, initialization/reset handling and exports; retain the normal per-column fatal-error status. Update the scalar-guard assertions and add cached-call diagnostic checks. Local CPU checks passed for the ErrInfo pytree, diagnostic output, driver debug gates and state reset/batching behavior (65 cases across five modules). * Correct the multiple-LHS diagnostic test's wp3 grid Allocate synthetic wp3 on thermodynamic levels, as required by calc_wp3_on_wp2, so the test reaches the multiple-LHS guard during tracing. The 21 selected CPU diagnostic checks pass, covering cached-call output, per-column fatal status and preservation of fields on the unsupported path. * Restore the C2 explanation in the multiple-LHS diagnostic Name C2rt, C2thl and C2rtthl when their unequal values prevent a single-LHS solve. Keep the direct runtime print and existing fatal-status handling. The focused CPU cached-call test passes, including the restored explanation, quiet valid calls and preservation of incoming fields on failure. * Revert "Restore the C2 explanation in the multiple-LHS diagnostic" This reverts commit 170a13e2bc7af24e92d8d8bc9de9d9db601921ed. * List single-LHS requirements in the JAX diagnostic Print the approved static requirements after the unsupported multiple-LHS message: explicit turbulent advection or ADG1, plus per-column equivalence of C2thl and C2rtthl to C2rt within the relative epsilon tolerance. The focused CPU cached-call test passes, checking the listed requirements, quiet valid calls and preservation of incoming fields and fatal status. 2026-10-17 12:58:05 -0500aa8c2aa4037dda69d686ca97bcbbe1a8208401fdRemove the JAX error-message code registry (#1401)editclubb_jax/pytests/auto_llm_generated_pytests/test_jit_compile.pyaddclubb_jax/pytests/auto_llm_generated_pytests/test_xp2_error_diagnostics.pyeditclubb_jax/src/CLUBB_core/numerical_check.pyeditclubb_jax/pytests/auto_llm_generated_pytests/test_windm_edsclrm_scalar_indices.pyeditclubb_jax/src/CLUBB_core/advance_xp2_xpyp_module.pyeditclubb_jax/src/advance_clubb_to_end.pyaddclubb_jax/pytests/auto_llm_generated_pytests/test_radiation_error_diagnostics.pyeditclubb_jax/src/CLUBB_core/err_info.pyeditclubb_jax/src/Radiation/radiation_module.pyeditclubb_python_api/JAX_INTEROPERABILITY_PROPOSAL.mdeditclubb_jax/src/CLUBB_core/__init__.pydeleteclubb_jax/src/CLUBB_core/err_info_codes.pyeditclubb_jax/src/CLUBB_core/advance_windm_edsclrm_module.py.agents/skills/clubb-jax-port/SKILL.md.agents/skills/clubb-documentation/change-messages/SKILL.md.agents/skills/clubb-testing-and-reporting/SKILL.md6e48f97c2a96bc78fb498c82f1ed0bb6902353491792260807000http://carson.math.uwm.edu/jenkins/user/noreplynoreplynoreply@github.comRestrict BIT_CHANGING guidance to Fortran result changes (#1399) Keep JAX numerical acceptance focused on agreement with the corresponding Fortran implementation rather than preserving earlier JAX output. Clarify the reporting rule and link it from the JAX porting workflow. All 21 skill schemas, 112 local links and 7 heading anchors pass, along with diff formatting. Only three skill Markdown files change. 2026-10-17 13:13:27 -05006e48f97c2a96bc78fb498c82f1ed0bb690235349Restrict BIT_CHANGING guidance to Fortran result changes (#1399)edit.agents/skills/clubb-documentation/change-messages/SKILL.mdedit.agents/skills/clubb-testing-and-reporting/SKILL.mdedit.agents/skills/clubb-jax-port/SKILL.mdgithttp://carson.math.uwm.edu/jenkins/user/noreplynoreplyfalse599http://carson.math.uwm.edu/jenkins/job/clubb_BFB_r8029_flags_gfortran_test/599/597http://carson.math.uwm.edu/jenkins/job/clubb_BFB_r8029_flags_gfortran_test/597/