Started by an SCM change Obtained jenkins_tests/clubb_BFB_multi_col_gpu_nvhpc_test/Jenkinsfile from git https://github.com/larson-group/clubb.git [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/larson-group/clubb.git > git init /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test # timeout=10 Fetching upstream changes from https://github.com/larson-group/clubb.git > git --version # timeout=10 > git --version # 'git version 2.34.1' > git fetch --tags --force --progress -- https://github.com/larson-group/clubb.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/larson-group/clubb.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 Checking out Revision fb12cc1c0be895d2db030ceed981bc5121277b0e (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f fb12cc1c0be895d2db030ceed981bc5121277b0e # timeout=10 Commit message: "The last commit should've been marked as non-BFB since it corrected a crashing bug - so comparing output to no output caused the output to be considered different. This was missed because the BFB checking tests didn't fail, and that was because the commit that caused the crashing was marked non-BFB, so an unusual combo. In general though any commit that fixes crashing behavior should be labelled as bit changing. RESOLVED:73dc365c77ab4bf2405d753f167307ae9be330a0" > git rev-list --no-walk 73dc365c77ab4bf2405d753f167307ae9be330a0 # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Update sys_admin Repo) [Pipeline] sh + git -C /home/pub/jenkins_BFB_test_configs/sys_admin pull Already up to date. [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Run Python Test) [Pipeline] lock Trying to acquire lock on [Resource: GPU_built-in] The resource [GPU_built-in] is locked by build clubb_plot_nvhpc_gpu_vs_cpu #180 #180 since May 29, 2026, 3:14 AM. [Resource: GPU_built-in] is not free, waiting for execution ... [Required resources: [GPU_built-in]] added into queue at position 0 Lock acquired on [Resource: GPU_built-in] [Pipeline] { [Pipeline] sh + ulimit -s 8388608 + source /etc/profile.d/larson-group.sh ++ export GIT_EDITOR=vi ++ GIT_EDITOR=vi ++ export SVN_EDITOR=vi ++ SVN_EDITOR=vi ++ export OMP_STACKSIZE=1048579 ++ OMP_STACKSIZE=1048579 ++ export LMOD_ROOT=/opt/lmod/ ++ LMOD_ROOT=/opt/lmod/ ++ source /opt/lmod//lmod/lmod/init/bash +++ '[' -z '' ']' +++ case "$-" in +++ __lmod_vx=x +++ '[' -n x ']' +++ set +x Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/local/spack/opt/spack/linux-pop22-cascadelake/gcc-12.2.0/lmod-8.7.37-bi3kyxcdrfgw3y7vv2k7c5rjxg75qzju/lmod/lmod/init/bash) Shell debugging restarted +++ unset __lmod_vx +++ find /usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core -print -quit ++ export MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core ++ MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core + module load nvhpc netcdf-fortran + '[' -z '' ']' + case "$-" in + __lmod_sh_dbg=x + '[' -n x ']' + set +x Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output Shell debugging restarted + unset __lmod_sh_dbg + return 0 + python3 /home/pub/jenkins_BFB_test_configs/sys_admin/python_nightly_test_suite/run_test.py -c /home/pub/jenkins_BFB_test_configs/clubb_BFB_multi_col_gpu_nvhpc_test_config CMake Warning: Manually-specified variables were not used by the project: NetCDF_C_INCLUDE_DIR NetCDF_C_LIBRARY NetCDF_Fortran_INCLUDE_DIR NetCDF_Fortran_LIBRARY Directory 1 is fb12cc1c0be895d2db030ceed981bc5121277b0e Directory 2 is 73dc365c77ab4bf2405d753f167307ae9be330a0 Using reporting threshold: 0.0 The following cases will be compared: ['arm', 'cobra', 'gabls3', 'mpace_b', 'rico_silhs'] ================================================ arm ================================================ >No differences detected by the linux diff in arm_stats.nc< >>>The linux diff could not detect any differences in the file pairs for case arm.<<< ===================================================================================================== ================================================ cobra ================================================ >No differences detected by the linux diff in cobra_stats.nc< >>>The linux diff could not detect any differences in the file pairs for case cobra.<<< ===================================================================================================== ================================================ gabls3 ================================================ >No differences detected by the linux diff in gabls3_stats.nc< >>>The linux diff could not detect any differences in the file pairs for case gabls3.<<< ===================================================================================================== ================================================ mpace_b ================================================ >No differences detected by the linux diff in mpace_b_stats.nc< >>>The linux diff could not detect any differences in the file pairs for case mpace_b.<<< ===================================================================================================== ================================================ rico_silhs ================================================ >No differences detected by the linux diff in rico_silhs_stats.nc< >>>The linux diff could not detect any differences in the file pairs for case rico_silhs.<<< ===================================================================================================== SUMMARY: Linux diff did not detect any differences in the compared files. Pass: 5 cases ['arm', 'cobra', 'gabls3', 'mpace_b', 'rico_silhs'] Fail: 0 cases [] =============================== Configuring Test =============================== Working directory set to: /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test Checking specified repo: /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test Using /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test with origin https://github.com/larson-group/clubb.git Output Save Mode: all Using branch: master Making list from 'UnresolvedCommits': CheckRange = True: Getting commit range Run Output Path: output Run commands: ['python compile.py -gpu openacc -debug', '\nrm -rf output', '\nmkdir output', "\nsed -i 's:lh_straight_mc.*:lh_straight_mc = true:g' input/tunable_parameters/configurable_model_flags.in", "\nsed -i 's:penta_solve_method.*:penta_solve_method = 2:g' input/tunable_parameters/configurable_model_flags.in", "\nsed -i 's:tridiag_solve_method.*:tridiag_solve_method = 2:g' input/tunable_parameters/configurable_model_flags.in", '\npython run_scripts/create_multi_col_params.py -n 4 -out_file run_scripts/clubb_params_multi_col.in', '\npython run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in arm', '\npython run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in rico_silhs', '\npython run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in cobra', '\npython run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in mpace_b', '\npython run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in gabls3'] Diffing command: run_scripts/run_bindiff_all.py =============================== Running Test =============================== Parsing message from commit: fb12cc1c0be895d2db030ceed981bc5121277b0e Found: RESOLVED:73dc365c77ab4bf2405d753f167307ae9be330a0 Checking Commit: fb12cc1c0be895d2db030ceed981bc5121277b0e Running: final commit to check, not labelled BIT_CHANGING Checking out fb12cc1c0be895d2db030ceed981bc5121277b0e Running: 'python compile.py -gpu openacc -debug' Running: 'rm -rf output' Running: 'mkdir output' Running: 'sed -i 's:lh_straight_mc.*:lh_straight_mc = true:g' input/tunable_parameters/configurable_model_flags.in' Running: 'sed -i 's:penta_solve_method.*:penta_solve_method = 2:g' input/tunable_parameters/configurable_model_flags.in' Running: 'sed -i 's:tridiag_solve_method.*:tridiag_solve_method = 2:g' input/tunable_parameters/configurable_model_flags.in' Running: 'python run_scripts/create_multi_col_params.py -n 4 -out_file run_scripts/clubb_params_multi_col.in' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in arm' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in rico_silhs' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in cobra' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in mpace_b' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in gabls3' Checking out 73dc365c77ab4bf2405d753f167307ae9be330a0 Running: 'python compile.py -gpu openacc -debug' Running: 'rm -rf output' Running: 'mkdir output' Running: 'sed -i 's:lh_straight_mc.*:lh_straight_mc = true:g' input/tunable_parameters/configurable_model_flags.in' Running: 'sed -i 's:penta_solve_method.*:penta_solve_method = 2:g' input/tunable_parameters/configurable_model_flags.in' Running: 'sed -i 's:tridiag_solve_method.*:tridiag_solve_method = 2:g' input/tunable_parameters/configurable_model_flags.in' Running: 'python run_scripts/create_multi_col_params.py -n 4 -out_file run_scripts/clubb_params_multi_col.in' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in arm' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in rico_silhs' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in cobra' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in mpace_b' Running: 'python run_scripts/run_scm.py -params run_scripts/clubb_params_multi_col.in gabls3' Diffing: 'run_scripts/run_bindiff_all.py fb12cc1c0be895d2db030ceed981bc5121277b0e 73dc365c77ab4bf2405d753f167307ae9be330a0' =============================== Test Results =============================== Commit Range Checked = 73dc365c77ab4bf2405d753f167307ae9be330a0...fb12cc1c0be895d2db030ceed981bc5121277b0e No new Unresolved commits found. No fake BIT_CHANGING commits found. WARNING: 73dc365c77ab4bf2405d753f167307ae9be330a0 labelled as RESOLVED, but was not listed as unresolved. No remaining Unresolved Commits. =============================== Updating Config File =============================== Saving into UnresolvedCommits: Saving into BaselineCommit: fb12cc1c0be895d2db030ceed981bc5121277b0e =============================== Cleanup =============================== Not removing any generated output. RENAMING fb12cc1c0be895d2db030ceed981bc5121277b0e to latest_revision_validated_fb12cc1c0be895d2db030ceed981bc5121277b0e ================== Unresolved Commit Status ================== No remaining Unresolved Commits. Exiting with code: 0 [Pipeline] } Lock released on resource [Resource: GPU_built-in] [Pipeline] // lock [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS