Started by an SCM change Obtained jenkins_tests/clubb_column_mirror/Jenkinsfile from git https://github.com/larson-group/clubb.git [Pipeline] Start of Pipeline [Pipeline] node Running on nelson in /home/jenkins/workspace/clubb_column_mirror [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/larson-group/clubb.git > git init /home/jenkins/workspace/clubb_column_mirror # timeout=10 Fetching upstream changes from https://github.com/larson-group/clubb.git > git --version # timeout=10 > git --version # 'git version 2.43.0' using GIT_ASKPASS to set credentials larsongroupsysadmin github token > git fetch --tags --force --progress -- https://github.com/larson-group/clubb.git +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision c47faf30cea562966e64eedc259a5074c6cf89b1 (refs/remotes/origin/master) > git config remote.origin.url https://github.com/larson-group/clubb.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f c47faf30cea562966e64eedc259a5074c6cf89b1 # timeout=10 Commit message: "Restore missing debug printouts in JAX core (#1390)" > git rev-list --no-walk 09499c45d0ef25810f48bbd43c8f92afa1497caa # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Run Column Mirror Test) [Pipeline] sh + source /etc/profile.d/larson-group.sh ++ export GIT_EDITOR=vi ++ GIT_EDITOR=vi ++ export SVN_EDITOR=vi ++ SVN_EDITOR=vi ++ export OMP_STACKSIZE=1048579 ++ OMP_STACKSIZE=1048579 ++ export LMOD_ROOT=/opt/lmod/ ++ LMOD_ROOT=/opt/lmod/ ++ . /opt/lmod//lmod/lmod/init/profile +++ '[' -z '' ']' +++ LMOD_ALLOW_ROOT_USE=yes +++ '[' -n '' ']' +++ '[' yes '!=' yes ']' +++ '[' -z '' ']' +++ export MODULEPATH_ROOT=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles +++ MODULEPATH_ROOT=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles +++ export USER=jenkins +++ USER=jenkins ++++ uname +++ export LMOD_sys=Linux +++ LMOD_sys=Linux +++ '[' -z '' ']' +++ '[' -f /etc/lmod/.modulespath ']' +++ LMOD_MODULEPATH_INIT=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/.modulespath +++ '[' -f /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/.modulespath ']' ++++ /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/libexec/addto --append MODULEPATH /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Linux /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Core +++ export MODULEPATH=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Linux:/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Core +++ MODULEPATH=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Linux:/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Core ++++ /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/libexec/addto --append MODULEPATH /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/modulefiles/Core +++ export MODULEPATH=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Linux:/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Core:/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/modulefiles/Core +++ MODULEPATH=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Linux:/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/modulefiles/Core:/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/modulefiles/Core +++ '[' -n '' ']' +++ export BASH_ENV=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/bash +++ BASH_ENV=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/bash +++ '[' -z '' ']' +++ export MANPATH=: +++ MANPATH=: ++++ /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/libexec/addto MANPATH /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/share/man +++ export MANPATH=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/share/man: +++ MANPATH=/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/share/man: +++ findExec READLINK_CMD /usr/bin/readlink readlink +++ Nm=READLINK_CMD +++ confPath=/usr/bin/readlink +++ execNm=readlink +++ eval READLINK_CMD=/usr/bin/readlink ++++ READLINK_CMD=/usr/bin/readlink +++ '[' '!' -x /usr/bin/readlink ']' +++ unset Nm confPath execNm +++ findExec PS_CMD /usr/bin/ps ps +++ Nm=PS_CMD +++ confPath=/usr/bin/ps +++ execNm=ps +++ eval PS_CMD=/usr/bin/ps ++++ PS_CMD=/usr/bin/ps +++ '[' '!' -x /usr/bin/ps ']' +++ unset Nm confPath execNm +++ findExec EXPR_CMD /usr/bin/expr expr +++ Nm=EXPR_CMD +++ confPath=/usr/bin/expr +++ execNm=expr +++ eval EXPR_CMD=/usr/bin/expr ++++ EXPR_CMD=/usr/bin/expr +++ '[' '!' -x /usr/bin/expr ']' +++ unset Nm confPath execNm +++ findExec BASENAME_CMD /usr/bin/basename basename +++ Nm=BASENAME_CMD +++ confPath=/usr/bin/basename +++ execNm=basename +++ eval BASENAME_CMD=/usr/bin/basename ++++ BASENAME_CMD=/usr/bin/basename +++ '[' '!' -x /usr/bin/basename ']' +++ unset Nm confPath execNm +++ unset -f findExec +++ '[' -f /proc/4192949/exe ']' ++++ /usr/bin/readlink /proc/4192949/exe +++ my_shell=/usr/bin/bash ++++ /usr/bin/expr /usr/bin/bash : '-*\(.*\)' +++ my_shell=/usr/bin/bash ++++ /usr/bin/basename /usr/bin/bash +++ my_shell=bash +++ case ${my_shell} in +++ '[' -f /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/bash ']' +++ . /usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/bash ++++ '[' -z '' ']' ++++ case "$-" in ++++ __lmod_vx=x ++++ '[' -n x ']' ++++ set +x Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/bash) Shell debugging restarted ++++ unset __lmod_vx +++ unset my_shell PS_CMD EXPR_CMD BASENAME_CMD MODULEPATH_INIT LMOD_ALLOW_ROOT_USE READLINK_CMD +++ find /usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core -print -quit ++ export MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core ++ MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core + module load gcc netcdf-fortran + '[' -z '' ']' + case "$-" in + __lmod_sh_dbg=x + '[' -n x ']' + set +x Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output Lmod Warning: ------------------------------------------------------------------------------- The following dependent module(s) are not currently loaded: gcc-runtime/13.1.0-wiuv7wc (required by: mpich/4.2.1, hdf5/1.14.3-cb5lzz7, netcdf-c/4.9.2, netcdf-fortran/4.5.3), glibc/2.35-tdr5czg (required by: mpich/4.2.1, hdf5/1.14.3-cb5lzz7, netcdf-c/4.9.2, netcdf-fortran/4.5.3) ------------------------------------------------------------------------------- Shell debugging restarted + unset __lmod_sh_dbg + return 0 + ulimit -s 8388608 + python3 tests/check_mirrored_multi_col_output.py -multicol 3 -workers 2 Installing uv 0.11.32 in /home/jenkins/workspace/clubb_column_mirror/.clubb-python-tools/bin downloading uv 0.11.32 x86_64-unknown-linux-gnu installing to /home/jenkins/workspace/clubb_column_mirror/.clubb-python-tools/bin uv uvx everything's installed! Creating CLUBB virtualenv: /home/jenkins/workspace/clubb_column_mirror/.venv-python Using CPython 3.12.3 interpreter at: /usr/bin/python3.12 Creating virtual environment at: .venv-python Activate with: source .venv-python/bin/activate Installing python requirements into /home/jenkins/workspace/clubb_column_mirror/.venv-python Using Python 3.12.3 environment at: .venv-python Resolved 12 packages in 317ms Downloading pygments (1.2MiB) Downloading cftime (1.6MiB) Downloading meson (1.1MiB) Downloading numpy (16.0MiB) Downloading netcdf4 (10.1MiB) Downloaded cftime Downloaded netcdf4 Downloaded meson Downloaded pygments Downloaded numpy Prepared 12 packages in 571ms Installed 12 packages in 17ms + certifi==2026.7.22 + cftime==1.6.6.1 + iniconfig==2.3.1 + meson==1.12.1 + netcdf4==1.7.5 + ninja==1.13.2 + numpy==2.5.4 + packaging==26.3 + pluggy==1.7.0 + pygments==2.21.0 + pytest==9.1.1 + tabulate==0.10.0 Ready: /home/jenkins/workspace/clubb_column_mirror/.venv-python/bin/python Logs and output: /home/jenkins/workspace/clubb_column_mirror/output/column_mirror_test/run_03yr4n29 [1/4] Configuring and compiling (gfortran, CPU) [2/4] ABC run Cases: arm, arm_97, astex_a209, atex, bomex, cgils_s6, cgils_s11, cgils_s12, clex9_nov02, clex9_oct14, dycoms2_rf01, dycoms2_rf01_fixed_sst, dycoms2_rf02_do, dycoms2_rf02_ds, dycoms2_rf02_nd, dycoms2_rf02_so, fire, gabls2, gabls3, gabls3_night, jun25_altocu, lba, mc3e, mpace_a, mpace_b, mpace_b_silhs, nov11_altocu, rico, rico_silhs, twp_ice, wangara Grid: -multicol C8/0.2:0.8/3 C8 = [0.2, 0.5, 0.8] [3/4] CBA run Cases: arm, arm_97, astex_a209, atex, bomex, cgils_s6, cgils_s11, cgils_s12, clex9_nov02, clex9_oct14, dycoms2_rf01, dycoms2_rf01_fixed_sst, dycoms2_rf02_do, dycoms2_rf02_ds, dycoms2_rf02_nd, dycoms2_rf02_so, fire, gabls2, gabls3, gabls3_night, jun25_altocu, lba, mc3e, mpace_a, mpace_b, mpace_b_silhs, nov11_altocu, rico, rico_silhs, twp_ice, wangara Grid: -multicol C8/0.2:0.8/3 (columns reversed) C8 = [0.8, 0.5, 0.2] [4/4] Comparison PASS arm_97 PASS arm PASS astex_a209 PASS atex PASS bomex PASS cgils_s11 PASS cgils_s12 PASS cgils_s6 PASS clex9_nov02 PASS clex9_oct14 PASS dycoms2_rf01_fixed_sst PASS dycoms2_rf01 PASS dycoms2_rf02_do PASS dycoms2_rf02_ds PASS dycoms2_rf02_nd PASS dycoms2_rf02_so PASS fire PASS gabls2 PASS gabls3_night PASS gabls3 PASS jun25_altocu PASS lba PASS mc3e PASS mpace_a PASS mpace_b_silhs PASS mpace_b PASS nov11_altocu PASS rico_silhs PASS rico PASS twp_ice PASS wangara [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS