Started by an SCM change Obtained jenkins_tests/cam_coarse_res_bin_diff_regression_gfortran_test/Jenkinsfile from git https://github.com/larson-group/cam.git [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /home/jenkins/workspace/cam_coarse_res_bin_diff_regression_gfortran_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git Cloning the remote Git repository Cloning repository https://github.com/larson-group/cam.git > git init /home/jenkins/workspace/cam_coarse_res_bin_diff_regression_gfortran_test # timeout=10 Fetching upstream changes from https://github.com/larson-group/cam.git > git --version # timeout=10 > git --version # 'git version 2.34.1' using GIT_ASKPASS to set credentials A token based key used by Jenkins to preform Github actions, created 6/21/2021 > git fetch --tags --force --progress -- https://github.com/larson-group/cam.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/larson-group/cam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch > git rev-parse refs/remotes/origin/clubb_silhs_devel^{commit} # timeout=10 Checking out Revision a6e8f378bc25c7cb5627d18a512fc00e8328c873 (refs/remotes/origin/clubb_silhs_devel) > git config core.sparsecheckout # timeout=10 > git checkout -f a6e8f378bc25c7cb5627d18a512fc00e8328c873 # timeout=10 Commit message: "Autoupdated CLUBB_core Commit 3f865924bdcf798fa45acaa67b57c44f40c32d89 Author: Gunther Huebler Date: Wed Jul 31 17:30:51 2024 -0500 Moving pdf_params copying to copyin. This is the data structure that contains pointers, and these pointers will be different on the host and device, so it is bad in theory to copy the structure back to the CPU as it might overwrite cpu memory pointers with gpu memory pointers. In practice though I've seen no problems caused by this, I'm just making this commit preemptively, and it has already been tested with the ECT test." > git rev-list --no-walk c0a710c144a2cb7367fd07d20be1f9ab92757a45 # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Clone SysAdmin to tmp) [Pipeline] sh + rm -rf /home/jenkins/cam_bit_diff_test_workspace/sys_admin + git clone https://github.com/larson-group/sys_admin.git /home/jenkins/cam_bit_diff_test_workspace/sys_admin Cloning into '/home/jenkins/cam_bit_diff_test_workspace/sys_admin'... [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Run Python Test) [Pipeline] sh + source /etc/profile.d/larson-group.sh ++ export GIT_EDITOR=vi ++ GIT_EDITOR=vi ++ export SVN_EDITOR=vi ++ SVN_EDITOR=vi ++ export OMP_STACKSIZE=1048579 ++ OMP_STACKSIZE=1048579 ++ export LMOD_ROOT=/usr/local/spack/opt/spack/linux-pop22-cascadelake/gcc-12.2.0/lmod-8.7.37-bi3kyxcdrfgw3y7vv2k7c5rjxg75qzju ++ LMOD_ROOT=/usr/local/spack/opt/spack/linux-pop22-cascadelake/gcc-12.2.0/lmod-8.7.37-bi3kyxcdrfgw3y7vv2k7c5rjxg75qzju ++ source /usr/local/spack/opt/spack/linux-pop22-cascadelake/gcc-12.2.0/lmod-8.7.37-bi3kyxcdrfgw3y7vv2k7c5rjxg75qzju/lmod/lmod/init/bash +++ '[' -z '' ']' +++ case "$-" in +++ __lmod_vx=x +++ '[' -n x ']' +++ set +x Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/local/spack/opt/spack/linux-pop22-cascadelake/gcc-12.2.0/lmod-8.7.37-bi3kyxcdrfgw3y7vv2k7c5rjxg75qzju/lmod/lmod/init/bash) Shell debugging restarted +++ unset __lmod_vx ++ export MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core ++ MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core + rm -rf /home/jenkins/cam_output/scratch/subcol_SILHS_UWM_debug_f10_f10_mg37 + python3 /home/jenkins/cam_bit_diff_test_workspace/sys_admin/python_nightly_test_suite/run_test.py -c /home/jenkins/cam_bit_diff_test_workspace/bit_diff_cam_config ERROR: Unknown Job Queue specified use --force to set submit_jobs case.run Submit job case.run No timing file found in run directory check for resubmit dout_s False mach larson-group resubmit_num 0 Traceback (most recent call last): File "/home/jenkins/cam_bit_diff_test_workspace/repo/run_scripts/set_up_cam_coarse_res.py", line 78, in <module> os.mkdir("ccs_config/machines/larson-group") FileExistsError: [Errno 17] File exists: 'ccs_config/machines/larson-group' ERROR: Unknown Job Queue specified use --force to set submit_jobs case.run Submit job case.run No timing file found in run directory check for resubmit dout_s False mach larson-group resubmit_num 0 PASS: All 3D output vars are zero. =============================== Configuring Test =============================== Working directory set to: /home/jenkins/cam_bit_diff_test_workspace Trying to clone repo... Cloned repo location: /home/jenkins/cam_bit_diff_test_workspace/repo Output Save Mode: none Using branch: clubb_silhs_devel Making list from 'UnresolvedCommits': Run Output Path: /home/jenkins/cam_output/scratch/subcol_SILHS_UWM_debug_f10_f10_mg37_larson-group_gnu/run/subcol_SILHS_UWM_debug_f10_f10_mg37_larson-group_gnu.cam.rh0.0001-01-02-00000.nc Run commands: ['rm -rf /home/jenkins/cam_output/scratch/subcol_SILHS_UWM_debug_f10_f10_mg37', '\nrepo/run_scripts/set_up_cam_coarse_res.py --caseroot /home/$USER/cam_output/caseroot/$CASE --reuse_build --silhs 4 repo', '\nrepo/run_scripts/run_cesm_uwm_coarse_res.sh'] Diffing command: python3 repo/run_scripts/netcdf_3D_vars_diff.py =============================== Running Test =============================== Parsing message from commit: a6e8f378bc25c7cb5627d18a512fc00e8328c873 Checking Commit: a6e8f378bc25c7cb5627d18a512fc00e8328c873 Running: final commit to check, not labelled BIT_CHANGING Checking out a6e8f378bc25c7cb5627d18a512fc00e8328c873 Running: 'rm -rf /home/jenkins/cam_output/scratch/subcol_SILHS_UWM_debug_f10_f10_mg37' Running: ' repo/run_scripts/set_up_cam_coarse_res.py --caseroot /home/$USER/cam_output/caseroot/$CASE --reuse_build --silhs 4 repo' Running: ' repo/run_scripts/run_cesm_uwm_coarse_res.sh' Checking out c0a710c144a2cb7367fd07d20be1f9ab92757a45 Running: 'rm -rf /home/jenkins/cam_output/scratch/subcol_SILHS_UWM_debug_f10_f10_mg37' Running: ' repo/run_scripts/set_up_cam_coarse_res.py --caseroot /home/$USER/cam_output/caseroot/$CASE --reuse_build --silhs 4 repo' Running: ' repo/run_scripts/run_cesm_uwm_coarse_res.sh' Diffing: 'python3 repo/run_scripts/netcdf_3D_vars_diff.py a6e8f378bc25c7cb5627d18a512fc00e8328c873 c0a710c144a2cb7367fd07d20be1f9ab92757a45' =============================== Test Results =============================== Commit Range Checked = c0a710c144a2cb7367fd07d20be1f9ab92757a45...a6e8f378bc25c7cb5627d18a512fc00e8328c873 No new Unresolved commits found. No fake BIT_CHANGING commits found. No remaining Unresolved Commits. =============================== Updating Config File =============================== Saving into UnresolvedCommits: Saving into BaselineCommit: a6e8f378bc25c7cb5627d18a512fc00e8328c873 =============================== Cleanup =============================== Removed repo. Removing all generated output REMOVING: c0a710c144a2cb7367fd07d20be1f9ab92757a45 REMOVING: a6e8f378bc25c7cb5627d18a512fc00e8328c873 ================== Unresolved Commit Status ================== No remaining Unresolved Commits. Exiting with code: 0 [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] script [Pipeline] { [Pipeline] cleanWs [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is used... [WS-CLEANUP] done [Pipeline] sh + chmod -R 755 /home/jenkins/cam_output [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS