Started by an SCM change
Obtained jenkins_tests/cam_clubb_error_handling_test/Jenkinsfile from git <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a>
<span class="pipeline-new-node" nodeId="2" startId="2">[Pipeline] Start of Pipeline
</span><span class="pipeline-new-node" nodeId="3" startId="3" enclosingId="2">[Pipeline] node
</span><span class="pipeline-node-3">Still waiting to schedule task
Waiting for next available executor
Running on <a href='/jenkins/computer/mitch/' class='jenkins-table__link model-link model-link--float'>mitch</a> in /home/jenkins/workspace/cam_clubb_error_handling_test
</span><span class="pipeline-new-node" nodeId="4" startId="4" enclosingId="3">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="5" startId="5" enclosingId="4">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="6" startId="6" enclosingId="5" label="Declarative: Checkout SCM">[Pipeline] { (Declarative: Checkout SCM)
</span><span class="pipeline-new-node" nodeId="7" enclosingId="6">[Pipeline] checkout
</span><span class="pipeline-node-7">The recommended git tool is: git
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a>
 &gt; git init /home/jenkins/workspace/cam_clubb_error_handling_test # timeout=10
Fetching upstream changes from <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a>
 &gt; git --version # timeout=10
 &gt; git --version # 'git version 2.43.0'
using GIT_ASKPASS to set credentials larsongroupsysadmin github token
 &gt; git fetch --tags --force --progress -- <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a> +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 1eca30798f2886be7d8594c8826320737b7a76cd (refs/remotes/origin/clubb_silhs_devel)
 &gt; git config remote.origin.url <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a> # timeout=10
 &gt; git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 &gt; git rev-parse refs/remotes/origin/clubb_silhs_devel^{commit} # timeout=10
 &gt; git config core.sparsecheckout # timeout=10
 &gt; git checkout -f 1eca30798f2886be7d8594c8826320737b7a76cd # timeout=10
Commit message: "Autoupdated CLUBB_core Commit a8c952996409f8c94d6a28714171145ed7d2c2ae Author: Daniel Ebel Date: Tue Oct 13 12:10:00 2026 -0500 Fixed bug in cubic equation solve function. Only relevant if pdf is L… (#1387) * Fixed bug in cubic equation solve function. Only relevant if pdf is Luhar or iiPDF_type=4"
 &gt; git rev-list --no-walk 6fda8e14438857fdf16e8afb5b32a52302cfd2e9 # timeout=10
</span><span class="pipeline-new-node" nodeId="8" startId="6">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="9" startId="5">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="10" startId="10" enclosingId="4">[Pipeline] withEnv
</span><span class="pipeline-new-node" nodeId="11" startId="11" enclosingId="10">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="12" startId="12" enclosingId="11">[Pipeline] withEnv
</span><span class="pipeline-new-node" nodeId="13" startId="13" enclosingId="12">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="14" startId="14" enclosingId="13">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="15" startId="15" enclosingId="14" label="Checkout Externals and Copy Custom Files">[Pipeline] { (Checkout Externals and Copy Custom Files)
</span><span class="pipeline-new-node" nodeId="16" enclosingId="15">[Pipeline] sh
</span><span class="pipeline-node-16">+ python run_scripts/set_up_repo.py .
Checking out externals...
Sparse checkout cam-sima fxsparse ../.sima_phys_utils_sparse_checkout
Successfully checked out             cam-sima at sima0_07_000
            cam-sima up to date.
Sparse checkout mpas fxsparse ../.mpas_sparse_checkout
Successfully checked out                 mpas at v8.4.0
                mpas up to date.
Sparse checkout cosp2 fxsparse ../.cosp_sparse_checkout
Successfully checked out                cosp2 at v2.2.1
               cosp2 up to date.
Sparse checkout ext_co2_cooling fxsparse ../.ext_co2_cooling_sparse_checkout
Successfully checked out      ext_co2_cooling at v1.1
     ext_co2_cooling up to date.
           chem_proc up to date.
               carma up to date.
        pumas-frozen up to date.
            ali_arms up to date.
          atmos_phys up to date.
Recursively checking out submodules of atmos_phys
                 fv3 up to date.
Recursively checking out submodules of fv3
             cloud_j up to date.
                hetp up to date.
               hemco up to date.
Recursively checking out submodules of hemco
         rrtmgp-data up to date.
                 rtm up to date.
              mosart up to date.
                 fms up to date.
Recursively checking out submodules of fms
          ccs_config up to date.
               share up to date.
               cdeps up to date.
Recursively checking out submodules of cdeps
         tools/CUPiD up to date.
Recursively checking out submodules of tools/CUPiD
               cmeps up to date.
               tuv-x up to date.
            geoschem up to date.
Recursively checking out submodules of geoschem
                cice up to date.
Recursively checking out submodules of cice
           mizuRoute up to date.
Recursively checking out submodules of mizuRoute
          parallelio up to date.
Recursively checking out submodules of HEMCO
Recursively checking out submodules of src
                 clm up to date.
Recursively checking out submodules of clm
                cime up to date.
Recursively checking out submodules of cime
Recursively checking out submodules of fates
Copying ccs_config files
Restricting NVHPC OpenACC rules to CLUBB
Applying CAM-SIMA Filepath workaround
</span><span class="pipeline-new-node" nodeId="17" startId="15">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="18" startId="14">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="19" startId="19" enclosingId="13">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="20" startId="20" enclosingId="19" label="Sed Commands">[Pipeline] { (Sed Commands)
</span><span class="pipeline-new-node" nodeId="21" enclosingId="20">[Pipeline] sh
</span><span class="pipeline-node-21">+ sed -i 's/DEBUG="TRUE"/DEBUG="FALSE"/g' run_scripts/run_scam.bash
</span><span class="pipeline-new-node" nodeId="22" startId="20">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="23" startId="19">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="24" startId="24" enclosingId="13">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="25" startId="25" enclosingId="24" label="Remove Old Output">[Pipeline] { (Remove Old Output)
</span><span class="pipeline-new-node" nodeId="26" enclosingId="25">[Pipeline] sh
</span><span class="pipeline-node-26">+ rm -rf /home/jenkins/cam_output/scratch/test_scam_ATEX_cam_clubb_error_handling_test /home/jenkins/cam_output/scratch/test_scam_ARM97_cam_clubb_error_handling_test
</span><span class="pipeline-new-node" nodeId="27" startId="27" enclosingId="25">[Pipeline] script
</span><span class="pipeline-new-node" nodeId="28" startId="28" enclosingId="27">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="29" startId="28">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="30" startId="27">[Pipeline] // script
</span><span class="pipeline-new-node" nodeId="31" startId="25">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="32" startId="24">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="33" startId="33" enclosingId="13">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="34" startId="34" enclosingId="33" label="Run test script">[Pipeline] { (Run test script)
</span><span class="pipeline-new-node" nodeId="35" startId="35" enclosingId="34">[Pipeline] dir
</span><span class="pipeline-node-35">Running in /home/jenkins/workspace/cam_clubb_error_handling_test/jenkins_tests/cam_clubb_error_handling_test
</span><span class="pipeline-new-node" nodeId="36" startId="36" enclosingId="35">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="37" enclosingId="36">[Pipeline] sh
</span><span class="pipeline-node-37">+ source /etc/profile.d/larson-group.sh
++ export GIT_EDITOR=vi
++ GIT_EDITOR=vi
++ export SVN_EDITOR=vi
++ SVN_EDITOR=vi
++ export OMP_STACKSIZE=1048579
++ OMP_STACKSIZE=1048579
++ export LMOD_ROOT=/opt/lmod/
++ LMOD_ROOT=/opt/lmod/
++ source /opt/lmod//lmod/lmod/init/bash
+++ '[' -z '' ']'
+++ case "$-" in
+++ __lmod_vx=x
+++ '[' -n x ']'
+++ set +x
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-fq24mybyn2li6got2bxzk62ejh5d3p4z/lmod/lmod/init/bash)
Shell debugging restarted
+++ unset __lmod_vx
+++ find /usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core -print -quit
++ export MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core
++ MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core
+ ./cam_test_fatal_error_handling.bash --case-suffix cam_clubb_error_handling_test
----- Case Setup -----
Did not find an alias or longname compset match for 2000_CAM60%SCAMATEX_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV 
Automatically adding SESP to compset
Compset forcing is 1972-2004
ATM component is CAM cam6 physics:
LND component is Stub land component
ICE component is Stub ice component
OCN component is DOCN   analytic aquaplanet sst - option 3
ROF component is Stub river component
GLC component is Stub glacier (land ice) component
WAV component is Stub wave component
ESP component is Stub external system processing (ESP) component
Pes     specification file is /home/jenkins/workspace/cam_clubb_error_handling_test/cime_config/config_pes.xml
Compset specific settings: name is RUN_STARTDATE and value is 1969-02-15
Compset specific settings: name is STOP_N and value is 47
Compset specific settings: name is STOP_OPTION and value is nhours
Compset specific settings: name is START_TOD and value is 0
Compset specific settings: name is PTS_MODE and value is TRUE
Compset specific settings: name is PTS_LAT and value is 15.0
Compset specific settings: name is PTS_LON and value is 345.0
Machine is larson-group
vid is comment
vid is ntasks
vid is nthrds
vid is rootpe
Pes setting: grid          is a%ne16np4_l%null_oi%ne16np4_r%null_g%null_w%null_z%null_m%gx1v7 
Pes setting: compset       is 2000_CAM60%SCAMATEX_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV_SESP 
Pes setting: tasks       is {'NTASKS_ATM': 1, 'NTASKS_LND': 1, 'NTASKS_ROF': 1, 'NTASKS_ICE': 1, 'NTASKS_OCN': 1, 'NTASKS_GLC': 1, 'NTASKS_WAV': 1, 'NTASKS_CPL': 1} 
Pes setting: threads     is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_GLC': 1, 'NTHRDS_WAV': 1, 'NTHRDS_CPL': 1} 
Pes setting: rootpe      is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 0, 'ROOTPE_GLC': 0, 'ROOTPE_WAV': 0, 'ROOTPE_CPL': 0} 
Pes setting: pstrid      is {} 
Pes setting: excl_stride is {} 
Pes other settings: {}
Pes other settings append: {}
Pes comments: none
setting additional fields from config_pes: {}, append {}
 Compset is: 2000_CAM60%SCAMATEX_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV_SESP 
 Grid is: a%ne16np4_l%null_oi%ne16np4_r%null_g%null_w%null_z%null_m%gx1v7 
 Components in compset are: ['cam', 'slnd', 'sice', 'docn', 'srof', 'sglc', 'swav', 'sesp'] 
Using project from config_machines.xml: none
No charge_account info available, using value from PROJECT
cesm model version found: cam_4ncar_20240605_2f3c896-3204-g1eca30798
Batch_system_type is none
 Creating Case directory /home/jenkins/workspace/cam_clubb_error_handling_test/cime/scripts/test_scam_ATEX_cam_clubb_error_handling_test
This component includes user_mods /home/jenkins/workspace/cam_clubb_error_handling_test/cime_config/usermods_dirs/scam_mandatory
Adding user mods directory /home/jenkins/workspace/cam_clubb_error_handling_test/cime_config/usermods_dirs/scam_mandatory
RUN: /home/jenkins/workspace/cam_clubb_error_handling_test/cime/scripts/test_scam_ATEX_cam_clubb_error_handling_test/shell_commands
FROM: /home/jenkins/workspace/cam_clubb_error_handling_test/cime/scripts/test_scam_ATEX_cam_clubb_error_handling_test
Invalid permissions to create /home/pub/cam_inputdata
ERROR: inputdata root is not a directory or is not readable: /home/pub/cam_inputdata
gzip: /home/jenkins/cam_output/scratch/test_scam_ATEX_cam_clubb_error_handling_test/run/cesm.log.*.gz: No such file or directory
mv: cannot stat '/home/jenkins/cam_output/scratch/test_scam_ATEX_cam_clubb_error_handling_test/run/cesm.log.*': No such file or directory
----- Case Setup -----
Did not find an alias or longname compset match for 2000_CAM60%SCAMATEX_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV 
Automatically adding SESP to compset
Compset forcing is 1972-2004
ATM component is CAM cam6 physics:
LND component is Stub land component
ICE component is Stub ice component
OCN component is DOCN   analytic aquaplanet sst - option 3
ROF component is Stub river component
GLC component is Stub glacier (land ice) component
WAV component is Stub wave component
ESP component is Stub external system processing (ESP) component
Pes     specification file is /home/jenkins/workspace/cam_clubb_error_handling_test/cime_config/config_pes.xml
Compset specific settings: name is RUN_STARTDATE and value is 1969-02-15
Compset specific settings: name is STOP_N and value is 47
Compset specific settings: name is STOP_OPTION and value is nhours
Compset specific settings: name is START_TOD and value is 0
Compset specific settings: name is PTS_MODE and value is TRUE
Compset specific settings: name is PTS_LAT and value is 15.0
Compset specific settings: name is PTS_LON and value is 345.0
Machine is larson-group
vid is comment
vid is ntasks
vid is nthrds
vid is rootpe
Pes setting: grid          is a%ne16np4_l%null_oi%ne16np4_r%null_g%null_w%null_z%null_m%gx1v7 
Pes setting: compset       is 2000_CAM60%SCAMATEX_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV_SESP 
Pes setting: tasks       is {'NTASKS_ATM': 1, 'NTASKS_LND': 1, 'NTASKS_ROF': 1, 'NTASKS_ICE': 1, 'NTASKS_OCN': 1, 'NTASKS_GLC': 1, 'NTASKS_WAV': 1, 'NTASKS_CPL': 1} 
Pes setting: threads     is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_GLC': 1, 'NTHRDS_WAV': 1, 'NTHRDS_CPL': 1} 
Pes setting: rootpe      is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 0, 'ROOTPE_GLC': 0, 'ROOTPE_WAV': 0, 'ROOTPE_CPL': 0} 
Pes setting: pstrid      is {} 
Pes setting: excl_stride is {} 
Pes other settings: {}
Pes other settings append: {}
Pes comments: none
setting additional fields from config_pes: {}, append {}
 Compset is: 2000_CAM60%SCAMATEX_SLND_SICE_DOCN%AQP3_SROF_SGLC_SWAV_SESP 
 Grid is: a%ne16np4_l%null_oi%ne16np4_r%null_g%null_w%null_z%null_m%gx1v7 
 Components in compset are: ['cam', 'slnd', 'sice', 'docn', 'srof', 'sglc', 'swav', 'sesp'] 
Using project from config_machines.xml: none
No charge_account info available, using value from PROJECT
cesm model version found: cam_4ncar_20240605_2f3c896-3204-g1eca30798
Batch_system_type is none
 Creating Case directory /home/jenkins/workspace/cam_clubb_error_handling_test/cime/scripts/test_scam_ATEX_cam_clubb_error_handling_test
This component includes user_mods /home/jenkins/workspace/cam_clubb_error_handling_test/cime_config/usermods_dirs/scam_mandatory
Adding user mods directory /home/jenkins/workspace/cam_clubb_error_handling_test/cime_config/usermods_dirs/scam_mandatory
RUN: /home/jenkins/workspace/cam_clubb_error_handling_test/cime/scripts/test_scam_ATEX_cam_clubb_error_handling_test/shell_commands
FROM: /home/jenkins/workspace/cam_clubb_error_handling_test/cime/scripts/test_scam_ATEX_cam_clubb_error_handling_test
Invalid permissions to create /home/pub/cam_inputdata
ERROR: inputdata root is not a directory or is not readable: /home/pub/cam_inputdata
gzip: /home/jenkins/cam_output/scratch/test_scam_ATEX_cam_clubb_error_handling_test/run/cesm.log.*.gz: No such file or directory
mv: cannot stat '/home/jenkins/cam_output/scratch/test_scam_ATEX_cam_clubb_error_handling_test/run/cesm.log.*': No such file or directory
</span><span class="pipeline-new-node" nodeId="38" startId="36">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="39" startId="35">[Pipeline] // dir
</span><span class="pipeline-new-node" nodeId="40" startId="34">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="41" startId="33">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="42" startId="42" enclosingId="13">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="43" startId="43" enclosingId="42" label="Check for wp2 NaN error from advance_wp2_wp3">[Pipeline] { (Check for wp2 NaN error from advance_wp2_wp3)
</span><span class="pipeline-new-node" nodeId="44" startId="44" enclosingId="43">[Pipeline] script
</span><span class="pipeline-new-node" nodeId="45" startId="45" enclosingId="44">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="46" enclosingId="45">[Pipeline] sh
</span><span class="pipeline-node-46">grep: run_scripts/advance_wp2_wp3_bad_wp2: No such file or directory
Error message 'Error calling advance_wp2_wp3' NOT found in log
CAM CLUBB error handling test for advance_wp2_wp3 FAILED
</span><span class="pipeline-new-node" nodeId="47" startId="45">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="48" startId="44">[Pipeline] // script
</span><span class="pipeline-new-node" nodeId="49" startId="43">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="50" startId="42">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="51" startId="51" enclosingId="13">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="52" startId="52" enclosingId="51" label="Check for wp2 NaN error from advance_wm_wpxp">[Pipeline] { (Check for wp2 NaN error from advance_wm_wpxp)
</span>Stage "Check for wp2 NaN error from advance_wm_wpxp" skipped due to earlier failure(s)
<span class="pipeline-new-node" nodeId="53" enclosingId="52">[Pipeline] getContext
</span><span class="pipeline-new-node" nodeId="54" startId="52">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="55" startId="51">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="56" startId="56" enclosingId="13">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="57" startId="57" enclosingId="56" label="Declarative: Post Actions">[Pipeline] { (Declarative: Post Actions)
</span><span class="pipeline-new-node" nodeId="58" startId="58" enclosingId="57">[Pipeline] script
</span><span class="pipeline-new-node" nodeId="59" startId="59" enclosingId="58">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="60" enclosingId="59">[Pipeline] cleanWs
</span><span class="pipeline-node-60">[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done
</span><span class="pipeline-new-node" nodeId="61" startId="59">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="62" startId="58">[Pipeline] // script
</span><span class="pipeline-new-node" nodeId="63" startId="63" enclosingId="57">[Pipeline] script
</span><span class="pipeline-new-node" nodeId="64" startId="64" enclosingId="63">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="65" enclosingId="64">[Pipeline] emailext
</span><span class="pipeline-node-65">Sending email to: messnermet@uwm.edu
</span><span class="pipeline-new-node" nodeId="66" startId="64">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="67" startId="63">[Pipeline] // script
</span><span class="pipeline-new-node" nodeId="68" startId="57">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="69" startId="56">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="70" startId="13">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="71" startId="12">[Pipeline] // withEnv
</span><span class="pipeline-new-node" nodeId="72" startId="11">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="73" startId="10">[Pipeline] // withEnv
</span><span class="pipeline-new-node" nodeId="74" startId="4">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="75" startId="3">[Pipeline] // node
</span><span class="pipeline-new-node" nodeId="76" startId="2">[Pipeline] End of Pipeline
</span>ERROR: script returned exit code 1
Finished: FAILURE