Started by user <a href='/jenkins/user/huebler' class='jenkins-table__link model-link model-link--float'>Gunther Huebler</a>
Obtained jenkins_tests/cam_gpu_test/Jenkinsfile from git <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a>
<span class="pipeline-new-node" nodeId="2" startId="2">[Pipeline] Start of Pipeline
</span><span class="pipeline-new-node" nodeId="3" startId="3" enclosingId="2">[Pipeline] node
</span><span class="pipeline-node-3">Running on <a href='/jenkins/computer/nelson/' class='jenkins-table__link model-link model-link--float'>nelson</a> in /home/jenkins/workspace/cam_gpu_test
</span><span class="pipeline-new-node" nodeId="4" startId="4" enclosingId="3">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="5" startId="5" enclosingId="4">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="6" startId="6" enclosingId="5" label="Declarative: Checkout SCM">[Pipeline] { (Declarative: Checkout SCM)
</span><span class="pipeline-new-node" nodeId="7" enclosingId="6">[Pipeline] checkout
</span><span class="pipeline-node-7">The recommended git tool is: git
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a>
 &gt; git init /home/jenkins/workspace/cam_gpu_test # timeout=10
Fetching upstream changes from <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a>
 &gt; git --version # timeout=10
 &gt; git --version # 'git version 2.34.1'
using GIT_ASKPASS to set credentials larsongroupsysadmin github token
 &gt; git fetch --tags --force --progress -- <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a> +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 8e7ddbdcdd022b2a08ec537f4bde139f940f0209 (refs/remotes/origin/clubb_silhs_devel)
 &gt; git config remote.origin.url <a href='https://github.com/larson-group/cam.git'>https://github.com/larson-group/cam.git</a> # timeout=10
 &gt; git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 &gt; git rev-parse refs/remotes/origin/clubb_silhs_devel^{commit} # timeout=10
 &gt; git config core.sparsecheckout # timeout=10
 &gt; git checkout -f 8e7ddbdcdd022b2a08ec537f4bde139f940f0209 # timeout=10
Commit message: "Merge pull request #218 from larson-group/cam6_4_183_clubb_silhs_merge"
 &gt; git rev-list --no-walk 692f0d3e4af08e9083d2c1fd9a3cd440949f9d89 # timeout=10
</span><span class="pipeline-new-node" nodeId="8" startId="6">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="9" startId="5">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="10" startId="10" enclosingId="4">[Pipeline] withEnv
</span><span class="pipeline-new-node" nodeId="11" startId="11" enclosingId="10">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="12" startId="12" enclosingId="11">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="13" startId="13" enclosingId="12" label="Checkout Externals and Copy Custom Files">[Pipeline] { (Checkout Externals and Copy Custom Files)
</span><span class="pipeline-new-node" nodeId="14" enclosingId="13">[Pipeline] sh
</span><span class="pipeline-node-14">+ python run_scripts/set_up_repo.py .
Checking out externals...
Sparse checkout cam-sima fxsparse ../.sima_phys_utils_sparse_checkout
Successfully checked out             cam-sima at sima0_07_000
            cam-sima up to date.
Sparse checkout mpas fxsparse ../.mpas_sparse_checkout
Successfully checked out                 mpas at v8.4.0
                mpas up to date.
Sparse checkout cosp2 fxsparse ../.cosp_sparse_checkout
Successfully checked out                cosp2 at v2.2.1
               cosp2 up to date.
Sparse checkout ext_co2_cooling fxsparse ../.ext_co2_cooling_sparse_checkout
Successfully checked out      ext_co2_cooling at v1.1
     ext_co2_cooling up to date.
           chem_proc up to date.
               carma up to date.
        pumas-frozen up to date.
            ali_arms up to date.
          atmos_phys up to date.
Recursively checking out submodules of atmos_phys
                 fv3 up to date.
Recursively checking out submodules of fv3
             cloud_j up to date.
                hetp up to date.
               hemco up to date.
Recursively checking out submodules of hemco
         rrtmgp-data up to date.
                 rtm up to date.
              mosart up to date.
                 fms up to date.
Recursively checking out submodules of fms
          ccs_config up to date.
               cdeps up to date.
Recursively checking out submodules of cdeps
               share up to date.
               cmeps up to date.
                cice up to date.
Recursively checking out submodules of cice
         tools/CUPiD up to date.
Recursively checking out submodules of tools/CUPiD
               tuv-x up to date.
            geoschem up to date.
Recursively checking out submodules of geoschem
          parallelio up to date.
Recursively checking out submodules of HEMCO
           mizuRoute up to date.
Recursively checking out submodules of mizuRoute
                 clm up to date.
Recursively checking out submodules of clm
                cime up to date.
Recursively checking out submodules of cime
Recursively checking out submodules of src
Recursively checking out submodules of fates
Copying ccs_config files
Restricting NVHPC OpenACC rules to CLUBB
Applying CAM-SIMA Filepath workaround
</span><span class="pipeline-new-node" nodeId="15" startId="13">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="16" startId="12">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="17" startId="17" enclosingId="11">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="18" startId="18" enclosingId="17" label="Remove Old Output">[Pipeline] { (Remove Old Output)
</span><span class="pipeline-new-node" nodeId="19" enclosingId="18">[Pipeline] sh
</span><span class="pipeline-node-19">+ rm -rf /home/jenkins/cam_output/scratch/UWM_ne3_gpu_no_stats_larson-group_nvhpc/ /home/jenkins/cam_output/caseroot/UWM_ne3_gpu_no_stats_larson-group_nvhpc/
</span><span class="pipeline-new-node" nodeId="20" startId="18">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="21" startId="17">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="22" startId="22" enclosingId="11">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="23" startId="23" enclosingId="22" label="Running ne3 with OpenACC">[Pipeline] { (Running ne3 with OpenACC)
</span><span class="pipeline-new-node" nodeId="24" enclosingId="23">[Pipeline] sh
</span><span class="pipeline-node-24">+ source /etc/profile.d/larson-group.sh
++ export GIT_EDITOR=vi
++ GIT_EDITOR=vi
++ export SVN_EDITOR=vi
++ SVN_EDITOR=vi
++ export OMP_STACKSIZE=1048579
++ OMP_STACKSIZE=1048579
++ export LMOD_ROOT=/opt/lmod/
++ LMOD_ROOT=/opt/lmod/
++ source /opt/lmod//lmod/lmod/init/bash
+++ '[' -z '' ']'
+++ case "$-" in
+++ __lmod_vx=x
+++ '[' -n x ']'
+++ set +x
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/bash)
Shell debugging restarted
+++ unset __lmod_vx
+++ find /usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core -print -quit
++ export MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core
++ MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core
+ ulimit -s 8388608
+ run_scripts/run_cesm_uwm_coarse_res_gpu_no_silhs.sh
Removing old code directory /home/jenkins/cam_output/caseroot/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc in 10 seconds.
To abort, press ctrl-c
----- Case Setup -----
Compset longname is HIST_CAM70%LT_CLM60%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV
Compset specification file is /home/jenkins/workspace/cam_gpu_test/cime_config/config_compsets.xml
Automatically adding SESP to compset
Compset forcing is Historic transient 
ATM component is CAM cam7 physics:CAM low top model 
LND component is clm6.0:Satellite phenology:
ICE component is Sea ICE (cice) model version 6 :prescribed cice
OCN component is DOCN   prescribed ocean mode
ROF component is MOSART: MOdel for Scale Adaptive River Transport
GLC component is Stub glacier (land ice) component
WAV component is Stub wave component
ESP component is Stub external system processing (ESP) component
Pes     specification file is /home/jenkins/workspace/cam_gpu_test/cime_config/config_pes.xml
Compset specific settings: name is RUN_STARTDATE and value is 1979-01-01
Compset specific settings: name is LND_SETS_DUST_EMIS_DRV_FLDS and value is FALSE
Machine is larson-group
Pes setting: grid match    is a%ne3np4 
Pes setting: grid          is a%ne3np4.pg3_l%ne3np4.pg3_oi%ne3np4.pg3_r%r05_g%null_w%null_z%null_m%gx3v7 
Pes setting: compset       is HIST_CAM70%LT_CLM60%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV_SESP 
Pes setting: tasks       is {'NTASKS_ATM': 24, 'NTASKS_LND': 24, 'NTASKS_ROF': 24, 'NTASKS_ICE': 24, 'NTASKS_OCN': 24, 'NTASKS_GLC': 24, 'NTASKS_WAV': 24, 'NTASKS_CPL': 24} 
Pes setting: threads     is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_GLC': 1, 'NTHRDS_WAV': 1, 'NTHRDS_CPL': 1} 
Pes setting: rootpe      is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 0, 'ROOTPE_GLC': 0, 'ROOTPE_WAV': 0, 'ROOTPE_CPL': 0} 
Pes setting: pstrid      is {} 
Pes setting: excl_stride is {} 
Pes other settings: {}
Pes other settings append: {}
Pes comments: none
setting additional fields from config_pes: {}, append {}
 Compset is: HIST_CAM70%LT_CLM60%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV_SESP 
 Grid is: a%ne3np4.pg3_l%ne3np4.pg3_oi%ne3np4.pg3_r%r05_g%null_w%null_z%null_m%gx3v7 
 Components in compset are: ['cam', 'clm', 'cice', 'docn', 'mosart', 'sglc', 'swav', 'sesp'] 
Using project from config_machines.xml: none
No charge_account info available, using value from PROJECT
cesm model version found: cam_4ncar_20240605_2f3c896-3134-g8e7ddbdcd
Batch_system_type is none
 Creating Case directory /home/jenkins/cam_output/caseroot/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc

WARNING: Macros of the form COMPILER_MACHINE.cmake are deprecated and should be replaced with the form MACHINE_COMPILER.cmake

Creating batch scripts
Writing case.run script from input template /home/jenkins/workspace/cam_gpu_test/ccs_config/machines/template.case.run
Creating file .case.run
Writing case.st_archive script from input template /home/jenkins/workspace/cam_gpu_test/ccs_config/machines/template.st_archive
Creating file case.st_archive
Writing case.cupid script from input template /home/jenkins/workspace/cam_gpu_test/ccs_config/machines/template.cupid
Creating file case.cupid
Creating user_nl_xxx files for components and cpl
Running cam.case_setup.py
If an old case build already exists, might want to run 'case.build --clean' before building
You can now run './preview_run' to get more info on how your case will be run
----- Compile Configuration -----
----- Compile -----
Building case in directory /home/jenkins/cam_output/caseroot/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
sharedlib_only is False
model_only is False
Generating component namelists as part of build
  2026-06-29 21:30:45 atm 
Create namelist for component cam
   Calling /home/jenkins/workspace/cam_gpu_test/cime_config/buildnml
     ...calling cam buildcpp to set build time options
CAM namelist copy: file1 /home/jenkins/cam_output/caseroot/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/Buildconf/camconf/atm_in file2 /home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/run/atm_in 
  2026-06-29 21:30:46 lnd 
Create namelist for component clm
   Calling /home/jenkins/workspace/cam_gpu_test/components/clm//cime_config/buildnml
  2026-06-29 21:30:46 ice 
Create namelist for component cice
   Calling /home/jenkins/workspace/cam_gpu_test/components/cice//cime_config/buildnml
RUN: /home/jenkins/workspace/cam_gpu_test/components/cice/bld/generate_cice_decomp.pl -ccsmroot /home/jenkins/workspace/cam_gpu_test -res ne3np4.pg3 -nx 486 -ny 1 -nproc 1 -thrds 1 -output all 
FROM: /home/jenkins/cam_output/caseroot/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
  output: 486 1 486 1 1 roundrobin square-ice

  2026-06-29 21:30:46 ocn 
Create namelist for component docn
   Calling /home/jenkins/workspace/cam_gpu_test/components/cdeps/docn/cime_config/buildnml
docn_mode is prescribed
  2026-06-29 21:30:46 rof 
Create namelist for component mosart
   Calling /home/jenkins/workspace/cam_gpu_test/components/mosart//cime_config/buildnml
  2026-06-29 21:30:46 glc 
Create namelist for component sglc
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/sglc/cime_config/buildnml
  2026-06-29 21:30:46 wav 
Create namelist for component swav
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/swav/cime_config/buildnml
  2026-06-29 21:30:46 esp 
Create namelist for component sesp
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/sesp/cime_config/buildnml
  2026-06-29 21:30:46 cpl 
Create namelist for component drv
   Calling /home/jenkins/workspace/cam_gpu_test/components/cmeps/cime_config/buildnml
Writing nuopc_runconfig for components ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF']
libs from case_support_libraries ['gptl', 'pio', 'csm_share', 'FTorch', 'CDEPS']
Building gptl with output to file /home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/gptl.bldlog.260629-213045
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/build_scripts/buildlib.gptl
Building pio with output to file /home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/pio.bldlog.260629-213045
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/build_scripts/buildlib.pio
Building csm_share with output to file /home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/csm_share.bldlog.260629-213045
   Calling /home/jenkins/workspace/cam_gpu_test/share/buildlib.csm_share
Build script /home/jenkins/workspace/cam_gpu_test/libraries/FTorch/buildlib for component FTorch not found.
Building CDEPS with output to file /home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/CDEPS.bldlog.260629-213045
   Calling /home/jenkins/workspace/cam_gpu_test/components/cdeps/cime_config/buildlib
ERROR: /home/jenkins/workspace/cam_gpu_test/components/cdeps/cime_config/buildlib FAILED, cat /home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/CDEPS.bldlog.260629-213045
Error building CAM
</span><span class="pipeline-new-node" nodeId="25" startId="23">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="26" startId="22">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="27" startId="27" enclosingId="11">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="28" startId="28" enclosingId="27" label="Declarative: Post Actions">[Pipeline] { (Declarative: Post Actions)
</span><span class="pipeline-new-node" nodeId="29" startId="29" enclosingId="28">[Pipeline] script
</span><span class="pipeline-new-node" nodeId="30" startId="30" enclosingId="29">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="31" enclosingId="30">[Pipeline] sh
</span><span class="pipeline-node-31">+ chmod -R 755 /home/jenkins/cam_output
</span><span class="pipeline-new-node" nodeId="32" enclosingId="30">[Pipeline] sh
</span><span class="pipeline-node-32">+ gzip -d '/home/jenkins/cam_output/scratch/UWM_ne3_gpu_no_stats_larson-group_nvhpc/bld/cesm.bldlog.*.gz'
gzip: /home/jenkins/cam_output/scratch/UWM_ne3_gpu_no_stats_larson-group_nvhpc/bld/cesm.bldlog.*.gz: No such file or directory
</span><span class="pipeline-new-node" nodeId="33" enclosingId="30">[Pipeline] echo
</span><span class="pipeline-node-33">WARNING: One or more of the build logs were not zipped, this means compilation did not finish
</span><span class="pipeline-new-node" nodeId="34" enclosingId="30">[Pipeline] sh
</span><span class="pipeline-node-34">+ gzip -d '/home/jenkins/cam_output/scratch/UWM_ne3_gpu_no_stats_larson-group_nvhpc/run/cesm.log.*.gz'
gzip: /home/jenkins/cam_output/scratch/UWM_ne3_gpu_no_stats_larson-group_nvhpc/run/cesm.log.*.gz: No such file or directory
</span><span class="pipeline-new-node" nodeId="35" enclosingId="30">[Pipeline] echo
</span><span class="pipeline-node-35">WARNING: One or more of the run logs were not zipped, this means the code did not run to completion
</span><span class="pipeline-new-node" nodeId="36" enclosingId="30">[Pipeline] sh
</span><span class="pipeline-node-36">+ cat '/home/jenkins/cam_output/scratch/UWM_ne3_gpu_no_stats_larson-group_nvhpc/bld/atm.bldlog.*'
cat: '/home/jenkins/cam_output/scratch/UWM_ne3_gpu_no_stats_larson-group_nvhpc/bld/atm.bldlog.*': No such file or directory
</span><span class="pipeline-new-node" nodeId="37" enclosingId="30">[Pipeline] echo
</span><span class="pipeline-node-37">WARNING: One of the log files (build or run) were not found - the failure must have occured before their creation
</span><span class="pipeline-new-node" nodeId="38" startId="30">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="39" startId="29">[Pipeline] // script
</span><span class="pipeline-new-node" nodeId="40" startId="40" enclosingId="28">[Pipeline] script
</span><span class="pipeline-new-node" nodeId="41" startId="41" enclosingId="40">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="42" startId="41">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="43" startId="40">[Pipeline] // script
</span><span class="pipeline-new-node" nodeId="44" startId="28">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="45" startId="27">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="46" startId="11">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="47" startId="10">[Pipeline] // withEnv
</span><span class="pipeline-new-node" nodeId="48" startId="4">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="49" startId="3">[Pipeline] // node
</span><span class="pipeline-new-node" nodeId="50" startId="2">[Pipeline] End of Pipeline
</span>ERROR: script returned exit code 1
Finished: FAILURE