Skip to content
Success

Console Output

Started by an SCM change
Obtained jenkins_tests/cam_gpu_test/Jenkinsfile from git https://github.com/larson-group/cam.git
[Pipeline] Start of Pipeline
[Pipeline] node
Running on nelson in /home/jenkins/workspace/cam_gpu_test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: git
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository https://github.com/larson-group/cam.git
 > git init /home/jenkins/workspace/cam_gpu_test # timeout=10
Fetching upstream changes from https://github.com/larson-group/cam.git
 > git --version # timeout=10
 > git --version # 'git version 2.34.1'
using GIT_ASKPASS to set credentials larsongroupsysadmin github token
 > git fetch --tags --force --progress -- https://github.com/larson-group/cam.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision d8aecdb15bc1aa14eb613f8841c6edea58a88750 (refs/remotes/origin/clubb_silhs_devel)
Commit message: "Pumas fix test (#220)"
 > git config remote.origin.url https://github.com/larson-group/cam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse refs/remotes/origin/clubb_silhs_devel^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d8aecdb15bc1aa14eb613f8841c6edea58a88750 # timeout=10
 > git rev-list --no-walk d0a7b41dce2ec8ab0dbd72951960f3ae7cc7f6bd # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Checkout Externals and Copy Custom Files)
[Pipeline] sh
+ python run_scripts/set_up_repo.py .
Checking out externals...
Sparse checkout cam-sima fxsparse ../.sima_phys_utils_sparse_checkout
Successfully checked out             cam-sima at sima0_07_000
            cam-sima up to date.
Sparse checkout mpas fxsparse ../.mpas_sparse_checkout
Successfully checked out                 mpas at v8.4.0
                mpas up to date.
Sparse checkout cosp2 fxsparse ../.cosp_sparse_checkout
Successfully checked out                cosp2 at v2.2.1
               cosp2 up to date.
Sparse checkout ext_co2_cooling fxsparse ../.ext_co2_cooling_sparse_checkout
Successfully checked out      ext_co2_cooling at v1.1
     ext_co2_cooling up to date.
           chem_proc up to date.
               carma up to date.
        pumas-frozen up to date.
            ali_arms up to date.
          atmos_phys up to date.
Recursively checking out submodules of atmos_phys
                 fv3 up to date.
Recursively checking out submodules of fv3
             cloud_j up to date.
                hetp up to date.
               hemco up to date.
Recursively checking out submodules of hemco
         rrtmgp-data up to date.
                 rtm up to date.
              mosart up to date.
          ccs_config up to date.
                 fms up to date.
Recursively checking out submodules of fms
               cdeps up to date.
Recursively checking out submodules of cdeps
               share up to date.
         tools/CUPiD up to date.
Recursively checking out submodules of tools/CUPiD
               cmeps up to date.
               tuv-x up to date.
           mizuRoute up to date.
Recursively checking out submodules of mizuRoute
                cice up to date.
Recursively checking out submodules of cice
          parallelio up to date.
            geoschem up to date.
Recursively checking out submodules of geoschem
Recursively checking out submodules of HEMCO
Recursively checking out submodules of src
                 clm up to date.
Recursively checking out submodules of clm
                cime up to date.
Recursively checking out submodules of cime
Recursively checking out submodules of fates
Copying ccs_config files
Restricting NVHPC OpenACC rules to CLUBB
Applying CAM-SIMA Filepath workaround
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Remove Old Output)
[Pipeline] sh
+ rm -rf /home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc /home/jenkins/cam_output/caseroot/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Running ne3 with OpenACC)
[Pipeline] sh
+ source /etc/profile.d/larson-group.sh
++ export GIT_EDITOR=vi
++ GIT_EDITOR=vi
++ export SVN_EDITOR=vi
++ SVN_EDITOR=vi
++ export OMP_STACKSIZE=1048579
++ OMP_STACKSIZE=1048579
++ export LMOD_ROOT=/opt/lmod/
++ LMOD_ROOT=/opt/lmod/
++ source /opt/lmod//lmod/lmod/init/bash
+++ '[' -z '' ']'
+++ case "$-" in
+++ __lmod_vx=x
+++ '[' -n x ']'
+++ set +x
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/bash)
Shell debugging restarted
+++ unset __lmod_vx
+++ find /usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core -print -quit
++ export MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core
++ MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core
+ ulimit -s 8388608
+ run_scripts/run_cesm_uwm_coarse_res_gpu_no_silhs.sh
Removing old case directories in 5 seconds:
  /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
  /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
To abort, press ctrl-c
----- Case Setup -----
Compset longname is HIST_CAM70%MT_CLM60%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV
Compset specification file is /home/jenkins/workspace/cam_gpu_test/cime_config/config_compsets.xml
Automatically adding SESP to compset
Compset forcing is Historic transient 
ATM component is CAM cam7 physics:CAM mid top model 
LND component is clm6.0:Satellite phenology:
ICE component is Sea ICE (cice) model version 6 :prescribed cice
OCN component is DOCN   prescribed ocean mode
ROF component is MOSART: MOdel for Scale Adaptive River Transport
GLC component is Stub glacier (land ice) component
WAV component is Stub wave component
ESP component is Stub external system processing (ESP) component
Pes     specification file is /home/jenkins/workspace/cam_gpu_test/cime_config/config_pes.xml
Compset specific settings: name is RUN_STARTDATE and value is 1979-01-01
Compset specific settings: name is LND_SETS_DUST_EMIS_DRV_FLDS and value is FALSE
Machine is larson-group
Pes setting: grid match    is a%ne3np4 
Pes setting: grid          is a%ne3np4.pg3_l%ne3np4.pg3_oi%ne3np4.pg3_r%r05_g%null_w%null_z%null_m%gx3v7 
Pes setting: compset       is HIST_CAM70%MT_CLM60%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV_SESP 
Pes setting: tasks       is {'NTASKS_ATM': 24, 'NTASKS_LND': 24, 'NTASKS_ROF': 24, 'NTASKS_ICE': 24, 'NTASKS_OCN': 24, 'NTASKS_GLC': 24, 'NTASKS_WAV': 24, 'NTASKS_CPL': 24} 
Pes setting: threads     is {'NTHRDS_ATM': 1, 'NTHRDS_LND': 1, 'NTHRDS_ROF': 1, 'NTHRDS_ICE': 1, 'NTHRDS_OCN': 1, 'NTHRDS_GLC': 1, 'NTHRDS_WAV': 1, 'NTHRDS_CPL': 1} 
Pes setting: rootpe      is {'ROOTPE_ATM': 0, 'ROOTPE_LND': 0, 'ROOTPE_ROF': 0, 'ROOTPE_ICE': 0, 'ROOTPE_OCN': 0, 'ROOTPE_GLC': 0, 'ROOTPE_WAV': 0, 'ROOTPE_CPL': 0} 
Pes setting: pstrid      is {} 
Pes setting: excl_stride is {} 
Pes other settings: {}
Pes other settings append: {}
Pes comments: none
setting additional fields from config_pes: {}, append {}
 Compset is: HIST_CAM70%MT_CLM60%SP_CICE%PRES_DOCN%DOM_MOSART_SGLC_SWAV_SESP 
 Grid is: a%ne3np4.pg3_l%ne3np4.pg3_oi%ne3np4.pg3_r%r05_g%null_w%null_z%null_m%gx3v7 
 Components in compset are: ['cam', 'clm', 'cice', 'docn', 'mosart', 'sglc', 'swav', 'sesp'] 
Using project from config_machines.xml: none
No charge_account info available, using value from PROJECT
cesm model version found: cam_4ncar_20240605_2f3c896-3144-gd8aecdb15
Batch_system_type is none
 Creating Case directory /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc

WARNING: Macros of the form COMPILER_MACHINE.cmake are deprecated and should be replaced with the form MACHINE_COMPILER.cmake

Creating batch scripts
Writing case.run script from input template /home/jenkins/workspace/cam_gpu_test/ccs_config/machines/template.case.run
Creating file .case.run
Writing case.st_archive script from input template /home/jenkins/workspace/cam_gpu_test/ccs_config/machines/template.st_archive
Creating file case.st_archive
Writing case.cupid script from input template /home/jenkins/workspace/cam_gpu_test/ccs_config/machines/template.cupid
Creating file case.cupid
Creating user_nl_xxx files for components and cpl
Running cam.case_setup.py
If an old case build already exists, might want to run 'case.build --clean' before building
You can now run './preview_run' to get more info on how your case will be run
----- Compile Configuration -----
----- Compile -----
Building case in directory /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
sharedlib_only is False
model_only is False
Generating component namelists as part of build
  2026-07-08 03:17:25 atm 
Create namelist for component cam
   Calling /home/jenkins/workspace/cam_gpu_test/cime_config/buildnml
     ...calling cam buildcpp to set build time options
CAM namelist copy: file1 /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/Buildconf/camconf/atm_in file2 /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/run/atm_in 
  2026-07-08 03:17:25 lnd 
Create namelist for component clm
   Calling /home/jenkins/workspace/cam_gpu_test/components/clm//cime_config/buildnml
  2026-07-08 03:17:25 ice 
Create namelist for component cice
   Calling /home/jenkins/workspace/cam_gpu_test/components/cice//cime_config/buildnml
RUN: /home/jenkins/workspace/cam_gpu_test/components/cice/bld/generate_cice_decomp.pl -ccsmroot /home/jenkins/workspace/cam_gpu_test -res ne3np4.pg3 -nx 486 -ny 1 -nproc 1 -thrds 1 -output all 
FROM: /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
  output: 486 1 486 1 1 roundrobin square-ice

  2026-07-08 03:17:25 ocn 
Create namelist for component docn
   Calling /home/jenkins/workspace/cam_gpu_test/components/cdeps/docn/cime_config/buildnml
docn_mode is prescribed
  2026-07-08 03:17:25 rof 
Create namelist for component mosart
   Calling /home/jenkins/workspace/cam_gpu_test/components/mosart//cime_config/buildnml
  2026-07-08 03:17:25 glc 
Create namelist for component sglc
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/sglc/cime_config/buildnml
  2026-07-08 03:17:25 wav 
Create namelist for component swav
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/swav/cime_config/buildnml
  2026-07-08 03:17:25 esp 
Create namelist for component sesp
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/sesp/cime_config/buildnml
  2026-07-08 03:17:26 cpl 
Create namelist for component drv
   Calling /home/jenkins/workspace/cam_gpu_test/components/cmeps/cime_config/buildnml
Writing nuopc_runconfig for components ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF']
libs from case_support_libraries ['gptl', 'pio', 'csm_share', 'FTorch', 'CDEPS']
Building gptl with output to file /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/gptl.bldlog.260708-031725
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/build_scripts/buildlib.gptl
Building pio with output to file /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/pio.bldlog.260708-031725
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/build_scripts/buildlib.pio
Building csm_share with output to file /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/csm_share.bldlog.260708-031725
   Calling /home/jenkins/workspace/cam_gpu_test/share/buildlib.csm_share
Build script /home/jenkins/workspace/cam_gpu_test/libraries/FTorch/buildlib for component FTorch not found.
Building CDEPS with output to file /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/CDEPS.bldlog.260708-031725
   Calling /home/jenkins/workspace/cam_gpu_test/components/cdeps/cime_config/buildlib
         - Building clm library 
Building lnd with output to /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/lnd.bldlog.260708-031725
clm built in 637.596003 seconds
         - Building atm Library 
Building atm with output to /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/atm.bldlog.260708-031725
         - Building ice Library 
Building ice with output to /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/ice.bldlog.260708-031725
         - Building ocn Library 
Building ocn with output to /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/ocn.bldlog.260708-031725
         - Building rof Library 
Building rof with output to /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/rof.bldlog.260708-031725
docn built in 0.518196 seconds
mosart built in 77.060220 seconds
cice built in 203.726109 seconds
nelson seems to be removed or offline (java.lang.InterruptedException); will wait for 5 min 0 sec for it to come back online
nelson is back online
nelson seems to be removed or offline (java.lang.InterruptedException); will wait for 5 min 0 sec for it to come back online
nelson is back online
cam built in 825.628435 seconds
Building cesm from /home/jenkins/workspace/cam_gpu_test/components/cmeps/cime_config/buildexe with output to /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/cesm.bldlog.260708-031725 
nelson seems to be removed or offline (java.lang.InterruptedException); will wait for 5 min 0 sec for it to come back online
nelson is back online
Total build time: 1815.963430 seconds
MODEL BUILD HAS FINISHED SUCCESSFULLY
----- Run configuration -----
----- Run Submission -----
  2026-07-08 03:47:44 atm 
Create namelist for component cam
   Calling /home/jenkins/workspace/cam_gpu_test/cime_config/buildnml
CAM namelist copy: file1 /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/Buildconf/camconf/atm_in file2 /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/run/atm_in 
  2026-07-08 03:47:44 lnd 
Create namelist for component clm
   Calling /home/jenkins/workspace/cam_gpu_test/components/clm//cime_config/buildnml
  2026-07-08 03:47:44 ice 
Create namelist for component cice
   Calling /home/jenkins/workspace/cam_gpu_test/components/cice//cime_config/buildnml
RUN: /home/jenkins/workspace/cam_gpu_test/components/cice/bld/generate_cice_decomp.pl -ccsmroot /home/jenkins/workspace/cam_gpu_test -res ne3np4.pg3 -nx 486 -ny 1 -nproc 1 -thrds 1 -output all 
FROM: /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
  output: 486 1 486 1 1 roundrobin square-ice

  2026-07-08 03:47:44 ocn 
Create namelist for component docn
   Calling /home/jenkins/workspace/cam_gpu_test/components/cdeps/docn/cime_config/buildnml
docn_mode is prescribed
  2026-07-08 03:47:44 rof 
Create namelist for component mosart
   Calling /home/jenkins/workspace/cam_gpu_test/components/mosart//cime_config/buildnml
  2026-07-08 03:47:44 glc 
Create namelist for component sglc
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/sglc/cime_config/buildnml
  2026-07-08 03:47:44 wav 
Create namelist for component swav
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/swav/cime_config/buildnml
  2026-07-08 03:47:44 esp 
Create namelist for component sesp
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/sesp/cime_config/buildnml
  2026-07-08 03:47:44 cpl 
Create namelist for component drv
   Calling /home/jenkins/workspace/cam_gpu_test/components/cmeps/cime_config/buildnml
Writing nuopc_runconfig for components ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF']
Checking that inputdata is available as part of case submission
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/docn.input_data_list'
Loading input file list: 'Buildconf/ctsm.input_data_list'
Check case OK
submit_jobs case.run
Submit job case.run
Starting job script case.run
Generating namelists for /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
  2026-07-08 03:47:45 atm 
Create namelist for component cam
   Calling /home/jenkins/workspace/cam_gpu_test/cime_config/buildnml
CAM namelist copy: file1 /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/Buildconf/camconf/atm_in file2 /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/run/atm_in 
  2026-07-08 03:47:46 lnd 
Create namelist for component clm
   Calling /home/jenkins/workspace/cam_gpu_test/components/clm//cime_config/buildnml
  2026-07-08 03:47:46 ice 
Create namelist for component cice
   Calling /home/jenkins/workspace/cam_gpu_test/components/cice//cime_config/buildnml
RUN: /home/jenkins/workspace/cam_gpu_test/components/cice/bld/generate_cice_decomp.pl -ccsmroot /home/jenkins/workspace/cam_gpu_test -res ne3np4.pg3 -nx 486 -ny 1 -nproc 1 -thrds 1 -output all 
FROM: /home/jenkins/cam_output/caseroot/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc
  output: 486 1 486 1 1 roundrobin square-ice

  2026-07-08 03:47:46 ocn 
Create namelist for component docn
   Calling /home/jenkins/workspace/cam_gpu_test/components/cdeps/docn/cime_config/buildnml
docn_mode is prescribed
  2026-07-08 03:47:46 rof 
Create namelist for component mosart
   Calling /home/jenkins/workspace/cam_gpu_test/components/mosart//cime_config/buildnml
  2026-07-08 03:47:46 glc 
Create namelist for component sglc
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/sglc/cime_config/buildnml
  2026-07-08 03:47:46 wav 
Create namelist for component swav
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/swav/cime_config/buildnml
  2026-07-08 03:47:46 esp 
Create namelist for component sesp
   Calling /home/jenkins/workspace/cam_gpu_test/cime/CIME/non_py/src/components/stub_comps_nuopc/sesp/cime_config/buildnml
  2026-07-08 03:47:46 cpl 
Create namelist for component drv
   Calling /home/jenkins/workspace/cam_gpu_test/components/cmeps/cime_config/buildnml
Writing nuopc_runconfig for components ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF']
-------------------------------------------------------------------------
 - Prestage required restarts into /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/run
 - Case input data directory (DIN_LOC_ROOT) is /home/pub/cam_inputdata 
 - Checking for required input datasets in DIN_LOC_ROOT
-------------------------------------------------------------------------
run command is mpirun  -np 1 /home/jenkins/cam_output/scratch/UWM_FHISTC_MTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/cesm.exe   >> cesm.log.$LID 2>&1  
nelson seems to be removed or offline (java.lang.InterruptedException); will wait for 5 min 0 sec for it to come back online
nelson is back online
nelson seems to be removed or offline (java.lang.InterruptedException); will wait for 5 min 0 sec for it to come back online
nelson is back online
nelson seems to be removed or offline (java.lang.InterruptedException); will wait for 5 min 0 sec for it to come back online
nelson is back online
nelson seems to be removed or offline (java.lang.InterruptedException); will wait for 5 min 0 sec for it to come back online
nelson is back online
nelson seems to be removed or offline (java.lang.InterruptedException); will wait for 5 min 0 sec for it to come back online
nelson is back online
nelson seems to be removed or offline (java.lang.InterruptedException); will wait for 5 min 0 sec for it to come back online
nelson is back online
check for resubmit
dout_s False 
mach larson-group 
resubmit_num 0
Submitted job case.run with id None
Success!
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ chmod -R 755 /home/jenkins/cam_output
[Pipeline] sh
+ gzip -d '/home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/cesm.bldlog.*.gz'
gzip: /home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/cesm.bldlog.*.gz: No such file or directory
[Pipeline] echo
WARNING: One or more of the build logs were not zipped, this means compilation did not finish
[Pipeline] sh
+ gzip -d '/home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/run/cesm.log.*.gz'
gzip: /home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/run/cesm.log.*.gz: No such file or directory
[Pipeline] echo
WARNING: One or more of the run logs were not zipped, this means the code did not run to completion
[Pipeline] sh
+ cat '/home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/atm.bldlog.*'
cat: '/home/jenkins/cam_output/scratch/UWM_FHISTC_LTso_cam7_ne3pg3_gpu_no_silhs_larson-group_nvhpc/bld/atm.bldlog.*': No such file or directory
[Pipeline] echo
WARNING: One of the log files (build or run) were not found - the failure must have occured before their creation
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS