Started by user Keegan Rhodes Checking out git https://github.com/larson-group/e3sm.git into /home/jenkins/workspace/branch_e3sm_test@script to read jenkins_tests/e3sm_ne4_silhs_run_gfortran_test/Jenkinsfile The recommended git tool is: git > git rev-parse --resolve-git-dir /home/jenkins/workspace/branch_e3sm_test@script/.git # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/larson-group/e3sm.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Fetching upstream changes from https://github.com/larson-group/e3sm.git > git --version # timeout=10 > git --version # 'git version 1.8.3.1' using GIT_ASKPASS to set credentials A token based key used by Jenkins to preform Github actions, created 6/21/2021 > git fetch --tags --progress https://github.com/larson-group/e3sm.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse refs/remotes/origin/stats_to_hosts^{commit} # timeout=10 Checking out Revision 89a5bb84d2469f135f7d600724cf0471a85f6cdc (refs/remotes/origin/stats_to_hosts) > git config core.sparsecheckout # timeout=10 > git checkout -f 89a5bb84d2469f135f7d600724cf0471a85f6cdc # timeout=10 Commit message: "Reflecting changes to nightly tests clubb_intr" > git rev-list --no-walk 89a5bb84d2469f135f7d600724cf0471a85f6cdc # timeout=10 Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /home/jenkins/workspace/branch_e3sm_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git > git rev-parse --resolve-git-dir /home/jenkins/workspace/branch_e3sm_test/.git # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/larson-group/e3sm.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Fetching upstream changes from https://github.com/larson-group/e3sm.git > git --version # timeout=10 > git --version # 'git version 1.8.3.1' using GIT_ASKPASS to set credentials A token based key used by Jenkins to preform Github actions, created 6/21/2021 > git fetch --tags --progress https://github.com/larson-group/e3sm.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse refs/remotes/origin/stats_to_hosts^{commit} # timeout=10 Checking out Revision 89a5bb84d2469f135f7d600724cf0471a85f6cdc (refs/remotes/origin/stats_to_hosts) > git config core.sparsecheckout # timeout=10 > git checkout -f 89a5bb84d2469f135f7d600724cf0471a85f6cdc # timeout=10 Commit message: "Reflecting changes to nightly tests clubb_intr" [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Checkout Branch and Submodules) [Pipeline] sh + git submodule init + git submodule update [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Remove Old Output) [Pipeline] sh + rm -rf /home/jenkins/E3SM_simulations [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Run) [Pipeline] sh + csh nightly_test/nightly_e3sm_ne4_silhs.csh run_e3sm: ++++++++ run_e3sm starting (Mon Jul 26 12:14:35 CDT 2021), version 3.0.22 ++++++++ run_e3sm: $case_name = nelson.mg2_def.ne4_ne4 run_e3sm: -------- Starting create_newcase -------- run_e3sm: /home/jenkins/workspace/branch_e3sm_test//cime/scripts/create_newcase --case nelson.mg2_def.ne4_ne4 --compset FC5AV1C-L --script-root /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/case_scripts --res ne4_ne4 --pecount M --handle-preexisting-dirs u --mach nelson --output-root /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4 --project condo Compset longname is 2000_CAM5%AV1C-L_CLM45%SPBC_CICE%PRES_DOCN%DOM_SROF_SGLC_SWAV Compset specification file is /home/jenkins/workspace/branch_e3sm_test/components/cam/cime_config/config_compsets.xml Automatically adding SIAC to compset Automatically adding SESP to compset Compset forcing is Com forcing is present day: ATM component is cam5 physics:CAM with complete set of E3SM atmospheric mods for V1 (72 layers model) and ACES4BGC SOAG emissions- v04p2: LND component is clm4.5 physics:clm4.5 Satellite phenology:clm4.5 Satellite phenology with black carbon deposition: ICE component is prescribed cice: OCN component is DOCN prescribed ocean mode ROF component is Stub river component GLC component is Stub glacier (land ice) component WAV component is Stub wave component IAC component is Stub iac component ESP component is Stub external system processing (ESP) component Pes specification file is /home/jenkins/workspace/branch_e3sm_test/cime/config/e3sm/allactive/config_pesall.xml Machine is nelson Pes setting: grid is a%ne4np4_l%ne4np4_oi%ne4np4_r%null_g%null_w%null_z%null_m%oQU240 Pes setting: compset is 2000_CAM5%AV1C-L_CLM45%SPBC_CICE%PRES_DOCN%DOM_SROF_SGLC_SWAV_SIAC_SESP Pes setting: tasks is {'NTASKS_ATM': 16, 'NTASKS_ICE': 16, 'NTASKS_CPL': 16, 'NTASKS_LND': 16, 'NTASKS_WAV': 16, 'NTASKS_ROF': 16, 'NTASKS_OCN': 16, 'NTASKS_GLC': 16} Pes setting: threads is {'NTHRDS_ICE': 1, 'NTHRDS_ATM': 1, 'NTHRDS_ROF': 1, 'NTHRDS_LND': 1, 'NTHRDS_WAV': 1, 'NTHRDS_OCN': 1, 'NTHRDS_CPL': 1, 'NTHRDS_GLC': 1} Pes setting: rootpe is {'ROOTPE_OCN': 0, 'ROOTPE_LND': 0, 'ROOTPE_ATM': 0, 'ROOTPE_ICE': 0, 'ROOTPE_WAV': 0, 'ROOTPE_CPL': 0, 'ROOTPE_ROF': 0, 'ROOTPE_GLC': 0} Pes setting: pstrid is {} Pes other settings: {} Pes comments: none Compset is: 2000_CAM5%AV1C-L_CLM45%SPBC_CICE%PRES_DOCN%DOM_SROF_SGLC_SWAV_SIAC_SESP Grid is: a%ne4np4_l%ne4np4_oi%ne4np4_r%null_g%null_w%null_z%null_m%oQU240 Components in compset are: ['cam', 'clm', 'cice', 'docn', 'srof', 'sglc', 'swav', 'siac', 'sesp', 'drv'] No charge_account info available, using value from PROJECT e3sm model version found: 89a5bb8 Batch_system_type is none Creating Case directory /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/case_scripts run_e3sm: -------- Finished create_newcase -------- run_e3sm: Project used for submission: condo run_e3sm: Configuring E3SM to use the COSP simulator. run_e3sm: -------- Starting case.setup -------- run_e3sm: /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/case_scripts/case.setup /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/case_scripts/env_mach_specific.xml already exists, delete to replace /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/case_scripts/env_mach_specific.xml already exists, delete to replace Creating batch scripts Writing case.run script from input template /home/jenkins/workspace/branch_e3sm_test/cime/config/e3sm/machines/template.case.run Creating file .case.run Writing case.st_archive script from input template /home/jenkins/workspace/branch_e3sm_test/cime/config/e3sm/machines/template.st_archive Creating file case.st_archive Creating file .case.run.sh Creating user_nl_xxx files for components and cpl If an old case build already exists, might want to run 'case.build --clean' before building Generating component namelists as part of setup Creating component namelists 2021-07-26 12:14:44 atm Calling /home/jenkins/workspace/branch_e3sm_test/components/cam/cime_config/buildnml 2021-07-26 12:14:45 lnd Calling /home/jenkins/workspace/branch_e3sm_test/components/clm/cime_config/buildnml 2021-07-26 12:14:45 ice Calling /home/jenkins/workspace/branch_e3sm_test/components/cice/cime_config/buildnml 2021-07-26 12:14:45 ocn Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/data_comps/docn/cime_config/buildnml 2021-07-26 12:14:45 rof Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/srof/cime_config/buildnml 2021-07-26 12:14:45 glc Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sglc/cime_config/buildnml 2021-07-26 12:14:45 wav Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/swav/cime_config/buildnml 2021-07-26 12:14:45 iac Running /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/siac/cime_config/buildnml 2021-07-26 12:14:45 esp Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sesp/cime_config/buildnml 2021-07-26 12:14:45 cpl Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/drivers/mct/cime_config/buildnml Finished creating component namelists You can now run './preview_run' to get more info on how your case will be run run_e3sm: -------- Finished case.setup -------- run_e3sm: Creating logical links to make navigating easier. run_e3sm: Note: Beware of using ".." with the links, since the behavior of shell commands can vary. run_e3sm: /home/jenkins/workspace/branch_e3sm_test/nightly_test/nightly_e3sm_ne4_silhs.csh=a_run_link run_e3sm: run_root /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4 run_e3sm: run_dir /home/jenkins/workspace/branch_e3sm_test/nightly_test/nightly_e3sm_ne4_silhs.csh=a_run_link setting case file to env_run.xml run_e3sm: -------- Starting Build -------- run_e3sm: WARNING: Make sure there are no double-slashes (//) in the build command below: => Will cause a build error run_e3sm: /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/case_scripts/case.build Building case in directory /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/case_scripts sharedlib_only is False model_only is False Generating component namelists as part of build Creating component namelists 2021-07-26 12:14:46 atm Calling /home/jenkins/workspace/branch_e3sm_test/components/cam/cime_config/buildnml 2021-07-26 12:14:46 lnd Calling /home/jenkins/workspace/branch_e3sm_test/components/clm/cime_config/buildnml 2021-07-26 12:14:47 ice Calling /home/jenkins/workspace/branch_e3sm_test/components/cice/cime_config/buildnml 2021-07-26 12:14:47 ocn Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/data_comps/docn/cime_config/buildnml 2021-07-26 12:14:47 rof Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/srof/cime_config/buildnml 2021-07-26 12:14:47 glc Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sglc/cime_config/buildnml 2021-07-26 12:14:47 wav Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/swav/cime_config/buildnml 2021-07-26 12:14:47 iac Running /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/siac/cime_config/buildnml 2021-07-26 12:14:47 esp Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sesp/cime_config/buildnml 2021-07-26 12:14:47 cpl Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/drivers/mct/cime_config/buildnml Finished creating component namelists Building gptl with output to file /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/gptl.bldlog.210726-121446 Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/build_scripts/buildlib.gptl Component gptl build complete with 1 warnings Building mct with output to file /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/mct.bldlog.210726-121446 Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/build_scripts/buildlib.mct Building pio with output to file /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/pio.bldlog.210726-121446 Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/build_scripts/buildlib.pio Building csm_share with output to file /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/csm_share.bldlog.210726-121446 Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/build_scripts/buildlib.csm_share Component csm_share build complete with 4 warnings - Building atm Library Building atm with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/atm.bldlog.210726-121446 - Building lnd Library Building lnd with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/lnd.bldlog.210726-121446 - Building ice Library - Building ocn Library Building ice with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/ice.bldlog.210726-121446 Building ocn with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/ocn.bldlog.210726-121446 - Building rof Library Building rof with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/rof.bldlog.210726-121446 - Building glc Library Building glc with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/glc.bldlog.210726-121446 - Building wav Library Building wav with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/wav.bldlog.210726-121446 - Building iac Library - Building esp Library Building iac with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/iac.bldlog.210726-121446 Building esp with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/esp.bldlog.210726-121446 srof built in 0.509947 seconds siac built in 0.491729 seconds sesp built in 0.491163 seconds sglc built in 0.524430 seconds swav built in 0.545066 seconds docn built in 22.913966 seconds Component ice build complete with 9 warnings Component ice build complete with 9 warnings cice built in 22.924463 seconds Component atm build complete with 12 warnings Component lnd build complete with 6 warnings Component atm build complete with 12 warnings Component lnd build complete with 6 warnings cam built in 116.781763 seconds clm built in 116.781969 seconds Building e3sm with output to /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/e3sm.bldlog.210726-121446 Component e3sm exe build complete with 3 warnings Time spent not building: 1.103191 sec Time spent building: 165.688593 sec MODEL BUILD HAS FINISHED SUCCESSFULLY run_e3sm: -------- Finished Build -------- Creating component namelists 2021-07-26 12:17:33 atm Calling /home/jenkins/workspace/branch_e3sm_test/components/cam/cime_config/buildnml 2021-07-26 12:17:33 lnd Calling /home/jenkins/workspace/branch_e3sm_test/components/clm/cime_config/buildnml 2021-07-26 12:17:33 ice Calling /home/jenkins/workspace/branch_e3sm_test/components/cice/cime_config/buildnml 2021-07-26 12:17:33 ocn Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/data_comps/docn/cime_config/buildnml 2021-07-26 12:17:34 rof Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/srof/cime_config/buildnml 2021-07-26 12:17:34 glc Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sglc/cime_config/buildnml 2021-07-26 12:17:34 wav Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/swav/cime_config/buildnml 2021-07-26 12:17:34 iac Running /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/siac/cime_config/buildnml 2021-07-26 12:17:34 esp Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sesp/cime_config/buildnml 2021-07-26 12:17:34 cpl Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/drivers/mct/cime_config/buildnml Finished creating component namelists run_e3sm: WARNING: This script does not have batch directives for $machine=nelson run_e3sm: Assuming default E3SM values. run_e3sm: condo not recognized as a group, results must be shared manually run_e3sm: $model_start_type = initial (This is NOT necessarily related to RUN_TYPE) run_e3sm: -------- Starting Submission to Run Queue -------- run_e3sm: SUBMITTING JOB: run_e3sm: /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/case_scripts/case.submit Creating component namelists 2021-07-26 12:17:37 atm Calling /home/jenkins/workspace/branch_e3sm_test/components/cam/cime_config/buildnml 2021-07-26 12:17:37 lnd Calling /home/jenkins/workspace/branch_e3sm_test/components/clm/cime_config/buildnml 2021-07-26 12:17:37 ice Calling /home/jenkins/workspace/branch_e3sm_test/components/cice/cime_config/buildnml 2021-07-26 12:17:37 ocn Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/data_comps/docn/cime_config/buildnml 2021-07-26 12:17:37 rof Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/srof/cime_config/buildnml 2021-07-26 12:17:37 glc Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sglc/cime_config/buildnml 2021-07-26 12:17:37 wav Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/swav/cime_config/buildnml 2021-07-26 12:17:37 iac Running /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/siac/cime_config/buildnml 2021-07-26 12:17:37 esp Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sesp/cime_config/buildnml 2021-07-26 12:17:37 cpl Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/drivers/mct/cime_config/buildnml Finished creating component namelists Checking that inputdata is available as part of case submission Loading input file list: 'Buildconf/cpl.input_data_list' Loading input file list: 'Buildconf/docn.input_data_list' Loading input file list: 'Buildconf/clm.input_data_list' Loading input file list: 'Buildconf/cam.input_data_list' Loading input file list: 'Buildconf/cice.input_data_list' Check case OK submit_jobs case.run Submit job case.run Starting job script case.run 2021-07-26 12:17:38 CASE.RUN BEGINS HERE 2021-07-26 12:17:38 RUN_MODEL BEGINS HERE 2021-07-26 12:17:38 PRE_RUN_CHECK BEGINS HERE 2021-07-26 12:17:38 NAMELIST CREATION BEGINS HERE Generating namelists for /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/case_scripts Creating component namelists 2021-07-26 12:17:38 atm Calling /home/jenkins/workspace/branch_e3sm_test/components/cam/cime_config/buildnml 2021-07-26 12:17:38 lnd Calling /home/jenkins/workspace/branch_e3sm_test/components/clm/cime_config/buildnml 2021-07-26 12:17:38 ice Calling /home/jenkins/workspace/branch_e3sm_test/components/cice/cime_config/buildnml 2021-07-26 12:17:38 ocn Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/data_comps/docn/cime_config/buildnml 2021-07-26 12:17:38 rof Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/srof/cime_config/buildnml 2021-07-26 12:17:38 glc Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sglc/cime_config/buildnml 2021-07-26 12:17:38 wav Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/swav/cime_config/buildnml 2021-07-26 12:17:38 iac Running /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/siac/cime_config/buildnml 2021-07-26 12:17:38 esp Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/components/stub_comps/sesp/cime_config/buildnml 2021-07-26 12:17:38 cpl Calling /home/jenkins/workspace/branch_e3sm_test/cime/src/drivers/mct/cime_config/buildnml Finished creating component namelists 2021-07-26 12:17:39 NAMELIST CREATION HAS FINISHED ------------------------------------------------------------------------- - Prestage required restarts into /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/run - Case input data directory (DIN_LOC_ROOT) is /home/pub/cam_inputdata - Checking for required input datasets in DIN_LOC_ROOT ------------------------------------------------------------------------- 2021-07-26 12:17:39 PRE_RUN_CHECK HAS FINISHED run command is /usr/local/mpi/MPICH-3.3.1-gfortran/bin/mpirun -np 1 -prepend-rank /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/e3sm.exe >> e3sm.log.$LID 2>&1 2021-07-26 12:17:39 SAVE_PRERUN_PROVENANCE BEGINS HERE 2021-07-26 12:17:39 SAVE_PRERUN_PROVENANCE HAS FINISHED 2021-07-26 12:17:39 MODEL EXECUTION BEGINS HERE 2021-07-26 12:17:39 MODEL EXECUTION HAS FINISHED Exception from case_run: ERROR: RUN FAIL: Command '/usr/local/mpi/MPICH-3.3.1-gfortran/bin/mpirun -np 1 -prepend-rank /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/build/e3sm.exe >> e3sm.log.$LID 2>&1 ' failed See log file for details: /home/jenkins/E3SM_simulations/nelson.mg2_def.ne4_ne4/run/e3sm.log.210726-121738 Submitted job case.run with id None run_e3sm: -------- Finished Submission to Run Queue -------- run_e3sm: ++++++++ run_e3sm Completed (Mon Jul 26 12:17:39 CDT 2021) ++++++++ [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS