Started by an SCM change
Obtained jenkins_tests/clubb_BFB_multi_col_gpu_nvhpc_test/Jenkinsfile from git <a href='https://github.com/larson-group/clubb.git'>https://github.com/larson-group/clubb.git</a>
<span class="pipeline-new-node" nodeId="2" startId="2">[Pipeline] Start of Pipeline
</span><span class="pipeline-new-node" nodeId="3" startId="3" enclosingId="2">[Pipeline] node
</span><span class="pipeline-node-3">Running on <a href='/jenkins/computer/nelson/' class='jenkins-table__link model-link model-link--float'>nelson</a> in /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test
</span><span class="pipeline-new-node" nodeId="4" startId="4" enclosingId="3">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="5" startId="5" enclosingId="4">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="6" startId="6" enclosingId="5" label="Declarative: Checkout SCM">[Pipeline] { (Declarative: Checkout SCM)
</span><span class="pipeline-new-node" nodeId="7" enclosingId="6">[Pipeline] checkout
</span><span class="pipeline-node-7">The recommended git tool is: git
Wiping out workspace first.
Cloning the remote Git repository
Cloning repository <a href='https://github.com/larson-group/clubb.git'>https://github.com/larson-group/clubb.git</a>
 &gt; git init /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test # timeout=10
Fetching upstream changes from <a href='https://github.com/larson-group/clubb.git'>https://github.com/larson-group/clubb.git</a>
 &gt; git --version # timeout=10
 &gt; git --version # 'git version 2.34.1'
using GIT_ASKPASS to set credentials larsongroupsysadmin github token
 &gt; git fetch --tags --force --progress -- <a href='https://github.com/larson-group/clubb.git'>https://github.com/larson-group/clubb.git</a> +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 77b73e25b7c54c85e9714178efbd6dc893834dc5 (refs/remotes/origin/master)
Commit message: "Making certain things differentiation safe, various parts had issues with sqrt or divisions that needed clipping for reverse mode calcualtions. Lot's of code sections seemed to already have worked around this with a safe_sqrt method (or some variation) that I standardized and put in advance_helper. (#1361)"
 &gt; git config remote.origin.url <a href='https://github.com/larson-group/clubb.git'>https://github.com/larson-group/clubb.git</a> # timeout=10
 &gt; git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 &gt; git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 &gt; git config core.sparsecheckout # timeout=10
 &gt; git checkout -f 77b73e25b7c54c85e9714178efbd6dc893834dc5 # timeout=10
 &gt; git rev-list --no-walk 17b29a7e938300692cec5c6c7e78aae7b9578568 # timeout=10
</span><span class="pipeline-new-node" nodeId="8" startId="6">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="9" startId="5">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="10" startId="10" enclosingId="4">[Pipeline] withEnv
</span><span class="pipeline-new-node" nodeId="11" startId="11" enclosingId="10">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="12" startId="12" enclosingId="11">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="13" startId="13" enclosingId="12" label="Update sys_admin Repo">[Pipeline] { (Update sys_admin Repo)
</span><span class="pipeline-new-node" nodeId="14" enclosingId="13">[Pipeline] sh
</span><span class="pipeline-node-14">+ git -C /home/pub/jenkins_BFB_test_configs/sys_admin pull
Already up to date.
</span><span class="pipeline-new-node" nodeId="15" startId="13">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="16" startId="12">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="17" startId="17" enclosingId="11">[Pipeline] stage
</span><span class="pipeline-new-node" nodeId="18" startId="18" enclosingId="17" label="Run Python Test">[Pipeline] { (Run Python Test)
</span><span class="pipeline-new-node" nodeId="19" startId="19" enclosingId="18">[Pipeline] lock
</span><span class="pipeline-node-19">Trying to acquire lock on [Resource: GPU_nelson]
The resource [GPU_nelson] is locked by build clubb_plot_nvhpc_gpu_vs_cpu #223 <a href='/jenkins/job/clubb_plot_nvhpc_gpu_vs_cpu/223/' class='jenkins-table__link model-link model-link--float'>#223</a> since Sep 19, 2026, 3:14 AM.
[Resource: GPU_nelson] is not free, waiting for execution ...
[Required resources: [GPU_nelson]] added into queue at position 0
Lock acquired on [Resource: GPU_nelson]
</span><span class="pipeline-new-node" nodeId="20" startId="20" enclosingId="19">[Pipeline] {
</span><span class="pipeline-new-node" nodeId="21" enclosingId="20">[Pipeline] sh
</span><span class="pipeline-node-21">+ ulimit -s 8388608
+ source /etc/profile.d/larson-group.sh
++ export GIT_EDITOR=vi
++ GIT_EDITOR=vi
++ export SVN_EDITOR=vi
++ SVN_EDITOR=vi
++ export OMP_STACKSIZE=1048579
++ OMP_STACKSIZE=1048579
++ export LMOD_ROOT=/opt/lmod/
++ LMOD_ROOT=/opt/lmod/
++ source /opt/lmod//lmod/lmod/init/bash
+++ '[' -z '' ']'
+++ case "$-" in
+++ __lmod_vx=x
+++ '[' -n x ']'
+++ set +x
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/local/spack/opt/spack/linux-pop22-skylake_avx512/gcc-12.2.0/lmod-8.7.37-qorv6nkkjzyzntx6wbjupmoxrot26t6r/lmod/lmod/init/bash)
Shell debugging restarted
+++ unset __lmod_vx
+++ find /usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core -print -quit
++ export MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core
++ MODULEPATH=/usr/local/spack/share/spack/lmod/linux-pop22-x86_64/Core
+ module load nvhpc netcdf-fortran
+ '[' -z '' ']'
+ case "$-" in
+ __lmod_sh_dbg=x
+ '[' -n x ']'
+ set +x
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output
Shell debugging restarted
+ unset __lmod_sh_dbg
+ return 0
+ python3 /home/pub/jenkins_BFB_test_configs/sys_admin/python_nightly_test_suite/run_test.py -c /home/pub/jenkins_BFB_test_configs/clubb_BFB_multi_col_gpu_nvhpc_test_config
CMake Warning:
  Manually-specified variables were not used by the project:

    NetCDF_C_INCLUDE_DIR
    NetCDF_C_LIBRARY
    NetCDF_Fortran_INCLUDE_DIR
    NetCDF_Fortran_LIBRARY


Directory 1 is 77b73e25b7c54c85e9714178efbd6dc893834dc5
Directory 2 is 17b29a7e938300692cec5c6c7e78aae7b9578568
Using reporting threshold:  0.0 


The following cases will be compared: ['arm', 'cobra', 'gabls3', 'mpace_b', 'rico_silhs']


================================================ arm ================================================
&gt;No differences detected by the linux diff in arm_stats.nc&lt;
&gt;&gt;&gt;The linux diff could not detect any differences in the file pairs for case arm.&lt;&lt;&lt;

=====================================================================================================

================================================ cobra ================================================
&gt;No differences detected by the linux diff in cobra_stats.nc&lt;
&gt;&gt;&gt;The linux diff could not detect any differences in the file pairs for case cobra.&lt;&lt;&lt;

=====================================================================================================

================================================ gabls3 ================================================
&gt;No differences detected by the linux diff in gabls3_stats.nc&lt;
&gt;&gt;&gt;The linux diff could not detect any differences in the file pairs for case gabls3.&lt;&lt;&lt;

=====================================================================================================

================================================ mpace_b ================================================
&gt;No differences detected by the linux diff in mpace_b_stats.nc&lt;
&gt;&gt;&gt;The linux diff could not detect any differences in the file pairs for case mpace_b.&lt;&lt;&lt;

=====================================================================================================

================================================ rico_silhs ================================================
&gt;No differences detected by the linux diff in rico_silhs_stats.nc&lt;
&gt;&gt;&gt;The linux diff could not detect any differences in the file pairs for case rico_silhs.&lt;&lt;&lt;

=====================================================================================================

SUMMARY:
Linux diff did not detect any differences in the compared files.
Pass: 5 cases ['arm', 'cobra', 'gabls3', 'mpace_b', 'rico_silhs']
Fail: 0 cases []

=============================== Configuring Test ===============================


Working directory set to: /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test
Checking specified repo: /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test

Using /home/jenkins/workspace/clubb_BFB_multi_col_gpu_nvhpc_test with origin <a href='https://github.com/larson-group/clubb.git'>https://github.com/larson-group/clubb.git</a>

Output Save Mode: all

Using branch: master

Making list from 'UnresolvedCommits':

CheckRange = True: Getting commit range

Run Output Path: output

Run commands: ['python compile.py -gpu openacc -debug', '\nrm -rf output', '\nmkdir output', '\npython run_scripts/run_scm.py -multicol 4 arm', '\npython run_scripts/run_scm.py -override l_lh_straight_mc=true -multicol 4 rico_silhs', '\npython run_scripts/run_scm.py -multicol 4 cobra', '\npython run_scripts/run_scm.py -multicol 4 mpace_b', '\npython run_scripts/run_scm.py -multicol 4 gabls3']

Diffing command: run_scripts/run_bindiff_all.py

=============================== Running Test ===============================

Parsing message from commit: af35e5202409ab58c575e8d2841f59d31ba80c94
Parsing message from commit: 5b5bf5f74f30e0332db965215e3de5e6baf72d6e
Parsing message from commit: 77b73e25b7c54c85e9714178efbd6dc893834dc5

Checking Commit: af35e5202409ab58c575e8d2841f59d31ba80c94
	Skipping: not final commit and not labelled BIT_CHANGING

Checking Commit: 5b5bf5f74f30e0332db965215e3de5e6baf72d6e
	Skipping: not final commit and not labelled BIT_CHANGING

Checking Commit: 77b73e25b7c54c85e9714178efbd6dc893834dc5
	Running: final commit to check, not labelled BIT_CHANGING
	Checking out 77b73e25b7c54c85e9714178efbd6dc893834dc5
	Running: 'python compile.py -gpu openacc -debug'
	Running: 'rm -rf output'
	Running: 'mkdir output'
	Running: 'python run_scripts/run_scm.py -multicol 4 arm'
	Running: 'python run_scripts/run_scm.py -override l_lh_straight_mc=true -multicol 4 rico_silhs'
	Running: 'python run_scripts/run_scm.py -multicol 4 cobra'
	Running: 'python run_scripts/run_scm.py -multicol 4 mpace_b'
	Running: 'python run_scripts/run_scm.py -multicol 4 gabls3'
	Checking out 17b29a7e938300692cec5c6c7e78aae7b9578568
	Running: 'python compile.py -gpu openacc -debug'
	Running: 'rm -rf output'
	Running: 'mkdir output'
	Running: 'python run_scripts/run_scm.py -multicol 4 arm'
	Running: 'python run_scripts/run_scm.py -override l_lh_straight_mc=true -multicol 4 rico_silhs'
	Running: 'python run_scripts/run_scm.py -multicol 4 cobra'
	Running: 'python run_scripts/run_scm.py -multicol 4 mpace_b'
	Running: 'python run_scripts/run_scm.py -multicol 4 gabls3'
	Diffing: 'run_scripts/run_bindiff_all.py 77b73e25b7c54c85e9714178efbd6dc893834dc5 17b29a7e938300692cec5c6c7e78aae7b9578568'

=============================== Test Results ===============================

Commit Range Checked = 17b29a7e938300692cec5c6c7e78aae7b9578568...77b73e25b7c54c85e9714178efbd6dc893834dc5

No new Unresolved commits found.

No fake BIT_CHANGING commits found.

No remaining Unresolved Commits.

=============================== Updating Config File ===============================


Saving into UnresolvedCommits: 

Saving into BaselineCommit: 77b73e25b7c54c85e9714178efbd6dc893834dc5

=============================== Cleanup ===============================

Not removing any generated output.
	RENAMING 77b73e25b7c54c85e9714178efbd6dc893834dc5 to latest_revision_validated_77b73e25b7c54c85e9714178efbd6dc893834dc5

================== Unresolved Commit Status ==================

No remaining Unresolved Commits.

Exiting with code: 0
</span><span class="pipeline-new-node" nodeId="22" startId="20">[Pipeline] }
</span><span class="pipeline-node-22">Lock released on resource [Resource: GPU_nelson]
</span><span class="pipeline-new-node" nodeId="23" startId="19">[Pipeline] // lock
</span><span class="pipeline-new-node" nodeId="24" startId="18">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="25" startId="17">[Pipeline] // stage
</span><span class="pipeline-new-node" nodeId="26" startId="11">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="27" startId="10">[Pipeline] // withEnv
</span><span class="pipeline-new-node" nodeId="28" startId="4">[Pipeline] }
</span><span class="pipeline-new-node" nodeId="29" startId="3">[Pipeline] // node
</span><span class="pipeline-new-node" nodeId="30" startId="2">[Pipeline] End of Pipeline
</span>Finished: SUCCESS