Skip to content
Success

Changes

Summary

  1. Tuner additions (#1324) (details)
Commit 95d5d12aef49035170773f0714a97f4109bcccdd by noreply
Tuner additions (#1324)

* Starting new additions for tuner - specifically an overall history of all samples and loss metrics with the eventual goal of having fancy SVD and spline things we can do with that

* Adding plots to tune tab to visualize data in the overall sample outputs

* Adding the enhanced simulated annealed to the tuner. Only real tweak it that it runs workers x batch_size chains independently so that we can utilize the CPU fully

* Filtering big values in parameter plot view and making default window length 3600s (1 hour)

* Bug fix to metric report - a 0 correlation was able to throw things off, which could occur when a field from either result (clubb or LES) is all 0 without the other one being all zero. Also some small display improvements

* Adding sensitivity plots

* Updating flag/param selecting method by created a general input/tunable_parameters directory that contains subdirectories for different flag set combos. These can be used by passing in their name to run_scm.py with -config. Also updating references to these files since they moved and also making tests and scripts default to using -override instead of editting files with sed.

* Adding buttons to select param/flag config in the tuner

* Quality of life improvements for plot tab - now view settings and UI settings are retained when reloading the same case

* Fixing missed comment change

* Various small UI improvements

* Improving tuner history check and little UI fix in run_tab

* Fixing tuner environment setup

* Fixing little netcdf version issue and adding new test for loss metrics consistency check

* Renaming tunable_parameters to parameter_and_flag_configs, also adding -config example to run_scripts README
The file was removedinput/tunable_parameters_Lscale/configurable_model_flags.in
The file was addedinput/parameter_and_flag_configs/e3sm_maint32/tunable_parameters.in
The file was modifiedutilities/loss_metrics.py (diff)
The file was modifieddash_app/assets/11_tab_run_theme.css (diff)
The file was modifiedjenkins_tests/clubb_nvhpc_gpu_vs_cpu_diffness/Jenkinsfile (diff)
The file was modifieddash_app/plot_tab/plot_types/subcolumn_plot.py (diff)
The file was modifieddash_app/run_tab/tab.py (diff)
The file was removedinput/tunable_parameters_compatible_r8029/configurable_model_flags.in
The file was modifieddash_app/tune_tab/callbacks_display.py (diff)
The file was modifiedrun_scripts/run_scm.py (diff)
The file was modifieddash_app/tune_tab/discovery.py (diff)
The file was modifiedCMakeLists.txt (diff)
The file was addedinput/parameter_and_flag_configs/Lscale/configurable_model_flags.in
The file was addedtuner/enhanced_simann_strategy.py
The file was modifiedtests/run_thread_test.py (diff)
The file was modifieddash_app/tune_tab/state.py (diff)
The file was modifiedinput/case_setups/README (diff)
The file was removedinput/tunable_parameters_compatible_r8029/tunable_parameters.in
The file was modifiedtests/run_loss_output_consistency.py (diff)
The file was removedrun_scripts/README
The file was modifiedtests/run_stats_output_consistency.py (diff)
The file was modifiedrun_scripts/run_tuner.py (diff)
The file was removedinput/tunable_parameters_coriolis_cases/ekman_parameters.in
The file was modifiedtuner/paths.py (diff)
The file was modifiedinput/case_setups/atex_long_model.in (diff)
The file was modifiedsrc/generalized_grid_test.F90 (diff)
The file was addedinput/parameter_and_flag_configs/compatible_r8029/configurable_model_flags.in
The file was removedinput/tunable_parameters_compatible_r8029/silhs_parameters.in
The file was modifiedtuner/request.py (diff)
The file was modifiedtuner/job_runtime.py (diff)
The file was removedinput/tunable_parameters_coriolis_cases/coriolis_test_parameters.in
The file was addedinput/parameter_and_flag_configs/default/silhs_parameters.in
The file was modifieddash_app/run_tab/callbacks_settings.py (diff)
The file was modifieddash_app/run_tab/state.py (diff)
The file was removedinput/tunable_parameters/configurable_model_flags.in
The file was removedinput/tunable_parameters/tunable_parameters.in
The file was modifiedcompile.py (diff)
The file was addedinput/parameter_and_flag_configs/e3sm_maint32/maint32_defaults_report.md
The file was addedinput/parameter_and_flag_configs/tauMatchLscale/configurable_model_flags.in
The file was modifiedutilities/create_multi_col_params.py (diff)
The file was modifieddash_app/plot_tab/plot_types/timeheight_plot.py (diff)
The file was addedinput/parameter_and_flag_configs/coriolis_cases/atex_long_parameters.in
The file was addedinput/parameter_and_flag_configs/e3sm/configurable_model_flags.in
The file was modifieddash_app/run_tab/namelist.py (diff)
The file was modifieddash_app/plot_tab/plot_types/profile_plot.py (diff)
The file was addedinput/parameter_and_flag_configs/e3sm/tunable_parameters.in
The file was addedtuner/sample_history.py
The file was addedinput/parameter_and_flag_configs/coriolis_cases/ekman_parameters.in
The file was modifieddash_app/plot_tab/plot_types/timeseries_plot.py (diff)
The file was removedinput/tunable_parameters_e3sm/configurable_model_flags.in
The file was modifieddash_app/assets/14_tab_compile_theme.css (diff)
The file was modifiedjenkins_tests/clubb_rev_grid_matrix_solve_test/Jenkinsfile (diff)
The file was addedtuner/system_defaults.py
The file was modifiedutilities/benchmark_converter.py (diff)
The file was addedinput/parameter_and_flag_configs/default/tunable_parameters.in
The file was addedinput/parameter_and_flag_configs/compatible_r8029/silhs_parameters.in
The file was modifiedens_tune/mytuner_movingfoot.bash (diff)
The file was modifieddash_app/tune_tab/runtime.py (diff)
The file was addedinput/parameter_and_flag_configs/coriolis_cases/coriolis_test_parameters.in
The file was modifiedjenkins_tests/clubb_backwards_compatibility_gfortran_test/Jenkinsfile (diff)
The file was modifiedcmake/deps/GPTLDependency.cmake (diff)
The file was modified.gitignore (diff)
The file was modifieddash_app/run_tab/layout.py (diff)
The file was modifieddash_app/assets/13_tab_tune_theme.css (diff)
The file was addedrun_scripts/README.md
The file was modifieddash_app/compile_tab/layout.py (diff)
The file was modifiedREADME (diff)
The file was modifiedtuner/test_tuning_strategy.py (diff)
The file was modifiedrun_scripts/run_tuner_job.py (diff)
The file was modifieddash_app/assets/12_tab_plots_theme.css (diff)
The file was modifiedrun_scripts/convergence_run/convergence_config.py (diff)
The file was modifiedjenkins_tests/clubb_nvhpc_gpu_column_mirror_test/Jenkinsfile (diff)
The file was modifieddash_app/tune_tab/tab.py (diff)
The file was modifieddash_app/tune_tab/callbacks_settings.py (diff)
The file was modifiedsrc/clubb_loss_driver_test.F90 (diff)
The file was modifiedinput/case_setups/coriolis_test_model.in (diff)
The file was removedinput/tunable_parameters/silhs_parameters.in
The file was modifiedsrc/clubb_loss_driver.F90 (diff)
The file was modifieddash_app/plot_tab/layout.py (diff)
The file was modifiedrun_scripts/run_scm_loss.py (diff)
The file was modifieddash_app/plot_tab/callbacks_controls.py (diff)
The file was modifiedjenkins_tests/clubb_new_tuner_test/Jenkinsfile (diff)
The file was modifieddash_app/plot_tab/state.py (diff)
The file was removedinput/tunable_parameters_e3sm/tunable_parameters.in
The file was removedinput/tunable_parameters_Lscale/tunable_parameters.in
The file was modifieddash_app/assets/10_theme.css (diff)
The file was modifieddash_app/tune_tab/callbacks_runs.py (diff)
The file was addeddash_app/run_tab/config_state.py
The file was modifieddash_app/run_tab/callbacks_runs.py (diff)
The file was modifieddash_app/plot_tab/callbacks_case.py (diff)
The file was modifieddash_app/plot_tab/plot_types/budget_plot.py (diff)
The file was removedinput/tunable_parameters_tauMatchLscale/configurable_model_flags.in
The file was modifieddash_app/tune_tab/layout.py (diff)
The file was modifiedtuner/tuning_strategy.py (diff)
The file was modifiedtests/test_monoflux_limiter_GPU.py (diff)
The file was addeddash_app/tunable_configs.py
The file was modifiedpostprocessing/output_scripts/cloud_feedback/README (diff)
The file was addedinput/stats/all_tuning_stats.in
The file was modifieddash_app/compile_tab/callbacks.py (diff)
The file was modifiedsrc/clubb_tuner.F90 (diff)
The file was modifieddash_app/run_tab/runtime.py (diff)
The file was addedinput/parameter_and_flag_configs/Lscale/tunable_parameters.in
The file was modifiedtuner/case_defaults.json (diff)
The file was addedinput/parameter_and_flag_configs/e3sm_maint32/configurable_model_flags.in
The file was addedinput/parameter_and_flag_configs/tauMatchLscale/tunable_parameters.in
The file was modifiedtests/run_benchmark_converter_test.py (diff)
The file was modifieddash_app/plot_tab/plot_types/shared.py (diff)
The file was modifiedtests/README (diff)
The file was modifiedjenkins_tests/clubb_driver_test/Jenkinsfile (diff)
The file was modifiedtuner/tuning_scheduler.py (diff)
The file was modifiedtuner/tuning_worker.py (diff)
The file was modifiedutilities/create_case_namelist.py (diff)
The file was removedinput/tunable_parameters_tauMatchLscale/tunable_parameters.in
The file was addedtests/analyze_tuner_sample_history.py
The file was modifiedinput/case_setups/ekman_model.in (diff)
The file was modifieddash_app/plot_tab/callbacks_params.py (diff)
The file was removedinput/tunable_parameters_coriolis_cases/atex_long_parameters.in
The file was addedinput/parameter_and_flag_configs/compatible_r8029/tunable_parameters.in
The file was addedtests/run_loss_metrics_test.py
The file was modifiedens_tune/mytuner_sequential.bash (diff)
The file was addedinput/parameter_and_flag_configs/e3sm_maint32/silhs_parameters.in
The file was addedinput/parameter_and_flag_configs/default/configurable_model_flags.in
The file was modifiedjenkins_tests/clubb_plot_nvhpc_gpu_vs_cpu/Jenkinsfile (diff)
The file was modifiedtuner/status.py (diff)
The file was modifiedjenkins_tests/clubb_plot_solver_compare/Jenkinsfile (diff)