Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates related to component land model (Combined PR#2107) #1845

Merged
merged 103 commits into from
Jan 26, 2024

Conversation

uturuncoglu
Copy link
Collaborator

@uturuncoglu uturuncoglu commented Jul 28, 2023

Blocked By

Requires ESMF 8.5.0 install on RDHPCS/WCOSS2

Description

This PR aims to update component version of the land model to bring fully coupled atmosphere-land coupling capability to UFS Weather Model. The new development also allows to use different resolutions in atmosphere and land components C96 atmosphere and C384 land. The fully coupled configuration is only tested with global application at this point and requires additional work to make it work with regional applications like HAFS.

The initial evolution of the fully coupled atmosphere-land configuration can be found in the following presentation that was given in Noah-MP Workshop & Tutorial.

https://docs.google.com/presentation/d/1VFZwNisxwXUkeGW0iSrVJj8W0YaPTfllFVhbF42MMOE/edit#slide=id.g223ef07cc4d_0_0

The details about the content of the PR can be seen as following;

  • The FMS dependency is removed from NOAHMP-Interface and NOAHMP component model along with switching ESMF multi-tile I/O support (requires using ESMF >=8.5.0)

  • Following new RTs are introduced,

    • control_p8_atmlnd: fully coupled ATM-LND configuration
    • control_restart_p8_atmlnd: tests restart capability of fully coupled ATM-LND configuration
    • datm_cdeps_lnd_era5: component land model forced with ERA5
    • datm_cdeps_lnd_era5_rst: tests restart capability of component land model forced with ERA5
  • Following RT is removed,

    • datm_cdeps_lnd_gswp3_rst: This is replaced with datm_cdeps_lnd_era5_rst

Commit Message

  • Enable fully could atmosphere-land configuration
  • Supplementary physics updates (MYNN, GF, RUC_LSM, smoke and dust) for RRFS code freeze

Input data additions/changes

  • No changes are expected to input data.
  • Changes are expected to input data:
    • New input data.
    • Updated input data.

The new input data folders can be found in following directories:
Cheyenne
/glade/work/turuncu/RT/NEMSfv3gfs/input-data-20221101/NOAHMP_IC/
/glade/work/turuncu/RT/NEMSfv3gfs/input-data-20221101/DATM_ERA5_input_data_v2/
Orion/Hercules
/work/noaa/nems/tufuk/RT/NEMSfv3gfs/input-data-20221101/NOAHMP_IC
/work/noaa/nems/tufuk/RT/NEMSfv3gfs/input-data-20221101/DATM_ERA5_input_data_v2

Anticipated changes to regression tests:

  • No changes are expected to any regression test.
  • Changes are expected to the following tests:
    • Component version of NoahMP land model related RTs will have answer changes since the entire I/O layer of NoahMP driver is rewritten with ESMF multi-tile I/O support to get rid of FMS dependency to read and write multi-tile netcdf files.
    • Along with usage of latest public release of ESMF (8.5.0), there might be some answer changes due to the fixes in the ESMf interpolation routines. The detailed release notes can be seen in https://github.com/esmf-org/esmf/releases/tag/v8.5.0. The RT suite will be run against new version of ESMF to identify possible answer changes once new version available on Cheyenne and Orion (see ESMF requirement for external land component #1542).
    • The following from the combined PR:
   control_p8_mynn_intel regional_control_intel regional_decomp_intel regional_2threads_intel regional_noquilt_intel regional_netcdf_parallel_intel regional_2dwrtdecomp_intel regional_wofs_intel rap_control_intel regional_spp_sppt_shum_skeb_intel rap_decomp_intel rap_2threads_intel rap_sfcdiff_intel rap_sfcdiff_decomp_intel hrrr_control_intel hrrr_control_decomp_intel hrrr_control_2threads_intel rrfs_v1beta_intel rrfs_v1nssl_intel rrfs_v1nssl_nohailnoccn_intel regional_control_faster_intel regional_debug_intel rap_control_debug_intel hrrr_control_debug_intel hrrr_gf_debug_intel hrrr_c3_debug_intel rap_unified_drag_suite_debug_intel rap_diag_debug_intel rap_cires_ugwp_debug_intel rap_unified_ugwp_debug_intel rap_lndp_debug_intel rap_progcld_thompson_debug_intel rap_noah_debug_intel rap_sfcdiff_debug_intel rap_noah_sfcdiff_cires_ugwp_debug_intel rrfs_v1beta_debug_intel rap_clm_lake_debug_intel rap_flake_debug_intel gnv1_c96_no_nest_debug_intel regional_spp_sppt_shum_skeb_dyn32_phy32_intel rap_control_dyn32_phy32_intel hrrr_control_dyn32_phy32_intel rap_2threads_dyn32_phy32_intel hrrr_control_2threads_dyn32_phy32_intel hrrr_control_decomp_dyn32_phy32_intel conus13km_control_intel rap_control_dyn64_phy32_intel rap_control_debug_dyn32_phy32_intel hrrr_control_debug_dyn32_phy32_intel conus13km_debug_intel conus13km_debug_qr_intel conus13km_debug_2threads_intel conus13km_radar_tten_debug_intel rap_control_dyn64_phy32_debug_intel gnv1_nested_intel rap_control_gnu rap_decomp_gnu rap_2threads_gnu rap_sfcdiff_gnu rap_sfcdiff_decomp_gnu hrrr_control_gnu hrrr_control_noqr_gnu hrrr_control_2threads_gnu hrrr_control_decomp_gnu rrfs_v1beta_gnu regional_debug_gnu rap_control_debug_gnu hrrr_control_debug_gnu hrrr_gf_debug_gnu hrrr_c3_debug_gnu rap_diag_debug_gnu rap_noah_sfcdiff_cires_ugwp_debug_gnu rap_progcld_thompson_debug_gnu rrfs_v1beta_debug_gnu rap_flake_debug_gnu rap_clm_lake_debug_gnu gnv1_c96_no_nest_debug_gnu rap_control_dyn32_phy32_gnu hrrr_control_dyn32_phy32_gnu rap_2threads_dyn32_phy32_gnu hrrr_control_2threads_dyn32_phy32_gnu hrrr_control_decomp_dyn32_phy32_gnu conus13km_control_gnu rap_control_dyn64_phy32_gnu rap_control_debug_dyn32_phy32_gnu hrrr_control_debug_dyn32_phy32_gnu conus13km_debug_gnu conus13km_debug_qr_gnu conus13km_debug_2threads_gnu conus13km_radar_tten_debug_gnu rap_control_dyn64_phy32_debug_gnu

Subcomponents involved:

  • AQM
  • CDEPS
  • CICE
  • CMEPS
  • CMakeModules
  • FV3
  • GOCART
  • HYCOM
  • MOM6
  • NOAHMP
  • WW3
  • stochastic_physics
  • none

Library Updates/Changes

The ESMF library needs to be updated to its latest public release (8.5.0). This also requires updating PIO library to its recent version 2.5.10.

Combined with PR's (If Applicable): #2107

Commit Queue Checklist:

  • Link PR's from all sub-components involved in section below
  • Confirm reviews completed in ALL sub-component PR's
  • Add all appropriate labels to this PR.
  • Run full RT suite on either Hera/Cheyenne AND attach log to a PR comment.
  • Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

Combined PR:

Testing Day Checklist:

  • This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
  • Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

  • RDHPCS
    • Hera
    • Orion
    • Jet
    • Gaea
    • Cheyenne
  • WCOSS2
    • Dogwood/Cactus
    • Acorn
  • CI
    • Completed
  • opnReqTest
    • N/A
    • Log attached to comment
      Need help to run opnReqTest for atmosphere-land coupled configuration. The RT that tests restart capability for atmosphere-land was passing with ESMF 8.5.0b17.

The code is first tested with existing ESMF version (8.3.0b09) to check the modification related with the component land is not affecting existing applications. All the tests are passed on Orion (with Intel) except land related RTs which requires newer version of the ESMF library. The 8.3.0b09 is not tested on Cheyenne since the beeline was missing.

The code will be tested against ESMF 8.5.0 once it is available on Orion and Cheyenne to identify possible answer changes due to the version changes since there is no any answer change with ESMF 8.3.0b09.

@uturuncoglu
Copy link
Collaborator Author

@jkbk2004 @zach1221 I updated submodule pointers and hashes. If you don't mind could you confirm that that are fine.

@zach1221 zach1221 requested a review from jkbk2004 January 26, 2024 16:44
DeniseWorthen
DeniseWorthen previously approved these changes Jan 26, 2024
Copy link
Collaborator

@DeniseWorthen DeniseWorthen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@uturuncoglu Good work.

@jkbk2004
Copy link
Collaborator

@uturuncoglu can you check noah-mp hash? I think correct one is NOAA-EMC/noahmp@0cd3e23

@zach1221
Copy link
Collaborator

@DeniseWorthen I think there's still some conversations that are unresolved. Could you review those as well?

@uturuncoglu
Copy link
Collaborator Author

@jkbk2004 @zach1221 I was having last time check and I am getting permission error on Orion for input directory - /work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/DATM_ERA5_input_data_v2. If you don't mind could you double check?

@jkbk2004
Copy link
Collaborator

@jkbk2004 @zach1221 I was having last time check and I am getting permission error on Orion for input directory - /work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/DATM_ERA5_input_data_v2. If you don't mind could you double check?

looks ok

(base) jongkim@Orion-login-3:/work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/DATM_ERA5_input_data_v2$ ls
ERA5_forcing_2010-12-31_fix.nc  ERA5_forcing_2011-01-01_fix.nc  ERA5_forcing_2011-01-02_fix.nc  ERA5_mesh.nc

@jkbk2004
Copy link
Collaborator

orion and hercules ran ok as well

@uturuncoglu
Copy link
Collaborator Author

@jkbk2004 I am getting permission error because the permission is not correct for that folder. See following output,

drwxrwsr-x  2 jongkim epic 16384 Sep 15 11:56 DATM_ERA5_input_data
drwxr-s---  2 jongkim epic 16384 Jan 22 08:03 DATM_ERA5_input_data_v2

As you can see DATM_ERA5_input_data_v2 does not have right user permission.

@uturuncoglu
Copy link
Collaborator Author

@jkbk2004 This is also same for /work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/NOAHMP_IC/ folder (incl. its sub directories). I am not sure about other platforms.

@jkbk2004
Copy link
Collaborator

@jkbk2004 This is also same for /work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/NOAHMP_IC/ folder (incl. its sub directories). I am not sure about other platforms.

fixed

drwxrwsr-x  2 jongkim epic 16384 Jan 22 08:03 DATM_ERA5_input_data_v2

@uturuncoglu
Copy link
Collaborator Author

@jkbk2004 Thanks. I am also getting error from following folders,

/work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/NOAHMP_IC/CLMNCEP/
/work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/NOAHMP_IC/ERA5

If you don't mind could you also fix those. Thanks again for your help.

@jkbk2004
Copy link
Collaborator

@jkbk2004 Thanks. I am also getting error from following folders,

/work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/NOAHMP_IC/CLMNCEP/
/work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20221101/NOAHMP_IC/ERA5

If you don't mind could you also fix those. Thanks again for your help.

all good now: I checked on other machines as well.

drwxrwsr-x  2 jongkim epic  16384 Jan 22 08:02 CLMNCEP
drwxrwsr-x  2 jongkim epic  16384 Jan 22 08:02 ERA5

@uturuncoglu
Copy link
Collaborator Author

@jkbk2004 Thanks again.

@jkbk2004
Copy link
Collaborator

@jkbk2004 Thanks again.

@uturuncoglu Sure! Thanks a lot for doing this heavy lifting PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Baseline Updates Current baselines will be updated. New Input Data Req'd This PR requires new data to be sync across platforms Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked.
Projects
None yet
10 participants