-
Notifications
You must be signed in to change notification settings - Fork 158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use MPI_Bcast instead of multiple p2p messages to update nest from parent #743
Merged
jkbk2004
merged 20 commits into
NOAA-EMC:develop
from
dkokron:hafs-BcastFillNestedGridCpl
Jan 22, 2024
Merged
Use MPI_Bcast instead of multiple p2p messages to update nest from parent #743
jkbk2004
merged 20 commits into
NOAA-EMC:develop
from
dkokron:hafs-BcastFillNestedGridCpl
Jan 22, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pl by a single MPI_Bcast
… structures associated with Bcast_comm
…epends on NOAA-GFDL/FMS#1246 to be functional. More efficient 'if' test in fill_nested_grid_cpl()
…fficient determination of ranks involved in the mpp_broadcast call in fill_nested_grid_cpl routine
Waiting on ACS approval and hash update and .gitmodules revert. |
jkbk2004
approved these changes
Jan 22, 2024
BrianCurtis-NOAA
approved these changes
Jan 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Performance profiling of a HAFS case on NOAA systems revealed significant of time was spent in fill_nested_grid_cpl(). The fill_nested_grid_cpl() routine from FV3/atmos_cubed_sphere/driver/fvGFS/atmosphere.F90 is showing up as a performance bottleneck. This routine gathers a global SST field (6,336,000 bytes) onto rank 0, then broadcasts that field to all ranks in the nest. The code uses point-to-point (p2p) messages (Isend/Recv) from rank 0 to the nest ranks. This communication pattern is maxing out the SlingShot-10 link on the first node resulting in a .15s hit every fifth time step.
The proposed fix is to modify the relevant FV3 code to use a single MPI_Bcast (via mpp_broadcast()) instead of multiple point-to-point messages. The use of mpp_broadcast depends on a fix to FMS that was merged on 16 June and is available in version 2023.02 of that package.
NOAA-GFDL/FMS#1246
This PR depends on merging of PR 272 into the GFDL_atmos_cubed_sphere
NOAA-GFDL/GFDL_atmos_cubed_sphere#272
I ran the UFS regression suite on acorn and cactus. Both runs resulted in "REGRESSION TEST WAS SUCCESSFUL"
This change is zero-diff. No need to update baselines