writing restart files on write grid comp

[junwang-noaa]

Description

It is found that when turning on the option to write restart files on the write grid comp in C768 coupled run, the writing is slow at this time as the ESMFproto_FieldBundleWrite is used (~90s) , the writing time is comparable to fms writing restart files when using sequential netcdf (~20s). Some code changes are required to use sequential netcdf on write grid component.

To Reproduce:

1. check out the latest ufs weather model develop branch and compile the code with FV3_GFS_v17_coupled_p8
2. run C768 coupled test (test case is available at: /lfs/h2/emc/ptmp/jun.wang/nco_benchmark/c768coupledDev)
3. in model_configure, turn on "quilting_restart"

Additional context

Output

[junwang-noaa]

@DusanJovic-NOAA Is this issue fixed?

[DusanJovic-NOAA]

@DusanJovic-NOAA Is this issue fixed?

I do not remember. What's the simplest way to test this again?

[junwang-noaa]

@DusanJovic-NOAA Can you try the HR3a test case on wcoss2 to see if there is slowness when the quilting_restart is turned on? Thanks

C768 on WCOSS2
/lfs/h2/emc/couple/noscrub/jessica.meixner/HR3/scoutruns/hr3scoutC768

[DusanJovic-NOAA]

@DusanJovic-NOAA Can you try the HR3a test case on wcoss2 to see if there is slowness when the quilting_restart is turned on? Thanks

C768 on WCOSS2 /lfs/h2/emc/couple/noscrub/jessica.meixner/HR3/scoutruns/hr3scoutC768

I do not see model run directory in this location, only logs and output grib files.

[junwang-noaa]

Sorry, the test run is at:

/lfs/h2/emc/couple/noscrub/jessica.meixner/HR3/scoutruns/t02C768/RUNDIRS/t02C768/fcst.55567

[DusanJovic-NOAA]

Is this on Cactus or Dogwood?

dusan.jovic@clogin03: ~
$ cd /lfs/h2/emc/couple/noscrub/jessica.meixner/HR3/scoutruns/t02C768/RUNDIRS/t02C768/fcst.55567
-bash: cd: /lfs/h2/emc/couple/noscrub/jessica.meixner/HR3/scoutruns/t02C768/RUNDIRS/t02C768/fcst.55567: No such file or directory

[junwang-noaa]

@JessicaMeixner-NOAA May I ask if you can provide a GFSv17 HR3 test case? Thanks

[JessicaMeixner-NOAA]

@GeorgeVandenberghe-NOAA has a canned case for the scout runs of HR3 running on various machines.

[GeorgeVandenberghe-NOAA]

This is accurate. The canned C768 case runs on wcoss2, hercules, orion and gaeaC5. There are some configuration issues on hercules and gaeaC5 which I am running down, in particular the ATM history write is not robust on gaea or hercules. The canned C1152 case runs on wcoss2, hercules and orion but has not been tried on GaeaC5. Spack-Stack has been rebuilt on GaeaC5 so I can build the ufs_model.x executable which now requires scotch.

…

On Tue, Jan 30, 2024 at 6:45 PM Jessica Meixner ***@***.***> wrote: @GeorgeVandenberghe-NOAA <https://github.com/GeorgeVandenberghe-NOAA> has a canned case for the scout runs of HR3 running on various machines. — Reply to this email directly, view it on GitHub <#1779 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FXKEV4C4QCB5RFAJJTYRE5T5AVCNFSM6AAAAAAYYUJW46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJXGY3TKMBSGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[junwang-noaa]

@GeorgeVandenberghe-NOAA can you provide the location of the run directory you have on wcoss2? Thanks

[GeorgeVandenberghe-NOAA]

It is on /lfs/h2/emc/ptmp/gwv/hr3/c768 on dogwood (and cactus)

…

On Wed, Jan 31, 2024 at 8:54 AM Jun Wang ***@***.***> wrote: @GeorgeVandenberghe-NOAA <https://github.com/GeorgeVandenberghe-NOAA> can you provide the location of the run directory you have on wcoss2? Thanks — Reply to this email directly, view it on GitHub <#1779 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FQZX4QNWIYSKN3SEGTYRJEGXAVCNFSM6AAAAAAYYUJW46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJZGE2DMNJWGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[junwang-noaa]

@DusanJovic-NOAA Do you have chance to test the HR3? Thanks

[DusanJovic-NOAA]

@DusanJovic-NOAA Do you have chance to test the HR3? Thanks

This commit should improve the timing of cubed sphere restart file writing:

NOAA-EMC/fv3atm@d59e80e

[junwang-noaa]

@DusanJovic-NOAA Just want to check if the code updates above are in the any of FV3ATM PRs? If not, can you add it to your no-warning PR ? Thanks

[DusanJovic-NOAA]

@DusanJovic-NOAA Just want to check if the code updates above are in the any of FV3ATM PRs? If not, can you add it to your no-warning PR ? Thanks

It's in this PR NOAA-EMC/fv3atm#775

[junwang-noaa]

@DusanJovic-NOAA The FV3 PR#775 was committed, can we close the issue?

[DusanJovic-NOAA]

@DusanJovic-NOAA The FV3 PR#775 was committed, can we close the issue?

Yes. Closing.