Regridding GEBCO bathymetry to curvilinear grid

[axelschweiger]

I'm trying to regrid the GEBCO bathymetry to a regional model grid for the northern hemisphere. I'm trying to do this like this:

/envs/pangeo310/bin/ESMF_RegridWeightGen -s 2024/GEBCO_2024.nc --dst_regional -d output.nc -w testWei
gthsScripts.nc -m bilinear

I'm getting lots of errors like this that fill my disk up very quickly:
20241121 134756.230 ERROR PET0 ESMCI_DistGrid.C:5101 ESMCI::DistGrid::getSequenceInde Invalid argument - SeqIndex type mismatch detected

The output .nc files looks like this. It is a curvelinear grid over the Arctic. The input file is a lat lon grid (pretty big 86400 by 43200)
dimensions:
x = 864 ;
y = 640 ;
variables:
float lat(y, x) ;
lat:_FillValue = 1.e+20f ;
lat:units = "degrees_north" ;
float lon(y, x) ;
lon:_FillValue = 1.e+20f ;
lon:units = "degrees_east" ;

Any advice?

I'm also wondering for tips on how to possibly speed this up running this as a regular python script using the xESMF tools. I can't quite figure out how to best take advantage of the multiple processors? Do I have to run this in a DASK cluster?

Thanks
Axel

[raphaeldussin]

Hi @axelschweiger

I found out that the regridding tools is particularly sensitive to netcdf metadata. Have you looked at this page?
https://xesmf.readthedocs.io/en/stable/large_problems_on_HPC.html

[aulemahal]

The regrid weights generation won't be faster with xESMF, sadly. If you have access to a lot of RAM and many cores, your best "performance" bet is running ESMF_RegridWeightGen in parallel with MPI. And then applying the weights with xESMF and dask.

xESMF still lacks a way to efficiently parallelize the regrid weights generation, the current one is a bit flaky and is only useful when going to higher resolution.

[axelschweiger]

Thanks. I checked all the meta-data. It was barking at me until I had it right. It now runs but the error message above is issued so many times to the log that it will overflow my disk. No idea if it is working properly or not. Wonder if it is possible to turn off the logging. The log file shows this but I supposed I would have to recompile the Fortran program to turn off the logging? Unless there is an environment variable to do so?

!!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20241121 134552.449 INFO PET0 !!! THIS MAY CAUSE SLOWDOWN IN PERFORMANCE !!!
20241121 134552.449 INFO PET0 !!! FOR PRODUCTION RUNS, USE: !!!
20241121 134552.449 INFO PET0 !!! ESMF_LOGKIND_Multi_On_Error !!!

[aulemahal]

In the technical doc of ESMF, in this section : https://earthsystemmodeling.org/docs/release/latest/ESMF_refdoc/node3.html#SECTION03026000000000000000
I see a --no_log option. Does it work ? If not, I suggest you open an issue on the esmf repo instead!

xESMF and ESMF are only loosely affiliated, so devs do not read these issues ( I think!) .