Add possibility to use pre-built shared libs during case.build

[bartgol]

The need for prebuilt libs stems from a couple of needs:

1. Large nightly testing might shave quite a few minutes if using pre-built shared libs. Yes, the testsuites can share the share libs, but up to a certain extent.
2. We would like to link a TPL to the model. This TPL should be prebuilt (very expensive to build, and/or very complicated configuration), and could benefit from using, say, pio or gptl. We would like to use the same version of pio/gptl that e3sm would link against.

Here's how I envision this to be implemented:

• Add an XML var USE_PREBUILT_LIBS (or something like that), with valid values none, all, or a comma-separated list of libs (e.g, USE_PREBUILT_LIBS=pio,gptl).
• In case.build, check this var, and only build the libs that are not to be used as prebuilt.
• In config_machines, set vars like ${LIBNAME}_ROOT. Then, case.build will add $compiler/$buildtype to get the full installation path (where buildtype is either debug or release). If we feel brave, we could also add some more buildtypes, based on other XML config options (I'm thinking about stuff like certain PIO compile-time options).

Additional comments:

• create_test could default to USE_PREBUILT_LIBS="none", for bwd compatibility. Or we could have config_machines store a default for this variable, so that e3sm/cesm can choose different approaches, or we could have different approaches on different machines. E.g., on new machines we start with "none", and if the maintainer finds time, they can install prebuilt libs.
• create_test could have a flag, like --use-prebuilt-libs=a,b,c to force using prebuilt libs for a,b,c, if the machine default is to build them on the fly. A symmetric option could be --build-shared=d,e, in case the machine default is do use pre-built. Obviously, using both at the same time is not allowed.
• Compilers change, so we could store a compiler version sha along the installation. When case.build runs, it can check the current compiler version against the stored sha, and, if they don't match, build the lib on the fly, maybe printing a big warning "WARNING, installation for lib X, compiler Y, build type Z is expired. Contact machine maintainer".
• I argue we don't need to have 1 installation for each flag/options configuration. If the user wants fine grain control over shared libs flags, they can have the libs built on the fly. Prebuilt libs are for users that are ok with some sort of default settings.

[jgfouca]

@jedwards4b , thoughts?

[jedwards4b]

We are using prebuilt pio libraries on cheyenne, it uses this code in the buildlib.pio:

installed_pio_version = os.environ.get("PIO_VERSION_MAJOR")
    logger.info(
        "pio_version_major = {} pio_version = {}".format(
            installed_pio_version, pio_version
        )
    )
    if installed_pio_version is not None and int(installed_pio_version) == pio_version:
        logger.info("Using installed PIO library")
        _set_pio_valid_values(case, os.environ.get("PIO_TYPENAME_VALID_VALUES"))
        return

this relies on the pio module setting the two env variables PIO_VERSION_MAJOR and PIO_TYPENAME_VALID_VALUES

[gold2718]

For components, how about something like NTASKS where compclass is used to set values (or flags in this instance) for each component.

[jedwards4b]

I think that the use of modules can simplify a lot of what @bartgol is suggesting.

[bartgol]

So if we prebuild pio, and have those two env vars defined, case.build will skip PIO build? That would work, I suppose.

[jedwards4b]

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   /glade/u/apps/ch/modulefiles/default/mpt/2.25/intel/19.1.1/pio/2.5.9.lua:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("pio v2.5.9")
help([[A high-level Parallel I/O Library for structured grid applications

Software website - https://ncar.github.io/ParallelIO/

Built on Mon Sep 19 15:24:24 MDT 2022
Modules used:
   ncarenv/1.3          
   mpt/2.25          
   pnetcdf/1.12.3
   intel/19.1.1         
   cmake/3.18.2
   ncarcompilers/0.5.0  
   netcdf-mpi/4.9.0
]])
prereq(atleast("netcdf-mpi","4.7.3"),atleast("pnetcdf","1.11.0"))
setenv("NCAR_ROOT_PIO","/glade/u/apps/ch/opt/pio/2.5.9/mpt/2.25/intel/19.1.1/")
setenv("NCAR_INC_PIO","/glade/u/apps/ch/opt/pio/2.5.9/mpt/2.25/intel/19.1.1/include")
setenv("NCAR_LDFLAGS_PIO","/glade/u/apps/ch/opt/pio/2.5.9/mpt/2.25/intel/19.1.1/lib")
setenv("PIO","/glade/u/apps/ch/opt/pio/2.5.9/mpt/2.25/intel/19.1.1/")
setenv("PIO_INCDIR","/glade/u/apps/ch/opt/pio/2.5.9/mpt/2.25/intel/19.1.1/include")
setenv("PIO_LIBDIR","/glade/u/apps/ch/opt/pio/2.5.9/mpt/2.25/intel/19.1.1/lib")
setenv("PIO_VERSION_MAJOR","2")
setenv("PIO_TYPENAME_VALID_VALUES","netcdf,pnetcdf,netcdf4p,netcdf4c")

[rljacob]

Doesn't creating a module require a sysadmin to do something? We can get that done on machines we control but not sure about NERSC, OLCF, ALCF.

[rljacob]

How do you link to pre-built ESMF? Module?

[bartgol]

Doesn't creating a module require a sysadmin to do something? We can get that done on machines we control but not sure about NERSC, OLCF, ALCF.

@rljacob We can prob just add a bunch of env vars in config machines, to point to the installation and declare the PIO_XYZ needed by cime...

[jedwards4b]

Yes - esmf is built as a module.
Users can install modules in their own space or a user can install and share, doesn't need root access - as spack becomes more widely used I think we will be seeing more of this.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[jedwards4b]

The request in this issue is largely already implemented in the build system.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.