What's happening on the NASA Openscapes Hub!?

[erinmr]

Hello @NASA-Openscapes/longtermaccess-2i2c , @NASA-Openscapes/workshopaccess-2i2c and @NASA-Openscapes/championsaccess-2i2c -

The NASA Openscapes Team knows that the JupyterHub is important, but we don't know all the ways that it's being used or what kind of work it has enabled. Please use this discussion thread to Show and Tell! If you'd rather jump on a quick call you can email me (erin @ metadatagamechangers.com).

These stories help us to continue to make the case for supporting the Hub, so nothing is too minor

[cassienickles]

1. PO.DAAC has been extensively using the hub to develop cloud workflows! All of the cloud versions of our tutorials listed on this page have been developed using the Openscapes Hub!!
2. It was a space we gave our summer intern access to for her to develop in-cloud tutorials and update all existing PO.DAAC tutorials to use the earthaccess python library. It was great because the environment was already setup for her in the cloud and she didn't have to worry about package requirements, etc. It all worked out of the box in the cloud! Specific outcome: this jupyter notebook tutorial created by our intern, Zoë Walschots.
3. We also use the hub for the workshops we facilitate across DAACs, it's been invaluable for us for a collaboration tool.

[clhepp]

Jupyter Hub 2i2c is an essential tool to help transition to the cloud. It alleviates much of the cumbersome configuration otherwise required to work directly on AWS EC2 resources, including but not limited to file sharing, configuration control, workflow management. The infrastructure of AWS and EC2 for example is highly complex and finding the correct setup for a particular application is VERY difficult - until you know how! The J-hub and 2i2c is a major meeting point for learning - thank you! Please keep up the development of the cookbook and how-to's esp as applied to Julia and Docker workflows. Chris Hepplewhite UMBC/GESTAR-2

…

On Mon, Nov 6, 2023 at 7:21 PM Cassie Nickles ***@***.***> wrote: 1. PO.DAAC has been extensively using the hub to develop cloud workflows! All of the cloud versions of our tutorials listed on this page <https://podaac.github.io/tutorials/quarto_text/CloudvsLocalWorkflows.html> have been developed using the Openscapes Hub!! 2. It was a space we gave our summer intern access to for her to develop in-cloud tutorials and update all existing PO.DAAC tutorials to use the earthaccess python library. It was great because the environment was already setup for her in the cloud and she didn't have to worry about package requirements, etc. It all worked out of the box in the cloud! Specific outcome: this jupyter notebook tutorial <https://podaac.github.io/tutorials/notebooks/datasets/MUR_SST_Washington_Comparison.html> created by our intern, Zoë Walschots. 3. We also use the hub for the workshops we facilitate across DAACs, it's been invaluable for us for a collaboration tool. — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKF7OM2Z5LNEETM64TVSODYDF5HDAVCNFSM6AAAAAA7AG5Q7CVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TIOJTGI3TE> . You are receiving this because you are on a team that was mentioned.Message ID: <NASA-Openscapes/2i2cAccessPolicies/repo-discussions/2/comments/7493272@ github.com>

[battistowx]

The 2i2c and Openscapes-managed Jupyterhub environment continues to be an invaluable resource for the User Needs team at NASA GES DISC. It provides a controlled environment for our staff who are familiarizing themselves with Python and cloud computing, and allows them to experiment with scalable computing resources. All of our us-west-2 cloud tutorials have been created using this environment, and the Openscapes team has been very helpful for fixing any bugs or adding improvements and features, such as the R Studio environment. Our User Working Group has also been able to use this resource, pushing the applications of our data into new projects at scales that, not too long ago, would have required access to a supercomputer. It goes without saying that the hub is definitely being used, and is becoming more useful all the time.

Chris Battisto - NASA GES DISC

[michele-personal]

The 2i2c Hub was instrumental in allowing the ORNL DAAC to build and deliver content for a hands-on Short Course held at this summer's Ecological Society of America (ESA) Conference. A tutorial <https://github.com/michele-ornl/ESA_2023_ANG> built and delivered in the 2i2c Hub demonstrated Earthdata cloud-based methods to discover, access, and explore NASA Earthdata. The tutorial demonstrated access to airborne surface reflectance data from the AVIRIS-NG (Airborne Visible/Infrared Imaging Spectrometer-Next Generation) Instrument. AVIRIS-NG provides imaging spectroscopy measurements in 425 contiguous spectral channels with wavelengths in the solar reflected spectral range (380-2510 nm). A study area from the NASA * Surface Biology and Geology High-Frequency Time Series <https://climate.nasa.gov/news/3157/california-field-campaign-is-helping-scientists-protect-diverse-ecosystems/> *(SHIFT) Campaign was used to determine which AVIRIS-NG Facility Instrument data are available for that spatial extent. A second tutorial <https://github.com/rupesh2/ewt_cwc> built and delivered in the 2i2c Hub showed how to calculate equivalent water thickness or canopy water content (CWC) from AVIRIS-NG <https://daac.ornl.gov/cgi-bin/dataset_lister.pl?p=47> data. Variation in CWC can indicate drought stress and wildfire risk. A simple fitting of spectral absorption features of liquid water was applied and use scripts available from ISOFIT package <https://github.com/isofit/isofit/tree/main>. The 2i2c Hub was used during the combined ORNL / LP DAAC's User Working Group (UWG) meeting which allowed members Earthdata AWS access.

…

On Mon, Nov 6, 2023 at 5:17 PM Erin Robinson ***@***.***> wrote: Hello @NASA-Openscapes/longtermaccess-2i2c <https://github.com/orgs/NASA-Openscapes/teams/longtermaccess-2i2c> , @NASA-Openscapes/workshopaccess-2i2c <https://github.com/orgs/NASA-Openscapes/teams/workshopaccess-2i2c> and @NASA-Openscapes/championsaccess-2i2c <https://github.com/orgs/NASA-Openscapes/teams/championsaccess-2i2c> - The NASA Openscapes Team knows that the JupyterHub is important, but we don't know all the ways that it's being used or what kind of work it has enabled. Please use this discussion thread to Show and Tell! If you'd rather jump on a quick call you can email me (erin @ metadatagamechangers.com). These stories help us to continue to make the case for supporting the Hub, so nothing is too minor — Reply to this email directly, view it on GitHub <#2>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARX4UQHM572YA5F6V6FRZ2DYDFOXFAVCNFSM6AAAAAA7AG5Q7CVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZVHAZDANBWG4> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[BriannaLind]

LP DAAC uses the 2i2c instance in many ways:

1. DAAC scientists develop scripts, tutorials, and explore potential workflows for LP DAAC
2. DAAC scientists develop scripts, tutorials, and explore potential workflows for the Cloud cookbook and Cloud resources more broadly
3. DAAC scientist and Science Team collaborators run workshops (in person, virtually, hybrid) As of 2023 these include:

The EMIT Series
EMIT Earthdata Webinar
Ecological Society of America: Earth Data and Ecology in the Cloud joint LP-ORNL-EMIT Science Team
UWG demo/workshop
ECOSTRESS Science Team Meeting, Joint ECOSTRESS-EMIT
Space Station Synergies ECOSTRESS-EMIT (VITALS) AGU Workshop (coming soon)

4. DAAC scientists encourage workshop attendees to explore cloud workflows in their own research
5. DAAC scientists encourage collaborators to develop, test, and contribute back science scripts and workflows to centralized resource repositories to enable more open and transparent science.

@mjami00 @ebolch @cpkrehbiel @amfriesz anything you would like to add?

[m11keluis]

The Openscapes tutorials developed by the LP DAAC have been the foundation for EMIT applications interns. Led by EMIT applications lead (@kdchadwick), the EMIT interns adapted the LP DAAC notebooks to investigate drought and wildfire impacts, lahar risk, acid mine drainage, harmful algal blooms, and mineral detection. Several notebooks are being adapted to share with the broader community and we will share links here as soon as there ready! Big thanks LP DAAC!

[erinmr]

Thanks so much to @cassienickles, @BriannaLind, @clhepp, @battistowx and @michele-personal for adding to this thread. 💯 The specifics and links are awesome.

[eeholmes-uw]

The NASA Openscapes JupyterHub and the Openscapes Docker images were used at the template for the International Training Centre for Operational Oceanography (ITCOocean) JupyterHub (itcoocean.2i2c.cloud) in Hyderabad, India. The ITCOocean JupyterHub was used in the Hack2Week event in Sept 2023 at ITCOocean: https://hackweek-itcoocean.github.io/2023-Hackbook/. This was the kick-off event for a 5-year UN Decade of the Ocean capacity-building initiative for Early Career Ocean Professionals in the Indian Ocean Rim (https://incois.gov.in/dcc-ior/index.jsp). The 2-week event trained 58 ocean professionals from across India and Bangladesh in using NASA and other remote-sensing data in programmatic pipelines (Python and R) for research on the effect of ocean climate change on fisheries and marine mammals in the Arabian Sea and Bay of Bengal. Having the Openscapes JupyterHub as an example was essential for setting up the ITCOocean JupyterHub and we would not have been able to do our geospatial tutorials without using the Docker images developed by the NASA Openscapes team. The ITCOocean hackweek organizing team did not have the time or DevOps experience to debug and develop those on our own. -- Eli Holmes, NOAA Fisheries

[sevfour]

The NASA Openscapes JupyterHub was used to sample the reference altimetry missions on the ECCO model SSH daily outputs over more than 20 years. Working in the cloud was very practical and efficient as we were working with a lot of data that were stored on the cloud. A paper will eventually be written and the Openscapes JupyterHub acknowledged!
-- Severine Fournier, JPL

[inthisclimate]

Grateful for Openscapes to communicate with DAAC scientists on emerging applications and products, such as GEDI and EMIT! Working through the tutorials, it's amazing to have direct access to packages and imagery. More so, if I have a accessibility question, I can easily share my work with DAAC to help quickly find solutions.

[jules32]

Some work-in-progress from Dr. Lucas Barbedo @lucasbarbedo83 (Research associate at the University of Southern Mississippi, 2024 NASA Openscapes Champion) (posted with permission)

I could do some advance in process PACE image data using the Openscapes platform.
I know it is just a first step but I could plot, geo-reference, extract spectral signature of PACE L1B dataset.
My next step is to find a way to apply atmospheric correction to obtain water-leaving reflectance.
I am glad to see that we may run NASA/SEADAS libraries on kernel because OCSSW can do almost everything about ocean color processing.

[lucasbarbedo83]

My next step is to figure out how to process this L1B data to L2.
I want to use Seadas library: https://seadas.gsfc.nasa.gov/downloads/
However, first of all, I need to access the L1B file.

here, I show my search from earthaccess.search:

pprint(L1B_files[1])
pace  = xarray.open_dataset(L1B_files[0])
pprint(pace)

The output shows a address for this file: ob-cumulus-prod-public/PACE_OCI.20240415T172837.L1B.nc

<File-like object S3FileSystem, ob-cumulus-prod-public/PACE_OCI.20240415T172837.L1B.nc>
<xarray.Dataset> Size: 0B
Dimensions:  ()
Data variables:
    *empty*
Attributes: (12/33)
    title:                          PACE OCI Level-1B Data
    instrument:                     OCI
    product_name:                   PACE_OCI.20240415T172337.L1B.nc
    processing_version:             V1.0
    processing_level:               L1B
    cdm_data_type:                  swath
    ...                             ...
    geospatial_lon_min:             -86.69395
    geospatial_lon_max:             -55.257805
    geospatial_bounds_crs:          EPSG:4326
    history:                        2024-04-18T14:29:24Z: l1agen_oci OCI_temp...
    time_coverage_start:            2024-04-15T17:23:37.082
    time_coverage_end:              2024-04-15T17:28:36.965

I try to find this file using "ls" a terminal commnad:

!ls s3://obdaac-tea.earthdatacloud.nasa.gov/ob-cumulus-prod-public/PACE_OCI.20240415T172337.L1B.nc

however:
ls: cannot access 's3://obdaac-tea.earthdatacloud.nasa.gov/ob-cumulus-prod-public/PACE_OCI.20240415T172337.L1B.nc': No such file or directory

[itcarroll]

The s3:// prefix refers to the AWS S3 service. Like something prefixed with https://, its not in your filesystem and not visible to the ls command. Turns out, I have just verified, those s3:// paths are not able to be read by the ocssw tools right now. More in your post below ...

[lucasbarbedo83]

NASA/Seadas installation,
Following the manual installation of OCSSW binary: https://seadas.gsfc.nasa.gov/downloads/
!$HOME/ocssw/bin/l2gen
The terminal respond with the help of l2gen, which indicate a success to install Seadas on Kernel.

Usage: l2gen argument-list

  The argument-list is a set of keyword=value pairs. The arguments can
  be specified on the commandline, or put into a parameter file, or the
  two methods can be used together, with commandline over-riding.

  return value: 0=OK, 1=error, 110=north,south,east,west does not intersect
  file data.

The list of valid keywords follows: ...

> Question: should I install seadas on my $HOME directory, or Seadas routines could be installed on Kernel and be avaliable to everbody?

[lucasbarbedo83]

In case, Seadas OCSSW roitines are correctly installed.
I would like to run l2gen (as below) to obtain the remote sensing reflectance in the PACE L2 file(i.e., pace_oci.L2.nc Rrs)

!$HOME/ocssw/bin/l2gen  ifile='s3://ob-cumulus-prod-public/PACE_OCI.20240415T172837.L1B.nc' ofile='pace_oci.L2.nc' l2prod='Rrs'

Should it work?

[itcarroll]

It's not a bad idea to build an image (e.g. Docker) with JupyterLab and the SeaDAS processing tools (i.e. ocssw) that you could use on the JupyterHub. That would be fairly popular with other OC researchers, I bet. If you are interested, we can find resources for you. But it should also work to install the ocssw tools in your /home/jovyan space, although it might not be the most performant (or cost effective for Openscapes). But this space is all about just trying it out, so go for it.

Will l2gen ifile='s3:// .... work? Unfortunately, no. The way l2gen is written, it does not know what to do with "s3://" paths. What to do? We need to find out if there's way to make the s3 URL like a normal file. This is what earthaccess.open does in a Python environment. Alternatively, we need to wrap the l2gen function in a Python API. Not sure either of these are trivial, and which could happen first. I'm going to keep working on getting you a better answer.

[battistowx]

Hey everone!

I've been working on a fun weekend project using the hub, where I made a Python notebook that generates area-averaged Skew-T log-p charts for MERRA data using the Metpy and earthaccess libraries. I've always been curious to see if severe weather parameters can be calculated using MERRA data, and indeed, they can! Here's an example for the area over central OK, during the evening of the May 3rd, 1999 tornado:

Next, I'm considering creating long-term monthly averages over some areas to see how anomalous a particular event was in terms of severe parameters and seasonality. I love the convenience of the hub-I'm able to leverage S3 access from anywhere to make notebooks like these!