Skip to content

Commit

Permalink
Added end-to-end flow for NGIAB 1.3.0 (#238)
Browse files Browse the repository at this point in the history
  • Loading branch information
arpita0911patel authored Nov 26, 2024
2 parents cd52301 + 6256701 commit a653741
Show file tree
Hide file tree
Showing 22 changed files with 4,916 additions and 2,595 deletions.
3 changes: 1 addition & 2 deletions blog/2023-10-29-intro-docuhub-blog.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ slug: Welcome-DocuHub-Blog
# authors: apatel
authors:
- name: Arpita Patel
title: Creator/Maintener of DocuHub
url: https://github.com/arpita0911patel
image_url: https://avatars.githubusercontent.com/u/54657?v=4
tags: [blog, docuhub, nextgen, ciroh]
Expand All @@ -20,7 +19,7 @@ hide_table_of_contents: false
### What file name to use?

DocuHub will extract a YYYY-MM-DD date from many patterns such as `YYYY-MM-DD-my-blog-post-title.md` or `YYYY/MM/DD/my-blog-post-title.md`. This enables you to easily group blog posts by year, by month, or to use a flat structure.

<!-- truncate -->
### Example (with Metadata/Front matter)

To publish in the blog, create a Markdown file within the blog directory.
Expand Down
7 changes: 5 additions & 2 deletions blog/2024-07-31-IT-Update.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@
title: "CIROH Research CyberInfrastructure Update"
description: "July Monthly IT Update"
slug: July Monthly IT Update

authors:
- name: Arpita Patel
url: https://github.com/arpita0911patel
image_url: https://avatars.githubusercontent.com/u/54657?v=4
tags: [ciroh, 2i2c, google, gpu, reu, bigquery, api, aws, nsf ]
hide_table_of_contents: false
image: /img/summer-institute.jpg
Expand All @@ -15,7 +18,7 @@ We're excited to share some recent developments and updates from CIROH's Researc
- CIROH's Google Cloud Account is now fully operational and managed by our team. You can find more information [here.](https://docs.ciroh.org/docs/services/cloudservices/google%20cloud/)
- We're in the process of migrating our 2i2c JupyterHub to CIROH's Google Cloud account.
- We've successfully deployed the Google BigQuery API (developed by BYU and Google) for NWM data in our cloud. To access this API, please contact us at [email protected]. Please refer to [NWM BigQuery API](https://docs.ciroh.org/docs/products/Data%20Management%20and%20Access%20Tools/bigqeury-api/) to learn more.

<!-- truncate -->
### Support and Services
- Monthly AWS office hours are ongoing. For more details on how to join, email us at [email protected].
- We provided IT support for the Summer Institute 2024, REU students, and team leads this summer.
Expand Down
7 changes: 5 additions & 2 deletions blog/2024-08-28-Case-Studies.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@
title: "CIROH Cloud User Success Story"
description: "August Monthly Blog Update"
slug: August Monthly Blog Update

authors:
- name: Arpita Patel
url: https://github.com/arpita0911patel
image_url: https://avatars.githubusercontent.com/u/54657?v=4
tags: [ciroh, 2i2c, aws, google, nextgen, jupyterhub, hpc, teehr, ngen]
hide_table_of_contents: false
image: https://github.com/CIROH-UA/ngen-datastream/blob/main/docs/images/ngen-datastream.jpg?raw=true
Expand All @@ -16,7 +19,7 @@ This month, we are excited to showcase two case studies that utilized our cyberi
<span className="hero-image" style={{ textAlign: 'center' }}>
<img src="https://github.com/CIROH-UA/ngen-datastream/blob/main/docs/images/ngen-datastream.jpg?raw=true" alt="ngen-datastream image" style={{ width: '80%' }} />
</span>

<!-- truncate -->
### Overview:

CIROH’s cloud computing resources have allowed for the development of ngen-datastream, which automates the process of collecting and formatting input data for NextGen, orchestrating the NextGen run through NextGen In a Box (NGIAB), and handling outputs. This software allows users to run NextGen in an efficient, relatively painless, and reproducible fashion, increasing community access to the NextGen framework. ngen-datastream is already community accessible (https://github.com/CIROH-UA/ngen-datastream/tree/main) and making an impact on research. A major component of this software is the Amazon Web Services (AWS) cloud-based research datastream (https://github.com/CIROH-UA/ngen-datastream/tree/main/research_datastream). The research datastream is a CONUS-wide recurring NextGen simulation configured by the community. The terraform to build the AWS infrastructure exists in the ngen-datastream repository and current development focuses on CI/CD and enabling community contribution to the research datastream via edits to the NextGen configuration. Ultimately, these tools help distribute access throughout the community to cutting edge hydrologic research, maximizing the pace of progress of research to operations in hydrology.
Expand Down
7 changes: 5 additions & 2 deletions blog/2024-09-30-BigQuery.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@
title: "Accessing National Water Model (NWM) Data via Google Cloud BigQuery API"
description: "September Monthly Blog Update"
slug: September Monthly Blog Update

authors:
- name: Arpita Patel
url: https://github.com/arpita0911patel
image_url: https://avatars.githubusercontent.com/u/54657?v=4
tags: [bigquery, byu, nwm, ciroh, google, cloud, api, partnership]
hide_table_of_contents: false
image: /img/gcp_architecture_diagram.png
Expand Down Expand Up @@ -61,7 +64,7 @@ Several important historical and ongoing National Water Model (NWM) datasets are
<td></td>
</tr>
</table>

<!-- truncate -->
To gain access and run queries on BigQuery:

- Submit the NWM BigQuery API Access Form, available at: https://docs.ciroh.org/docs/products/Data%20Management%20and%20Access%20Tools/bigqeury-api/
Expand Down
2 changes: 1 addition & 1 deletion blog/2024-1-31-NextGen-Update-Monthly.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ tags: [news]
hide_table_of_contents: false
---
Welcome to the January edition of the CIROH DocuHub blog, where we share the latest updates and news about the Community NextGen project monthly. NextGen is a cutting-edge hydrologic modeling framework that aims to advance the science and practice of hydrology and water resources management. In this month's blog, we will highlight some of the recent achievements and developments of the Community NextGen team.

<!-- truncate -->
First, we are excited to announce that NextGen In A Box (NGIAB) is now available with Singularity support. This means that you can run NGIAB on any HPC system that does not support Docker, using Singularity containers. Singularity is a popular tool for creating and running portable and reproducible computational environments. To learn how to use NGIAB with Singularity, please visit our GitHub repository: [Ngen-Singularity](https://github.com/CIROH-UA/Ngen-Singularity).

Second, we have made several improvements and enhancements to NGIAB, such as updating the sample input data, upgrading the Boost library, adding auto mode run, and supporting geopackage format. You can find more details about these updates on our GitHub repository: [NGIAB-CloudInfra](https://github.com/CIROH-UA/NGIAB-CloudInfra).
Expand Down
9 changes: 6 additions & 3 deletions blog/2024-10-29-ScienceMeeting.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,21 @@
title: "CIROH Science Meeting 2024"
description: "October Monthly Blog Update"
slug: October Monthly Blog Update

authors:
- name: Arpita Patel
url: https://github.com/arpita0911patel
image_url: https://avatars.githubusercontent.com/u/54657?v=4
tags: [ciroh, science, conference, impact]
hide_table_of_contents: false
image: /img/2024CSM_GroupPano.png
---

The 2024 CIROH Science Meeting was a huge success, bringing together researchers, federal partners, and consortium members both in person and virtually. We're excited to share the valuable resources from this year's meeting with the wider CIROH community.
Presentation Materials Available

Slides and pictures from the various sessions, keynotes, and the Federal Town Hall have all been uploaded to a shared drive for easy access. You can find links to these materials here: [Access the Shared Drive with Presentation Slides](https://drive.google.com/drive/folders/1NsJEWHQBD92ndc3FD_bFEKgqTUqTxVM3)

Slides and pictures from the various sessions, keynotes, and the Federal Town Hall have all been uploaded to a shared drive for easy access. You can find links to these materials here: [Access the Shared Drive with Presentation Slides](https://drive.google.com/drive/folders/1NsJEWHQBD92ndc3FD_bFEKgqTUqTxVM3)

<!-- truncate -->

Thank you to everyone who participated in making the 2024 CIROH Science Meeting a resounding success. We look forward to continuing our collaborative research efforts across the consortium.

Expand Down
2 changes: 1 addition & 1 deletion blog/2024-2-29-NextGen-Update-Monthly.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Welcome to the February edition of the CIROH DocuHub blog, where we bring you th
Our team has been hard at work enhancing CIROH's Infrastructure and Community NextGen tools. Here are some highlights from February 2024:

1. We successfully launched our new On-premises Infrastructure, which is now fully operational. You can find documentation for it [here](/docs/services/on-prem/).

<!-- truncate -->
2. For NGIAB, we've made improvements to the CI pipeline for pull requests submitted with forked repositories. Now, we automatically build and test these submissions using the CI pipeline.

3. We've added documentation for the NWMURL python package, which offers utility functions tailored for accessing National Water Model (NWM) data URLs. This library streamlines the process of accessing NWM data for various purposes, including analysis, modeling, and visualization. You can explore the documentation [here](/docs/products/Data%20Management%20and%20Access%20Tools/dataaccess/).
Expand Down
2 changes: 1 addition & 1 deletion blog/2024-3-31-NextGen-Update-Monthly.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ hide_table_of_contents: false


The CIROH team has been diligently accelerating research cyberinfrastructure capabilities this month. We're thrilled to share key milestones achieved in enhancing the Community NextGen project and our cloud/on-premises platforms.

<!-- truncate -->
A significant highlight was the successful launch of our new fully operational on-premises infrastructure. Comprehensive documentation is now available [here](/docs/services/on-prem/), ensuring seamless access and utilization. Additionally, we've fortified the NextGen in a Box (NGIAB) ecosystem with bug fixes, repository enhancements, and initiated work on automating the CI pipeline for the [Singularity Repo](https://github.com/CIROH-UA/Ngen-Singularity)

Empowering our community remains a top priority. We've expanded the DocuHub knowledge base with dedicated sections on on-premises access guidelines, as well as policies and best practices for optimized infrastructure usage [here](/docs/policies/intro) . Furthermore, our team represented CIROH at the AWRA Geospatial Water Technology Conference in Orlando, sharing insights on leveraging geospatial data for water research. Refer [here](https://github.com/CIROH-UA/Conferences/tree/main/2024-03-26\_AWRA\_GeospatialWaterTechnology)
Expand Down
2 changes: 1 addition & 1 deletion blog/2024-4-16-AWRATuscaloosa.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ image: /img/awra2024.jpeg
The **CIROH CyberInfrastructure team** recently participated in the **AWRA 2024 Spring Conference**, co-hosted by **the Alabama Water Institute at the University of Alabama.**

Themed "Water Risk and Resilience: Research and Sustainable Solutions," the conference brought together a diverse group of water professionals to exchange knowledge and explore cutting-edge research in the field.

<!-- truncate -->
CIROH CyberInfrastructure team presented on these topics:
* **Accelerating Community Contribution to the Next Generation Water Resources Modeling Framework**
* **Creating a community dataset for high-speed national water model data access**
Expand Down
2 changes: 1 addition & 1 deletion blog/2024-4-16-GoogleCloudNext2024.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ image: /img/googlenext-3.jpeg
Hello everyone, and thanks for stopping by!

I recently had the incredible opportunity to attend Google Cloud Next 2024 in person for the first time, and it was truly an **amazing experience**. From **insightful keynote presentations and workshops to vibrant booths** buzzing with connections, the event was a whirlwind of innovation and inspiration.

<!-- truncate -->
One of the highlights was undoubtedly the abundance of AI announcements and advancements. Google continues to push the boundaries of what's possible, and it was exciting to witness the future of technology unfold.

Among the many highlights, **CIROH** achieved a significant milestone with its **first-ever session at Google Cloud Next**. The presentation, titled **"Channel the Flood Data Deluge: Unlocking the American National Water Model,"** [link](https://cloud.withgoogle.com/next/session-library?session=DEV209&utm_source=copylink&utm_medium=unpaidsoc&utm_campaign=FY24-Q2-global-ENDM33-physicalevent-er-next-2024-mc&utm_content=next-homepage-social-share&utm_term=-#all> ) led by **Kel Markert (Google), Dr. Dan Ames (BYU), and Michael Ames(SADA)** was a resounding success. The session [link](https://assets.swoogo.com/uploads/3771985-6615e3b5da2cf.pdf) shed light on the immense potential of the National Water Model and its ability to revolutionize water resource management.
Expand Down
2 changes: 1 addition & 1 deletion blog/2024-5-02-DevCon24.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ image: /img/devcon24_02.jpeg
</div>

The CIROH team recently participated in the **2nd Annual CIROH Developers Conference (DevCon24)**, held from May 29th to June 1st,2024. The conference brought together a diverse group of water professionals to exchange knowledge and explore cutting-edge research in the field of hydrological forecasting.

<!-- truncate -->
Reflecting CIROH's current research focus, the conference explored topics including hydrological modeling (NextGen), flood inundation mapping, hydroinformatics, social science, and community engagement. Attendees got the opportunity to delve deeper into specific areas through its well-structured training track. This year the tracks were:
* **NextGen**
* **Flood Inundation Mapping (FIM)**
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
sidebar_position: 3
sidebar_position: 4
title: "NextGen Datastream"
description: "NextGen Datastream"
tags:
Expand Down

This file was deleted.

Binary file not shown.
Binary file not shown.
Binary file not shown.

This file was deleted.

This file was deleted.

Loading

0 comments on commit a653741

Please sign in to comment.