[EPIC] [DevelopmentSeed partnership] Integrate jupyterhub-home-nfs helm chart into 2i2c infrastructure

[yuvipanda]

This issue is being refined by @sgibson91

Original issue body

@sunu is working on NASA-IMPACT/veda-jupyterhub#41, where he'll create a helm chart named jupyterhub-home-nfs (or something like that) to provide home directories with per-user quotas on JupyterHub.

This issue tracks @sgibson91 working with @sunu, @batpad and me in helping integrate this into our infrastructure repository and practices. We will meet soon to figure out what next steps are, and who is responsible for what.

Context

A new helm chart is being developed by the devseed team which will provide home directories with per-user quotas on JupyterHub. This issue is tracking the tasks to complete so in order to integrate that helm chart into our infrastructure for reuse.

Task List

Tasks

Preview Give feedback

1. [devseed] Create a helm chart called jupyterhub-home-nfs (or similar) #4724
   +0
2. [devseed] Setup chartpress for jupyterhub-home-nfs helm chart #4725
   +0
3. [devseed] jupyterhub-home-nfs is configured as an opt-in dependency of basehub #4726
   +0
4. [devseed] Add ability to provision an AWS EBS volume to terraform and deploy for VEDA cluster #4735
   +0
5. [devseed] Expand deployer exec homes command to optionally include EBS volumes for AWS clusters #4794
   +0
6. [devseed] Migrate existing VEDA staging data from EFS instance into in-cluster NFS server #4728
   +0
7. [devseed] jupyterhub-home-nfs is enabled VEDA staging hub and storage switched over to in-cluster NFS #4729
   +0
8. [devseed] Document how to enable and configure the jupyterhub-home-nfs chart #4730
   +0

TBD: Who is responsible for executing each task? Division of labour between 2i2c and devseed. UPDATE: Decided #4647 (comment)

Definition of Done

• A helm chart enabling home directories with per-user storage quotas for JupyterHub exists
• The staging veda hub has been migrated to use this new chart
• There's documentation on how this can be enabled for other clusters in the 2i2c infrastructure repository

Pre-defined Definition of Done

• All the tasks have been completed
• It has been tested to work and to not break any existing functionality
• Documentation was added so other engineers can implement this in the future if needed

[yuvipanda]

This looks good. I've proposed a deadline of sep 13 (or alternatively sep 20) on slack. waiting to hear back.

@sgibson91 can you split this into multiple smaller chunks and put them in refined? Thanks!

[sgibson91]

In a meeting with Tarashish on Friday, it was agreed that he would be responsible for completing the technical work and I would be available for mentoring/answering questions/generally helping with the integration into the 2i2c/infrastructure repo.

[sgibson91]

Hey @sunu 👋🏻 I have broken up this epic into smaller task issues and tried to refine them. Can you take a look and let me know if anything's unclear? We should also try and set some deadlines for each of these tasks such that this whole epic can be wrapped up by Sep 20, could you let me know some estimates based on where you currently are with the work please?

[sunu]

Hi @sgibson91! Thanks so much for breaking it down down into smaller tasks!

I'm aiming to finish the first three tasks this week. Let's plan to get on a brief call early next week to discuss the data migration process. We can aim to get the data migrated and switch veda staging hub over to the in-cluster NFS server next week. We can keep the following week as buffer and use it to document the whole process. Does that sound reasonable to you?

[sgibson91]

Thank you for setting up the meeting, and I've added rough due dates to each of the tasks. I shall also get you write access to the repo so you can merge yourself. Here are our review and merge guidelines for your reference but I am happy to be pinged on anything you'd like to chat about: https://infrastructure.2i2c.org/contributing/code-review/

[sgibson91]

I have opened #4795 to track any technical assistance I give (e.g., like today when we worked in the AWS account because I have access) and this issue will be tracked by 2i2c's engineering sprint board.

[sgibson91]

Done! 🎉

[Gman0909]

Closing PB issue https://2i2c.productboard.com/roadmap/7803626-product-delivery-flow/features/27821087

[yuvipanda]

@Gman0909 I think we should wait until at least #4835 is done before closing the PB issue.