Objective JH-1: Enforce home directory quotas per-user

[batpad]

Motivation

User home directories are meant to store code and notebooks, with data being stored in object storage if possible. They are mounted via NFS, and currently can grow unbounded. The unboundedness can cause cloud cost problems (unlike memory / CPU, which are bounded inherently), as well as encourage coding patterns that are not cloud optimized (like downloading a TB of .nc files to home directory before working on them).

Proposal

We will enforce home directory storage quotas for each user, as well as an overall fixed size for the entire hub. This will require migrating from the existing AWS EFS based setup to running an in-cluster NFS server with XFS based projectquota.

Owners

• @yuvipanda (2i2c)
• @batpad (Development Seed)
• @sunu (Development Seed)

Success Criteria

• Users can not use more than their allowed quota of home directory space
• Migration to this new home directory set up is completed for at least one hub (hub.openveda.io to start)

[batpad]

@yuvipanda are you able to outline a little bit here what steps would be, and give me and @sunu some reading to get started?

And then we should probably get on a call early next week to kick this off.

Thanks!

[yuvipanda]

yes let's do a call soon to kick this off!

Here's some prior reading that may be useful for @sunu!

• Enforce User Storage Limits pangeo-data/pangeo-cloud-federation#654 is where a lot of prior discussion about this feature happened. Basically, us doing this work is (in my head) completion of that work. Reading up on that issue may help!
• We'll be creating a new helm chart, which will primarily provide the following two items:
  • a userspace NFS server, https://github.com/nfs-ganesha/nfs-ganesha, serving from a PVC attached to a deployment
  • A sidecar in the same deployment running https://github.com/yuvipanda/get-quota-your-home/ so it sets quotas appropriately on the same disk that the NFS server is running from
• We will need to write at least a little integration test for this helm chart

I hope that's a useful start, @sunu!

@batpad how do we schedule this?

[sunu]

Thanks @yuvipanda! This is very useful. I'll give those threads a read.

[sunu]

Here's an update on the progress on this issue so far:

• I have set up a repo at https://github.com/sunu/jupyter-home-nfs and created CI workflow to build and publish docker images for the latest NFS Ganesha version
• I have created some example k8s manifests to test the workflow end-to-end with https://github.com/yuvipanda/get-quota-your-home
• So far, everything works pretty well.
  • storage quota is enforced effectively. Here's an example of enforcing a 1MB quota:

    $ kubectl exec -it -n nfs-test nfs-client -- sh
    / # 
    / # dd if=/dev/zero of=1mb.bin bs=1M count=1
    1+0 records in
    1+0 records out
    1048576 bytes (1.0MB) copied, 0.002145 seconds, 466.2MB/s
    / # dd if=/dev/zero of=2mb.bin bs=1M count=2
    2+0 records in
    2+0 records out
    2097152 bytes (2.0MB) copied, 0.002712 seconds, 737.5MB/s
    / # cp 1mb.bin /data/
    / # cp 2mb.bin /data/
    cp: can't create '/data/2mb.bin': No space left on device

  • User files are retained after deleting and re-creating the cluster as long as we use a pre-provisioned disk

cc @batpad @yuvipanda

[yuvipanda]

@sunu AMAZING! This looks great, continue on! I'd like more comments in https://github.com/sunu/jupyter-home-nfs/blob/main/nfs-ganesha/start.sh but otherwise awesome job!

Use https://github.com/jupyterhub/chartpress for CI and packaging. You can also git merge the get-quota-the-home repo into your repo (preserving history) to make this process easier too.

Excited to see this go!

[batpad]

This is moving along really nicely and progress can be tracked here: 2i2c-org/infrastructure#4647

This should be on track to be able to do a full test on staging, ideally by end of this week.

[Gman0909]

Should this be closed now the tracked issue is complete?