0.13: [Bug]: garden cleanup namespace fails in clean environment when using kubernetes deploy action

[alexkuretz]

Garden Bonsai (0.13) Bug

Current Behavior

We run BUILDVERSION=pr-some-bug-fix garden cleanup namespace --env pr in a Github Action when a PR is closed to clean up the Kubernetes namespace for that PR. This has been working fine in our Garden 0.12 projects as well as Garden 0.13 projects that use container deploy actions. We've started the heavy lift of moving to kubernetes deploy actions since there is no kubernetes-container action and we need more control over our deployments.

These cleanup runs fail in our projects that have been converted to use the kubernetes deploy action when run in this "clean" environment as it appears to depend on files from the initial build/deploy.

Deleting deployments...
ℹ deploy.task-py            → Cleaning up...
✖ deploy.task-py            → Failed deleting task-py (took 0.8 sec)
✖ deploy.task-py            → Failed processing delete Deploy type=kubernetes name=task-py) (took 0.78 sec). This is what happened:

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Invalid manifest file path(s) declared in Deploy type=kubernetes name=task-py. Cannot find manifest file(s) at ./manifests/task-py.yml in /home/akuretz/work/my-project/.garden/build/task-py directory.
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Failed to delete Deploy type=kubernetes name=task-py): Error: delete Deploy type=kubernetes name=task-py) failed: Error: Invalid manifest file path(s) declared in Deploy type=kubernetes name=task-py. Cannot find manifest file(s) at ./manifests/task-py.yml in /home/akuretz/work/my-project/.garden/build/task-py directory.

Expected behavior

Garden cleanup shouldn't depend on outputs from initial build/deploy.

Reproducible example

I reproduced this in a dev deployment by running a Garden 0.13 project that uses kubernetes deploy action with a simple manifest.

kind: Deploy 
type: kubernetes
name: task-py
build: task-py
spec:
  files: ['./manifests/task-py.yml']

Run garden deploy successfully.
Run rm -rf .garden
Run garden cleanup namespace
See failure.

Workaround

kubectl delete ns my-project-namespace

Suggested solution(s)

Additional context

Your environment

• OS: Ubuntu Github runners, Ubuntu dev environment
• How I'm running Kubernetes: EKS
  garden version: 0.13.28

[thsig]

Hi @alexkuretz—thanks for the detailed writeup here!

To properly fix the cleanup namespace for cases like this, where the deploy fails unless the build has been staged, we'd also need to include build dependencies when deleting services. (which is part of what the cleanup namespace command does). That wouldn't be the desired behaviour for users who don't need Build steps to generate files that are required for deleting the services (since they wouldn't want to build before deleting services).

This is a current limitation of how the cleanup namespace command works. The workaround you suggested makes sense (i.e. just deleting the namespace using kubectl), but we'll keep this issue open for now.

[alexkuretz]

I don't understand why you labeled this a "feature request" and not a bug. I'm doing things the way garden instructs me to do them, and it doesn't work.

[thsig]

Yeah, agreed. I'll re-label this as a bug, since it's a limitation we really should look into removing.