[BUG] Adding custom annotation with nested json using juicefs/mount-annotations not working

[deanpatel2]

What happened:

I am trying to add a custom annotation to mount pods via juicefs/mount-annotations in StorageClass, but the annotation never shows up on the pods.

What you expected to happen:

I expected the mount pod annotation to be present when inspecting the pod with this command:

kubectl describe pod <mount-pod> -n <namespace>

How to reproduce it (as minimally and precisely as possible):

This is my StorageClass configuration. My goal is to have the mount pod's annotated with the "ad.datadoghq.com/..." annotation which allows the Datadog agent in my Kubernetes cluster to scrape the prometheus metrics exposed at the /metrics endpoint.

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: juicefs-sc
mountOptions:
- writeback
parameters:
  csi.storage.k8s.io/node-publish-secret-name: <secret-name>
  csi.storage.k8s.io/node-publish-secret-namespace: <namespace>
  csi.storage.k8s.io/provisioner-secret-name: <secret-name>
  csi.storage.k8s.io/provisioner-secret-namespace: <namespace>
  juicefs/clean-cache: "true"
  juicefs/mount-annotations: |
    {
      "ad.datadoghq.com/juicefs-metrics.checks": {
        "openmetrics": {
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:9567/metrics",
              "metrics": [
                "juicefs.*"
              ],
              "tags": [
                "env:<env>",
                "service:juicefs",
                "version:<version>",
                "department:=<department>",
                "team:<team>"
              ],
              "send_distribution_buckets": "true",
              "send_histograms_buckets": "true",
              "send_monotonic_counter": "true"
            }
          ]
        }
      }
    }
  juicefs/mount-cpu-limit: "0"
  juicefs/mount-cpu-request: "100m"
  juicefs/mount-memory-limit: "1Gi"
  juicefs/mount-memory-request: "500Mi"
provisioner: csi.juicefs.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

I get error like this:

MountVolume.SetUp failed for volume : rpc error: code = Internal desc = Could not mount juicefs: rpc error: code = InvalidArgument desc = Parse yaml or json error: json: cannot unmarshal object into Go value of type string

Passing as stringified JSON does not work either, instead I get this error:

MountVolume.SetUp failed for volume : rpc error: code = Internal desc = Could not mount juicefs: rpc error: code = InvalidArgument desc = Parse yaml or json error: json: cannot unmarshal string into Go value of type map[string]string

In what format do I add custom annotations with StorageClass? When I try to override via storageClasses.mountPod.annotations https://github.com/juicedata/charts/blob/main/charts/juicefs-csi-driver/values.yaml#L419, it does not work either.

Anything else we need to know?

Environment:

• JuiceFS CSI Driver version (which image tag did your CSI Driver use): 0.23.5
• Kubernetes version (e.g. kubectl version): v1.28.2
• Object storage (cloud provider and region):
• Metadata engine info (version, cloud provider managed or self maintained): Redis self-managed
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others:

[zxh326]

annotations only support string: string, you need stringify nested json like this

  juicefs/mount-annotations: '{"ad.datadoghq.com/juicefs-metrics.checks": "{\"openmetrics\":{\"instances\":[{\"openmetrics_endpoint\":\"http://%%host%%:9567/metrics\",\"metrics\":[\"juicefs.*\"],\"tags\":[\"env:<env>\",\"service:juicefs\",\"version:<version>\",\"department:=<department>\",\"team:<team>\"],\"send_distribution_buckets\":\"true\",\"send_histograms_buckets\":\"true\",\"send_monotonic_counter\":\"true\"}]}}"}'

[deanpatel2]

Thank you this worked. Is the same true for controller.annotations?