Configuring spring boot kubernetes watcher to use actuator/shutdown endpoint doesn’t work as expected

[MeravYa]

Version: 3.1.3

I’m using the basic spring boot kubernetes config watcher image. When trying to set the reload strategy to shutdown - the spring boot kubernetes watcher is still using the /actuator/refresh endpoint.
I was adding this environment var to the watcher deployment:
spring.cloud.kubernetes.reload.strategy: shutdown

But when trying to change the config map, the application is only refreshed and not shuts down.

[wind57]

Yes, you are correct iirc. If you are using "kubernetes config watcher image" we will only be calling /refresh and spring.cloud.kubernetes.reload.strategy: shutdown has no meaning here, meaning we do not care about this property inside the configuration watcher.

This could be seen as an enhancement and we could be calling /shutdown or /refresh, based on a property.

@ryanjbaxter wdyt?

[ryanjbaxter]

Agree I can take a look at this one

[MeravYa]

@wind57 that could be great since refresh is insufficient for my use case, the environment is not reloaded as expected.
Meanwhile - is there any reference where I can understand what I am doing wrong? I am setting all spring properties as environment vars in my application deployment, which is obviously not reloaded upon refresh.
Thanks a lot for your quick response!

[wind57]

without a clear example (like a github repo) where you can show us what is going on, its going to be impossible to help you. Essentially, you help us with a good sample, we help you in figuring out the problem

[MeravYa]

Ok sure I will try to show a sample:
This is how my application deployment looks like:

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: sample-app 
  labels: app: sample-app 
spec: 
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
        - name:  sample-app
          image: "some-image:some-tag"
          ports:
            - containerPort: 8888
              protocol: TCP
          env:
            - name: SOME_PROPERTY
              valueFrom:
                configMapKeyRef:
                  name: configmap
                  key: SOME_PROPERTY

The config map will look like this:

apiVersion: v1 
kind: ConfigMap
metadata: 
  name: configmap 
  labels: 
    spring.cloud.kubernetes.config: "true" 
  annotations: 
    spring.cloud.kubernetes.configmap.apps: sample-app 
data: 
  SOME_PROPERTY: some_value

The property is initialized using
@configuration
@ConfigurationProperties

Now, when trying to reload the property either by hitting the /refresh manually, or by using the watcher - the property is not reloading. Only a complete shutdown will do the reload.

Thanks

[wind57]

can you put this into a github project that I can import easily? Also show instructions on how I should reproduce the issue please. Thank you

[MeravYa]

Please follow this link: https://github.com/MeravYa/sample-app/tree/master/sample-app

thanks

[wind57]

Unfortunately, I can not even start to help you, because:

Spring Cloud Dalston, Edgware, Finchley, Greenwich, 2020.0 (aka Ilford), 2021.0 (aka Jubilee), and 2022.0 (aka Kilburn) have all reached end of life status and are no longer supported.

So you need to upgrade, first.

[MeravYa]

Ok I understand. After doing further research I think my problem occurs since I'm not using config map as mount to the spring boot application pod.
Since the config map mount approach is less suitable for my use case - will you be able to add the ability to use shutdown endpoint for the spring watcher? (I'm using 3.1.3 version for the watcher).

[wind57]

yes, as you have seen Ryan will be taking this work.

[MeravYa]

Ok thanks a lot to both of you

[ryanjbaxter]

Supporting shutdown as a mode of refreshing configuration seems doable when using the config watcher in combination with HTTP because we can call the actuators /shutdown endpoint.

However it becomes a bit more complicated when the config watcher is using Spring Cloud Bus because Bus has no concept of a "shutdown event". We would first have to add support for a shutdown even in Spring Cloud Bus and then leverage that in the config watcher.

I will discuss that with the team and see what they think.

[wind57]

@MeravYa I've read your use case today in the morning again, and I think that /refresh should be more then enough for you. If you want to upgrade to a supported version and prepare a sample, I will take a look and see what is going on. The /shutdown from the comment above will take some time, thus my suggestion.

[MeravYa]

Hi @wind57 ,
I've updated the sample

[ryanjbaxter]

The first step in making the shutdown use case was merged into spring-cloud-bus here spring-cloud/spring-cloud-bus#277

[wind57]

sorry for responding a bit late... I had to do a bit of changes to make the app running in my local cluster, see here

And now if I run it and issue:

root@kind-control-plane:/# curl 10.244.0.6:8888/api/v1/sample/hello
Hello, config value: new-valueroot@kind-control-plane

I see the value injected from the configmap.

And the question is: what next? Because you do not have the set-up you use for the configuration watcher in the sample... So if you want to update the sample to include those steps also (exactly how to re-produce the problem), I can take a look further. Thank you.

[MeravYa]

@wind57 for some reason I missed the README in my commit. Please view it now please, I've also added the spring boot watcher. You can either reproduce by manually triggering the refresh endpoint, or by using the config watcher.

Thanks

[ryanjbaxter]

The ability to shut down the application when the apps configuration changes via the configuration watcher was merged into main #1799

[wind57]

so your problem lies in the fact (from official docs):

Although the value of the key inside the ConfigMap has changed, the environment variable in the Pod still shows the earlier value. This is because environment variables for a process running inside a Pod are not updated when the source data changes; if you wanted to force an update, you would need to have Kubernetes replace your existing Pods. The new Pods would then run with the updated information.

So there is not much we can do about. Also, I understand why you would want for such case the shutdown to be able to work with the configuration watcher.

But, we have support for slightly a different set-up, where things will work. If you can use spring.config.import, then things will work. I would encourage you to look at this one, as an example that works.

Also, if you are deploying the config watcher, this would be a read you should invest time into.

I hope that helps.