Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Watcher isn't triggering refresh when the target service is in another namespace. #1781

Open
fnarvaez-applydigital opened this issue Oct 23, 2024 · 8 comments
Open

Watcher isn't triggering refresh when the target service is in another namespace. #1781

fnarvaez-applydigital opened this issue Oct 23, 2024 · 8 comments
Labels
feedback-provided

Comments

@fnarvaez-applydigital
Copy link

fnarvaez-applydigital commented Oct 23, 2024 •
edited
Loading

Describe the bug
Watcher is not triggering refresh when the target service is in another namespace.

I have this configmap which need to be shared between multiple services in different namespaces, I change the roles and the bindings to cluster roles, but isn't working. there is anything else that I should do or is not intended to work like that?

I leave the definitions for each service, config and logs.

watcher.yaml 
apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: spring-cloud-kubernetes-configuration-watcher
      name: spring-cloud-kubernetes-configuration-watcher
      namespace: spring-cloud-system
    spec:
      ports:
        - name: http
          port: 8888
          targetPort: 8888
      selector:
        app: spring-cloud-kubernetes-configuration-watcher
      type: ClusterIP
  - apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        app: spring-cloud-kubernetes-configuration-watcher
      name: spring-cloud-kubernetes-configuration-watcher
      namespace: spring-cloud-system
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app: spring-cloud-kubernetes-configuration-watcher
      name: spring-cloud-kubernetes-configuration-watcher:view
    roleRef:
      kind: ClusterRole
      apiGroup: rbac.authorization.k8s.io
      name: watcher-namespace-reader
    subjects:
      - kind: ServiceAccount
        name: spring-cloud-kubernetes-configuration-watcher
        namespace: spring-cloud-system
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: watcher-namespace-reader
    rules:
      - apiGroups: ["", "extensions", "apps"]
        resources: ["configmaps", "pods", "services", "endpoints", "secrets"]
        verbs: ["get", "list", "watch"]
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: spring-cloud-kubernetes-configuration-watcher-deployment
      namespace: spring-cloud-system
    spec:
      selector:
        matchLabels:
          app: spring-cloud-kubernetes-configuration-watcher
      template:
        metadata:
          labels:
            app: spring-cloud-kubernetes-configuration-watcher
        spec:
          serviceAccountName: spring-cloud-kubernetes-configuration-watcher
          containers:
            - name: spring-cloud-kubernetes-configuration-watcher
              image: springcloud/spring-cloud-kubernetes-configuration-watcher:3.1.3
              imagePullPolicy: IfNotPresent
              env:
                - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_CLOUD_KUBERNETES_CONFIGURATION_WATCHER
                  value: DEBUG
                - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_CLOUD_KUBERNETES_CLIENT_CONFIG_RELOAD
                  value: DEBUG
                - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_CLOUD_KUBERNETES_COMMONS_CONFIG_RELOAD
                  value: DEBUG
                - name: SPRING_CLOUD_KUBERNETES_RELOAD_NAMESPACES_0
                  value: spring-cloud-system
                - name: SPRING_CLOUD_KUBERNETES_CONFIGURATION_WATCHER_REFRESHDELAY
                  value: "30000"
                - name: SPRING_CLOUD_KUBERNETES_RELOAD_ENABLED
                  value: "TRUE"
              readinessProbe:
                httpGet:
                  port: 8888
                  path: /actuator/health/readiness
              livenessProbe:
                httpGet:
                  port: 8888
                  path: /actuator/health/liveness
              ports:
                - containerPort: 8888
discovery.yaml 
---
apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: spring-cloud-kubernetes-discoveryserver
      name: spring-cloud-kubernetes-discoveryserver
      namespace: spring-cloud-system
    spec:
      ports:
        - name: http
          port: 80
          targetPort: 8761
      selector:
        app: spring-cloud-kubernetes-discoveryserver
      type: ClusterIP
  - apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        app: spring-cloud-kubernetes-discoveryserver
      name: spring-cloud-kubernetes-discoveryserver
      namespace: spring-cloud-system
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        app: spring-cloud-kubernetes-discoveryserver
      name: spring-cloud-kubernetes-discoveryserver:view
    roleRef:
      kind: ClusterRole
      apiGroup: rbac.authorization.k8s.io
      name: discovery-namespace-reader
    subjects:
      - kind: ServiceAccount
        name: spring-cloud-kubernetes-discoveryserver
        namespace: spring-cloud-system
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: discovery-namespace-reader
    rules:
      - apiGroups: ["", "extensions", "apps"]
        resources: ["pods", "services", "endpoints"]
        verbs: ["get", "list", "watch"]
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: spring-cloud-kubernetes-discoveryserver-deployment
      namespace: spring-cloud-system
    spec:
      selector:
        matchLabels:
          app: spring-cloud-kubernetes-discoveryserver
      template:
        metadata:
          labels:
            app: spring-cloud-kubernetes-discoveryserver
        spec:
          serviceAccountName: spring-cloud-kubernetes-discoveryserver
          containers:
            - name: spring-cloud-kubernetes-discoveryserver
              image: springcloud/spring-cloud-kubernetes-discoveryserver:3.1.0
              imagePullPolicy: IfNotPresent
              readinessProbe:
                httpGet:
                  port: 8761
                  path: /actuator/health/readiness
              livenessProbe:
                httpGet:
                  port: 8761
                  path: /actuator/health/liveness
              ports:
                - containerPort: 8761
config.yaml 
kind: ConfigMap
apiVersion: v1
metadata:
  name: example-configmap
  namespace: spring-cloud-system
  labels:
    spring.cloud.kubernetes.config: "true"
  annotations:
    spring.cloud.kubernetes.configmap.apps: "foo-service"
data:
  features.newFeatureEnabled: "false"
watcher logs 

2024-10-23T15:11:58.316Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap example-configmap was updated in namespace spring-cloud-system
2024-10-23T15:11:58.316Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap example-configmap was updated in namespace spring-cloud-system
2024-10-23T15:11:58.316Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] o.s.c.k.c.watcher.WatcherUtil            : will schedule remote refresh based on apps : [helloworld-service]
2024-10-23T15:11:58.316Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : onEvent config-map: class V1ConfigMap {
    apiVersion: v1
    binaryData: null
    data: {asd=asdd, features.newFeatureEnabled=false}
    immutable: null
    kind: ConfigMap
    metadata: class V1ObjectMeta {
        annotations: {kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","data":{"asd":"asdd","features.newFeatureEnabled":"false"},"kind":"ConfigMap","metadata":{"annotations":{"spring.cloud.kubernetes.configmap.apps":"helloworld-service"},"labels":{"spring.cloud.kubernetes.config":"true"},"name":"example-configmap","namespace":"spring-cloud-system"}}
        , spring.cloud.kubernetes.configmap.apps=helloworld-service}
        creationTimestamp: 2024-10-23T15:10:14Z
        deletionGracePeriodSeconds: null
        deletionTimestamp: null
        finalizers: null
        generateName: null
        generation: null
        labels: {spring.cloud.kubernetes.config=true}
        managedFields: [class V1ManagedFieldsEntry {
            apiVersion: v1
            fieldsType: FieldsV1
            fieldsV1: {f:data={.={}, f:asd={}, f:features.newFeatureEnabled={}}, f:metadata={f:annotations={.={}, f:kubectl.kubernetes.io/last-applied-configuration={}, f:spring.cloud.kubernetes.configmap.apps={}}, f:labels={.={}, f:spring.cloud.kubernetes.config={}}}}
            manager: kubectl-client-side-apply
            operation: Update
            subresource: null
            time: 2024-10-23T15:11:58Z
        }]
        name: example-configmap
        namespace: spring-cloud-system
        ownerReferences: null
        resourceVersion: 47856690
        selfLink: null
        uid: 03b56dbb-4465-4668-9b31-c433cacf0aad
    }
}
2024-10-23T15:11:58.317Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] o.s.c.k.c.watcher.WatcherUtil            : Scheduling remote refresh event to be published for config-map: with appName : helloworld-service to be published in 30000 milliseconds
2024-10-23T15:11:58.324Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : environment from locateMapPropertySources : ApplicationReactiveWebEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[MapPropertySource {name='server.ports'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, CompositePropertySource {name='composite-configmap', propertySources=[KubernetesClientConfigMapPropertySource {name='configmap.spring-cloud-kubernetes-configuration-watcher.spring-cloud-system'}]}, CompositePropertySource {name='composite-secrets', propertySources=[]}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.yml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='spring.integration.poller'}, MapPropertySource {name='kafkaBinderDefaultProperties'},  {name='Management Server'}]}
2024-10-23T15:11:58.324Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : sources from locateMapPropertySources : [KubernetesClientConfigMapPropertySource {name='configmap.spring-cloud-kubernetes-configuration-watcher.spring-cloud-system'}]
2024-10-23T15:11:58.324Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : environment from findPropertySources: ApplicationReactiveWebEnvironment {activeProfiles=[kubernetes], defaultProfiles=[default], propertySources=[MapPropertySource {name='server.ports'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, CompositePropertySource {name='composite-configmap', propertySources=[KubernetesClientConfigMapPropertySource {name='configmap.spring-cloud-kubernetes-configuration-watcher.spring-cloud-system'}]}, CompositePropertySource {name='composite-secrets', propertySources=[]}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.yml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='spring.integration.poller'}, MapPropertySource {name='kafkaBinderDefaultProperties'},  {name='Management Server'}]}
2024-10-23T15:11:58.324Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : environment sources from findPropertySources : [MapPropertySource {name='server.ports'}, ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, CachedRandomPropertySource {name='cachedrandom'}, MapPropertySource {name='KUBERNETES_NAMESPACE_PROPERTY_SOURCE'}, CompositePropertySource {name='composite-configmap', propertySources=[KubernetesClientConfigMapPropertySource {name='configmap.spring-cloud-kubernetes-configuration-watcher.spring-cloud-system'}]}, CompositePropertySource {name='composite-secrets', propertySources=[]}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.yml]' via location 'optional:classpath:/''}, MapPropertySource {name='springCloudClientHostInfo'}, MapPropertySource {name='spring.integration.poller'}, MapPropertySource {name='kafkaBinderDefaultProperties'},  {name='Management Server'}]
2024-10-23T15:11:58.324Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : findPropertySources : [configmap.spring-cloud-kubernetes-configuration-watcher.spring-cloud-system]
2024-10-23T15:11:58.324Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : no changes found, reload will not happen
2024-10-23T15:11:58.324Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] o.s.c.k.c.c.reload.ConfigReloadUtil      : No change detected in config maps/secrets, reload will not happen
2024-10-23T15:15:12.125Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 47856690 and Watch connection expired: too old resource version: 47856690 (47857398) , will retry w/o resourceVersion next time
2024-10-23T15:15:12.126Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Watch expired, will re-list-watch soon
2024-10-23T15:15:13.126Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Start listing and watching...
2024-10-23T15:15:13.134Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap example-configmap was updated in namespace spring-cloud-system
2024-10-23T15:15:13.135Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:15:13.135Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap kube-root-ca.crt was updated in namespace spring-cloud-system
2024-10-23T15:15:13.135Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:16:20.904Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 47856690 and Watch connection expired: too old resource version: 47856690 (47858280) , will retry w/o resourceVersion next time
2024-10-23T15:16:20.905Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Watch expired, will re-list-watch soon
2024-10-23T15:16:21.905Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Start listing and watching...
2024-10-23T15:16:21.912Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap example-configmap was updated in namespace spring-cloud-system
2024-10-23T15:16:21.913Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:16:21.913Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap kube-root-ca.crt was updated in namespace spring-cloud-system
2024-10-23T15:16:21.913Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:22:46.146Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 47859239 and Watch connection expired: too old resource version: 47859239 (47863297) , will retry w/o resourceVersion next time
2024-10-23T15:22:46.146Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Watch expired, will re-list-watch soon
2024-10-23T15:22:47.146Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Start listing and watching...
2024-10-23T15:22:47.154Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap example-configmap was updated in namespace spring-cloud-system
2024-10-23T15:22:47.154Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:22:47.155Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap kube-root-ca.crt was updated in namespace spring-cloud-system
2024-10-23T15:22:47.155Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:24:21.922Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 47860135 and Watch connection expired: too old resource version: 47860135 (47864577) , will retry w/o resourceVersion next time
2024-10-23T15:24:21.922Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Watch expired, will re-list-watch soon
2024-10-23T15:24:22.922Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Start listing and watching...
2024-10-23T15:24:22.929Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap example-configmap was updated in namespace spring-cloud-system
2024-10-23T15:24:22.930Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:24:22.930Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap kube-root-ca.crt was updated in namespace spring-cloud-system
2024-10-23T15:24:22.930Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:27:59.164Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 47865172 and Watch connection expired: too old resource version: 47865172 (47867390) , will retry w/o resourceVersion next time
2024-10-23T15:27:59.164Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Watch expired, will re-list-watch soon
2024-10-23T15:28:00.165Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Start listing and watching...
2024-10-23T15:28:00.172Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap example-configmap was updated in namespace spring-cloud-system
2024-10-23T15:28:00.172Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:28:00.172Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap kube-root-ca.crt was updated in namespace spring-cloud-system
2024-10-23T15:28:00.172Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-10-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:31:59.939Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : ResourceVersion 47866423 and Watch connection expired: too old resource version: 47866423 (47870575) , will retry w/o resourceVersion next time
2024-10-23T15:31:59.939Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Watch expired, will re-list-watch soon
2024-10-23T15:32:00.940Z  INFO 1 --- [spring-cloud-kubernetes-configuration-watcher] [s.V1ConfigMap-1] i.k.c.informer.cache.ReflectorRunnable   : class io.kubernetes.client.openapi.models.V1ConfigMap#Start listing and watching...
2024-10-23T15:32:00.947Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap example-configmap was updated in namespace spring-cloud-system
2024-10-23T15:32:00.947Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
2024-10-23T15:32:00.947Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : ConfigMap kube-root-ca.crt was updated in namespace spring-cloud-system
2024-10-23T15:32:00.947Z DEBUG 1 --- [spring-cloud-kubernetes-configuration-watcher] [ool-15-thread-1] sClientEventBasedConfigMapChangeDetector : data in configmap has not changed, will not reload
@wind57
Copy link
Contributor

wind57 commented Oct 23, 2024

sorry, "but isn't working" is not something we can work with. We need a sample with clear instructions what you expect to happen and it does not. May be a github repo you can create, with clear steps:

  • I deploy configmap XXX via kubectl apply -f ....
  • I expect the watcher to invoke a service in ... and to see the value via going to http:// ...

Something along these lines. If you could create such a sample, things would be much simpler for us to re-produce and help you.

May be this is worth a read also.

Btw, I see this in your logs: "Scheduling remote refresh event to be published for config-map: with appName : helloworld-service to be published in 30000 milliseconds", so the watcher detected the change and did send it to your helloworld-service. Does such a service exists? Does it have the refresh endpoint enabled?

@fnarvaez-applydigital
Copy link
Author

fnarvaez-applydigital commented Oct 23, 2024 •
edited
Loading

Hi @wind57 , thanks for your comment.

I am expecting the watcher to call the refresh endpoint after 30 seconds once the change was detected, but it is not happening. the watcher detected the change and scheduled the refresh but after 30 seconds it did not send the request, it does not log anything about it, in fact the log you should see is something like the following:

Refresh sent to helloworld-service at URI address http://<ip>:<port> returned a xxx

The helloworld-service exists in the hello-world namespace and the watcher is deployed in the spring-cloud-system namespace.

What I did was the following

kubectl create ns hello-world
kubectl create ns spring-cloud-system
kubectl config set-context --current --namespace=hello-world
# install a springboot app as a deployment with a service account with clusterroles and roles.
# Apply the config.yaml in spring-cloud-system ns and then call /actuator/refresh manually and the helloworld service applied the changes successfully.
# after that I install the watcher and the discovery as follow.
kubectl config set-context --current --namespace=spring-cloud-system
kubectl apply -f watcher.yaml
kubectl apply -f discovery.yaml
# modified the config.yaml before test it again
kubectl apply -f config.yaml

and then I wait 30 seconds until magic happens, but it didn't send the request T.T...

Once thing important to mention, I did the same exercise installing the watcher and discovery in the same namespace as helloworld-service and it works like a charm.

PS: Ill prepare a repository with the example.

@wind57
Copy link
Contributor

wind57 commented Oct 24, 2024

I'll wait for your example so that I could reproduce it, because we have integration tests that test this specific case: watcher in one namespace, service in another namespace...

@wind57
Copy link
Contributor

wind57 commented Oct 25, 2024

@ryanjbaxter can you add please the feedback label in here?

@paschdan
Copy link

Hello,

we have exactly the same problem.

@wind57 if you are refering to those integration tests:

https://github.com/spring-cloud/spring-cloud-kubernetes/blob/main/spring-cloud-kubernetes-integration-tests/spring-cloud-kubernetes-k8s-client-configuration-watcher/src/test/java/org/springframework/cloud/kubernetes/configuration/watcher/ActuatorRefreshMultipleNamespacesIT.java

the config-watcher and the wiremock are deployed to the same namespace (default)

so this quote imo is not true

'll wait for your example so that I could reproduce it, because we have integration tests that test this specific case: watcher in one namespace, service in another namespace...

@wind57
Copy link
Contributor

wind57 commented Oct 29, 2024

I actually did not search when posting that quote, I should have added a "iirc".

If you have the same problem, can you provide a reproducer then? thank you

@wind57
Copy link
Contributor

wind57 commented Oct 29, 2024

I might squeeze some time in the near future to test this one out too

@fnarvaez-applydigital
Copy link
Author

fnarvaez-applydigital commented Oct 30, 2024 •
edited
Loading

@wind57 I was finally able to work on packaging the example https://github.com/framled/hello-configmaps-watcher

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feedback-provided
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ryanjbaxter @wind57 @paschdan @spring-cloud-issues @fnarvaez-applydigital