Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[kube-prometheus-stack] global.ImageRegistry does not affect prometheus image for all instances #4734

Open
omeryahud opened this issue Jul 24, 2024 · 2 comments · May be fixed by #4776
Open

[kube-prometheus-stack] global.ImageRegistry does not affect prometheus image for all instances #4734

omeryahud opened this issue Jul 24, 2024 · 2 comments · May be fixed by #4776
Labels
bug Something isn't working

Comments

@omeryahud
Copy link

Describe the bug a clear and concise description of what the bug is.

When using the following values.yaml file, some images (specifically prometheus in my case) are pulled from upstream registries instead of the internal registry:

global:
  imageRegistry: registry.local/monitoring

The default prometheus instance's pod (within the stacks' namespace) pulls the image from the correct registry:

# kubectl get pod -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -oyaml | grep image:
    image: registry.local/monitoring/prometheus/prometheus:v2.52.0 # <----------------
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus/prometheus:v2.52.0 # <----------------
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0

However, instances in different namespaces still attempt to pull the upstream image:

# kubectl get pod -n test prometheus-test-0 -oyaml | grep image:
    image: quay.io/prometheus/prometheus:v2.52.0 # <----------------
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0
  - image: quay.io/prometheus/prometheus:v2.52.0 # <----------------
    image: registry.local/monitoring/prometheus-operator/prometheus-config-reloader:v0.74.0

After looking into the chart, and this template specifically: https://github.com/prometheus-community/helm-charts/blob/kube-prometheus-stack-59.1.0/charts/kube-prometheus-stack/templates/prometheus-operator/deployment.yaml#L88
I managed to workaround this issue by modifying my values.yaml file:

global:
  imageRegistry: registry.local/monitoring
prometheusOperator:
  prometheusDefaultBaseImage: prometheus/prometheus

which I don't think is the desired behavior

What's your helm version?

version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}

What's your kubectl version?

Client Version: v1.29.6 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.6

Which chart?

kube-prometheus-stack

What's the chart version?

59.1.0

What happened?

Prometheus instances that reside in a different namespace than the stack pull upstream prometheus images even when configured to pull them from an internal registry

What you expected to happen?

All prometheus instances in the cluster to pull all of their images from the internal registry configured in the custom values.yaml file

How to reproduce it?

  1. Run the following helm command:
helm upgrade -i kube-prometheus-stack --create-namespace -n monitoring ./kube-prometheus-stack --set global.imageRegistry="registry.local/monitoring"
  1. Create a second Prometheus instance in a different namespace than the stack's namespace
  2. Inspect the second instance's pods to see that the prometheus image is being pulled from upstream

Enter the changed values of values.yaml?

global:
  imageRegistry: registry.local/monitoring

Enter the command that you execute and failing/misfunctioning.

helm upgrade -i kube-prometheus-stack --create-namespace -n monitoring ./kube-prometheus-stack -f prom-values.yaml

Anything else we need to know?

No response
@omeryahud omeryahud added the bug Something isn't working label Jul 24, 2024
@jkroepke
Copy link
Member

Are you interest in a PR?

@omeryahud
Copy link
Author

Are you interest in a PR?

Sure I can take it, I'll get a PR ready in a couple of days

@omeryahud omeryahud linked a pull request Aug 7, 2024 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@jkroepke @omeryahud