Error pulling the coredns image from a custom repository

[viluon]

What Happened?

This is a reopening of

• --image-repository + coredns pulling issue due to namespace removed #13324,

which was closed incorrectly.

Using a custom repository, e.g.

minikube \
    start \
    --memory=8192mb \
    --cpus=4 \
    --image-repository=remote-docker.artifactory.REDACTED

leads to an image pull error. Minikube requests coredns:v1.11.1 when it should be pulling coredns/coredns:v1.11.1 instead.

Attach the log file

 * minikube v1.34.0 on Ubuntu 22.04 (amd64)
* Automatically selected the docker driver
* Using image repository remote-docker.artifactory.dbgcloud.io
* Using Docker driver with root privileges
* Starting "minikube" primary control-plane node in "minikube" cluster
* Pulling base image v0.0.45 ...

<snip>

 stdout:
[init] Using Kubernetes version: v1.31.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
 	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR ImagePull]: failed to pull image remote-docker.artifactory.REDACTED/coredns:v1.11.1: failed to pull image remote-docker.artifactory.REDACTED/coredns:v1.11.1: Error response from daemon: manifest for remote-docker.artifactory.REDACTED/coredns:v1.11.1 not found: manifest unknown: The named manifest is not known to the registry.

Operating System

Ubuntu

Driver

Docker

[neolit123]

remote-docker.artifactory.REDACTED/coredns:v1.11.1

put it in remote-docker.artifactory.REDACTED/coredns:v1.11.1

it's something the coredns maintainers decided to do and kubeadm people decided to be backwards compat.

Image paths between the default registry.k8s.io and a custom repository specified using imageRepository may differ for backwards compatibility reasons. For example, one image might have a subpath at registry.k8s.io/subpath/image, but be defaulted to my.customrepository.io/image when using a custom repository.

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images

default registry: uses coredns subpath
custom registry: doesn't use coredns subpath

[viluon]

@neolit123 this discrepancy is a bug from my PoV. In a sandboxed environment where only access to an Artifactory virtual repo (mirror of some upstream like registry.k8s.io) is available, --image-repository doesn't work, even if the repository mirrors the upstream perfectly.

I'm happy to open an issue for kubeadm but I think this one should stay open until the behaviour matches that of the default registry.

[neolit123]

i'm not a fan but it cannot be changed anymore.
also it breaks users in both options, once e.g. etcd start using a subpath as well.

your fix is to:

• pass an override using the kubeadm config clusterconfiguration.dns.image with the image that you want.
• tag images in a custom way.

[hajnej]

Hey @neolit123, is there way how to pass that custom kubeadm config? The --extra-config does not support overriding the dns.image and there is not any other way how to override the kubeadm config created by minikube kubeadm bootstrapper.

minikube start --image-repository=remote-docker.PRIVATE_DOMAIN --extra-config='kubeadm.dns.image=remote-docker.PRIVATE_DOMAIN/coredns/coredns:1.11.1'
😄  minikube v1.34.0 on Ubuntu 22.04
✨  Automatically selected the docker driver. Other choices: qemu2, ssh

❌  Exiting due to MK_USAGE: Sorry, the kubeadm.dns.image parameter is currently not supported by --extra-config

[neolit123]

that's a Q for the minikube maintainers.