k3s kube-system pods failure when using --bind-address

[shakibamoshiri]

Environmental Info:
K3s Version:
latest version v1.30.3+k3s1

Node(s) CPU architecture, OS, and Version:
Just one master node x86_64 Debian 11

Cluster Configuration:

Describe the bug:
I have several servers that each has its own private IP tunneled over VPN
I would like to setup the master node and bind the k3s address to the VPN IP e.g 192.168.254.204 (a layer 2)
If I use the --bind-address=192.168.254.204 the setup is done with no error but kube-system pods fail

kk get pod -n kube-system 
NAME                                      READY   STATUS             RESTARTS      AGE
coredns-576bfc4dc7-fjlpk                  0/1     Running            0             97s
local-path-provisioner-6795b5f9d8-js27t   0/1     CrashLoopBackOff   3 (49s ago)   97s
metrics-server-557ff575fb-p8thv           0/1     CrashLoopBackOff   3 (40s ago)   97s

and see error in k3s service log

Aug 03 20:13:23 204 k3s[1387]: I0803 20:13:23.770137    1387 garbagecollector.go:826] "failed to discover some groups" groups="<internal error: json: unsupported type: map[schema.GroupVersion]error>"
Aug 03 20:13:23 204 k3s[1387]: time="2024-08-03T20:13:23+03:30" level=info msg="Connecting to proxy" url="wss://91.107.210.195:6443/v1-k3s/connect"
Aug 03 20:13:23 204 k3s[1387]: time="2024-08-03T20:13:23+03:30" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 91.107.210.195:6443: connect: connection refused"
Aug 03 20:13:23 204 k3s[1387]: time="2024-08-03T20:13:23+03:30" level=error msg="Remotedialer proxy error; reconnecting..." error="dial tcp 91.107.210.195:6443: connect: connection refused" url="wss://91.107.210.195:6443/v1-k3s/connect"

next I uninstall k3s and started a new setup with both

• --bind-address=192.168.254.204
• --advertise-address=192.168.254.204

Then kube-system pod got working (expect the metric)

kk get pods -n kube-system 
NAME                                      READY   STATUS    RESTARTS       AGE
coredns-576bfc4dc7-fjlpk                  1/1     Running   0              10m
local-path-provisioner-6795b5f9d8-js27t   1/1     Running   11 (58s ago)   10m
metrics-server-557ff575fb-p8thv           0/1     Running   11 (58s ago)   10m

but still I see metric server error log

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  36s                default-scheduler  Successfully assigned kube-system/metrics-server-557ff575fb-ccrxq to 204
  Normal   Pulling    33s                kubelet            Pulling image "rancher/mirrored-metrics-server:v0.7.0"
  Normal   Pulled     30s                kubelet            Successfully pulled image "rancher/mirrored-metrics-server:v0.7.0" in 3.477s (3.477s including waiting). Image size: 19434712 bytes.
  Normal   Created    30s                kubelet            Created container metrics-server
  Normal   Started    30s                kubelet            Started container metrics-server
  Warning  Unhealthy  29s (x2 over 29s)  kubelet            Readiness probe failed: Get "https://10.42.0.4:10250/readyz": dial tcp 10.42.0.4:10250: connect: connection refused
  Warning  Unhealthy  1s (x16 over 28s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500

Steps To Reproduce:
run

curl -sfL https://get.k3s.io | sh -s - server --cluster-init --bind-address=192.168.254.204 --disable=traefik 

or

curl -sfL https://get.k3s.io | sh -s - server --cluster-init --bind-address=192.168.254.204 --advertise-address=192.168.254.204 --disable=traefik 

Expected behavior:
I need a working cluster without exposed api-server to public Internet, I run k3s on VMs that have public IPs but all VMs also connected via VPN and each one has its own private IP (it is not a cloud)

Actual behavior:
from other location (servers connected the VPN network) I can access (e.g from home) to the cluster via private IP (e.g 192.168.254.204) but seems there is a bug or I am setting it up incorrectly

Additional context / logs:

the kube-config address:

server: https://192.168.254.204:6443

but etcd configs shows the VM public IP has been set

/var/lib/rancher/k3s/server/db/etcd/config:advertise-client-urls: https://91.107.210.195:2379
/var/lib/rancher/k3s/server/db/etcd/config:initial-advertise-peer-urls: https://91.107.210.195:2380
/var/lib/rancher/k3s/server/db/etcd/config:initial-cluster: 204-5f774753=https://91.107.210.195:2380
/var/lib/rancher/k3s/server/db/etcd/config:listen-client-urls: https://127.0.0.1:2379,https://91.107.210.195:2379
/var/lib/rancher/k3s/server/db/etcd/config:listen-peer-urls: https://127.0.0.1:2380,https://91.107.210.195:2380
etc
etc

[shakibamoshiri]

version

k3s --version 
k3s version v1.30.3+k3s1 (f6466040)
go version go1.22.5

I did a new test with --bind-address and see the same error

kubectl get pods -n kube-system  
NAME                                      READY   STATUS             RESTARTS        AGE
coredns-576bfc4dc7-9k8m2                  0/1     Running            0               20m
helm-install-traefik-crd-glb9q            0/1     CrashLoopBackOff   8 (4m28s ago)   20m
helm-install-traefik-xsmml                0/1     CrashLoopBackOff   8 (4m42s ago)   20m
local-path-provisioner-6795b5f9d8-vqx5j   0/1     CrashLoopBackOff   8 (4m38s ago)   20m
metrics-server-557ff575fb-5k6w8           0/1     CrashLoopBackOff   8 (4m35s ago)   20m

further testing shows that for example this pod metrics-server-557ff575fb-5k6w8 error is because of TCP failure .
I can ping the pods IP but cannot netcat it

describe pod

Successfully assigned kube-system/metrics-server-557ff575fb-5k6w8 to 208
Pulling image "rancher/mirrored-metrics-server:v0.7.0"
Successfully pulled image "rancher/mirrored-metrics-server:v0.7.0" in 6.911s (6.911s including waiting). Image size: 19434712 bytes.
Readiness probe failed: Get "https://10.52.0.5:10250/readyz": dial tcp 10.52.0.5:10250: connect: connection refused
Readiness probe failed: Get "https://10.52.0.5:10250/readyz": read tcp 10.52.0.1:55902->10.52.0.5:10250: read: connection reset by peer
Created container metrics-server
Started container metrics-server
Container image "rancher/mirrored-metrics-server:v0.7.0" already present on machine
Readiness probe failed: Get "https://10.52.0.5:10250/readyz": read tcp 10.52.0.1:57652->10.52.0.5:10250: read: connection reset by peer
Back-off restarting failed container metrics-server in pod metrics-server-557ff575fb-5k6w8_kube-system(a8c21f81-0a9e-4df5-967b-899035bb3236)

ping (okay)

ping -c4 10.52.0.5
PING 10.52.0.5 (10.52.0.5) 56(84) bytes of data.
64 bytes from 10.52.0.5: icmp_seq=1 ttl=64 time=0.123 ms
64 bytes from 10.52.0.5: icmp_seq=2 ttl=64 time=0.108 ms
64 bytes from 10.52.0.5: icmp_seq=3 ttl=64 time=0.110 ms
64 bytes from 10.52.0.5: icmp_seq=4 ttl=64 time=0.097 ms

netcat (fails)

nc -v 10.52.0.5 10250
nc: connect to 10.52.0.5 port 10250 (tcp) failed: Connection refused

[brandond]

Have you tried just setting --node-ip and --flannel-iface instead of mucking about with --advertise-address and --bind-address?

[shakibamoshiri]

it is not MUCKING
When we are learning anything we have to go with trail-and-error

I am running k8s but recently I am looking for a lighter distro like yours to have multi-region multi-cluster dual-stack (did not do it before)
For security reason and using my own overlay network (VPN) I waned all services listen on private IPs, thus I tried --bind-address so the cluster be configured for cross node communication via VPN network and no port be opened on any other IPs or Interfaces (just on VPN IP), I do not want any services listen to public IPs and I thought --bind-address is for this purpose

Yes after seeing errors , I tried the other way. I tested --flannel-iface and --node-external-ip , Because my setup is on-perm (no cloud) this way I can have flannel running on a private network and ingress-controller working with public ip (set by --node-external-ip)
And for disallowing open ports of k3s, I tested nftables and faced some issues that I mentioned here k3s nftables rule of ingress controller bypasses host nftables #10693

Any other software I have in mind (nginx, haproxy, ssh, etc) , when we set an IP to bind to, they listen on that IP and working fine. This is a common expectation of any command or option to bind an address. Thus if I did it wrong and it is not a bug excuse me for such a silly thing, you can close the issue. If it is a bug , let it be open
Thank you

I have not tried --node-ip yet , but I will try and update here

[brandond]

I am looking for a lighter distro like yours to have multi-region multi-cluster dual-stack

If this is what you're going for, I would suggest reading the docs at https://docs.k3s.io/networking/distributed-multicloud#embedded-k3s-multicloud-solution - distributed clusters are hard to get right, and using the integrated wireguard/tailscale integration is your best bet.

[shakibamoshiri]

I have seen the wg or ts approach , unfortunately I cannot use them because UDP is being blocked time to time and I have to use SSL-VPN to pass Firewalls . Initially I thought it will be easy to setup reliability but now I see it is not that much easy :|

I will try the --node-ip options and update here
Thank you