Consul/Vault registration results in DNS issues for vault.service.consul

[originaltrini0]

Overview of the Issue

I have a three-node cluster consisting of Nomad, Consul, and Vault.
I also use DNS forwarding of the .consul domain on my internal DNS servers.

While chasing down a DNS forwarder issue, specifically with vault.service.consul, I discovered that DNS answers are CNAMEs, and I do not get all the results of my unsealed Vault servers.

Queries against nomad.service.consul and consul.service.consul works fine.
I stood up a test BIND server during my troubleshooting, and I will show an example query below.

I believe this is a bug:

1. A query against vault.service.consul should output all unsealed healthy instances as documented here
2. I believe the results should be A and not CNAME records. Documentation that I could find shows the same.

Let me know if you require any further information.

$ named-checkconf -p
options {
	directory "/var/cache/bind";
	listen-on-v6  {
		"any";
	};
	dnssec-validation no;
	forwarders {
		1.1.1.1;
		9.9.9.9;
	};
};
zone "consul" IN {
	type forward;
	forward only;
	forwarders {
		192.168.100.10 port 8600;
	};
};
zone "." {
	type hint;
	file "/usr/share/dns/root.hints";
};
zone "localhost" {
	type master;
	file "/etc/bind/db.local";
};
zone "127.in-addr.arpa" {
	type master;
	file "/etc/bind/db.127";
};
zone "0.in-addr.arpa" {
	type master;
	file "/etc/bind/db.0";
};
zone "255.in-addr.arpa" {
	type master;
	file "/etc/bind/db.255";
};


$ dig @192.168.100.10 -p 8600 vault.service.consul

; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> @192.168.100.10 -p 8600 vault.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29059
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;vault.service.consul.		IN	A

;; ANSWER SECTION:
vault.service.consul.	0	IN	CNAME	prod-core-services03.

;; Query time: 1 msec
;; SERVER: 192.168.100.10#8600(192.168.100.10) (UDP)
;; WHEN: Mon Nov 18 21:40:59 UTC 2024
;; MSG SIZE  rcvd: 83


$ dig @127.0.0.1 vault.service.consul

; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> @127.0.0.1 vault.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 7892
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 59186158b326193101000000673bb48caa5ecc2764a9541a (good)
;; QUESTION SECTION:
;vault.service.consul.		IN	A

;; ANSWER SECTION:
vault.service.consul.	0	IN	CNAME	prod-core-services03.

;; AUTHORITY SECTION:
.			2654	IN	SOA	a.root-servers.net. nstld.verisign-grs.com. 2024111801 1800 900 604800 86400

;; Query time: 10 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Mon Nov 18 21:41:32 UTC 2024
;; MSG SIZE  rcvd: 186

---

Reproduction Steps

1. Create a cluster with three server nodes
2. Unseal all Vault instances
3. Configure DNS forwarding
4. Perform a DNS query against vault.fqdn yields an NXDOMAIN response
5. Perform a dig against vault.service.consul yields one CNAME out of the total of three unsealed Vault instances.

Consul info for both Client and Server

Client info

Output from client 'consul info' command here

Client agent HCL config

Server info

$ consul info
agent:
	check_monitors = 0
	check_ttls = 1
	checks = 5
	services = 5
build:
	prerelease = 
	revision = cddc6181
	version = 1.20.0
	version_metadata = 
consul:
	acl = disabled
	bootstrap = false
	known_datacenters = 1
	leader = true
	leader_addr = 192.168.100.10:8300
	server = true
raft:
	applied_index = 1721463
	commit_index = 1721463
	fsm_pending = 0
	last_contact = 0
	last_log_index = 1721463
	last_log_term = 138465
	last_snapshot_index = 1714448
	last_snapshot_term = 138463
	latest_configuration = [{Suffrage:Voter ID:REDACTED Address:192.168.100.12:8300} {Suffrage:Voter ID:REDACTED Address:192.168.100.11:8300} {Suffrage:Voter ID:REDACTED Address:192.168.100.10:8300}]
	latest_configuration_index = 0
	num_peers = 2
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 138465
runtime:
	arch = arm64
	cpu_count = 4
	goroutines = 235
	max_procs = 4
	os = linux
	version = go1.22.7
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 115521
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 46268
	members = 3
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 2018
	members = 3
	query_queue = 0
	query_time = 1

datacenter = "homelab"
data_dir = "/opt/consul/data"
encrypt = "REDACTED"
retry_join = [
  "192.168.100.11",
  "192.168.100.12"
]
server = true
bind_addr = "192.168.100.10"
client_addr = "127.0.0.1 192.168.100.10"
ui_config {
  enabled = true
}
log_level  = "INFO"

Operating system and Environment details

Ubuntu 24.10
Vault 1.18
Consul 1.20

Unsealed Vault instances

$ consul catalog nodes -service=vault
Node                  ID        Address         DC
prod-core-services01  fdaa9e18  192.168.100.10  homelab
prod-core-services02  8de9943e  192.168.100.11  homelab
prod-core-services03  36374725  192.168.100.12  homelab

Log Fragments