DNS lookup does not work properly

[justin-sto]

(check apply)

• read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Problem

In order to increase security, we recently wanted to switch to Consul Transparent Proxy. However, due to that localhost will no longer be properly resolved. Instead we need to use the service name on Kubernetes.

However, due to that change, the fluentd output plugin for elasticsearch does not work anymore. It always complains, that it cannot resolve the hostname. It seems that this is a known issue, as I already found multiple issues here.

Steps to replicate

Deploy Consul with Transparent Proxy
Deploy the Fluentd Helm Chart on K8s
Try to establish a connection between fluentd and elasticsearch, which is protected by transparent proxy.

We are using the following output config:

<label @out_container_log>
  <match *.**>
    @type copy
    <store>
    @type elasticsearch
    default_elasticsearch_version 8
    host elasticsearch-master
    port 9200
    user elastic
    password PASS
    type_name fluentd
    logstash_format true
    logstash_prefix fluentd
    include_timestamp true
    reload_connections false
    reload_on_failure true
    reconnect_on_error true
    flush_mode default
    flush_interval 10s
    </store>
  </match>
</label>
<match apc_app_log.**>
  @type elasticsearch
  default_elasticsearch_version 8
  host elasticsearch-master
  port 9200
  user elastic
  password PASS
  suppress_type_name true
  logstash_format true
  target_index_key index
  include_timestamp true
  id_key _id
  routing_key routing
  remove_keys routing_key, index, _id
  reload_connections false
  reload_on_failure true
  reconnect_on_error true
  <buffer>
      @type file
      path /var/log/fluentd/buffer
      flush_mode immediate
      flush_thread_count 20
      chunk_limit_size 256m
      total_limit_size 20G
      queue_limit_length 200
      overflow_action throw_exception
      retry_type periodic
  </buffer>
</match>

Expected Behavior or What you need to ask

Other services can resolve the elasticsearch-master service, which results to the following output. However, unfortunately, something seems to be wrong with the way that DNS works in the plugin, if it cannot resolve to localhost.
Below is the nslookup command executed from another pod container

nslookup elasticsearch-master
;; Got recursion not available from 127.0.0.1, trying next server
;; Got recursion not available from 10.0.0.10
Server:         10.0.0.10
Address:        10.0.0.10#53

Name:   elasticsearch-master.default.svc.cluster.local
Address: 172.28.160.107
Name:   elasticsearch-master.default.svc.cluster.local
Address: 172.28.162.22

Using Fluentd and ES plugin versions

• Bare Metal or within Docker or Kubernetes or others? Kubernetes
• Fluentd v0.12 or v0.14/v1.0 - fluentd 1.17.0
• ES plugin 3.x.y/2.x.y or 1.x.y
  • paste boot log of fluentd or td-agent

2024-08-07 12:02:45 +0000 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.
2024-08-07 12:02:46 +0000 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="no address for elasticsearch-master (Resolv::ResolvError)"
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elastic-transport-8.3.2/lib/elastic/transport/transport/base.rb:324:in `rescue in perform_request'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elastic-transport-8.3.2/lib/elastic/transport/transport/base.rb:285:in `perform_request'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elastic-transport-8.3.2/lib/elastic/transport/transport/http/faraday.rb:36:in `perform_request'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elastic-transport-8.3.2/lib/elastic/transport/client.rb:197:in `perform_request'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-8.13.0/lib/elasticsearch.rb:86:in `verify_elasticsearch'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-8.13.0/lib/elasticsearch.rb:69:in `method_missing'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-api-8.13.0/lib/elasticsearch/api/actions/info.rb:43:in `info'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:498:in `detect_es_major_version'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:489:in `block in handle_last_seen_es_major_version'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/elasticsearch_index_template.rb:56:in `retry_operate'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:486:in `handle_last_seen_es_major_version'
  2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:338:in `configure'

• paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

abbrev (default: 0.1.0)
activesupport (7.1.3.2)
addressable (2.8.6)
aws-eventstream (1.3.0)
aws-partitions (1.927.0)
aws-sdk-core (3.195.0)
aws-sdk-kms (1.80.0)
aws-sdk-s3 (1.149.1)
aws-sdk-sqs (1.73.0)
aws-sigv4 (1.8.0)
base64 (0.2.0, default: 0.1.1)
benchmark (default: 0.2.0)
bigdecimal (3.1.8, default: 3.1.1)
bundler (2.5.10)
cgi (default: 0.3.6)
concurrent-ruby (1.2.3)
connection_pool (2.4.1)
cool.io (1.8.0)
csv (3.3.0, default: 3.2.5)
date (default: 3.2.2)
debug (1.6.3)
delegate (default: 0.2.0)
did_you_mean (default: 1.6.1)
digest (default: 3.1.0)
digest-crc (0.6.5)
domain_name (0.6.20240107)
drb (2.2.1, default: 2.1.0)
elastic-transport (8.3.2)
elasticsearch (8.14.0, 8.13.0)
elasticsearch-api (8.14.0, 8.13.0)
elasticsearch-xpack (7.17.10)
english (default: 0.7.1)
erb (default: 2.2.3)
error_highlight (default: 0.3.0)
etc (default: 1.3.0)
excon (0.110.0)
faraday (2.9.0)
faraday-excon (2.1.0)
faraday-net_http (3.1.0)
fcntl (default: 1.0.1)
ffi (1.16.3)
ffi-compiler (1.3.2)
fiddle (default: 1.1.0)
fileutils (default: 1.6.0)
find (default: 0.1.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-detect-exceptions (0.0.15)
fluent-plugin-elasticsearch (5.4.3)
fluent-plugin-grafana-loki (1.2.20)
fluent-plugin-kafka (0.19.2)
fluent-plugin-kubernetes_metadata_filter (3.4.0)
fluent-plugin-multi-format-parser (1.1.0)
fluent-plugin-parser-cri (0.1.1)
fluent-plugin-prometheus (2.1.0)
fluent-plugin-record-modifier (2.2.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-systemd (1.0.5)
fluentd (1.17.0, 1.16.5)
forwardable (default: 1.3.2)
getoptlong (default: 0.1.1)
http (5.2.0)
http-accept (1.7.0)
http-cookie (1.0.5)
http-form_data (2.3.0)
http_parser.rb (0.8.0)
i18n (1.14.5)
io-console (default: 0.5.11)
io-nonblock (default: 0.1.0)
io-wait (default: 0.2.1)
ipaddr (default: 1.2.4)
irb (default: 1.4.1)
jmespath (1.6.2)
json (default: 2.6.1, 2.1.0)
jsonpath (1.1.5)
kubeclient (4.11.0)
llhttp-ffi (0.5.0)
logger (default: 1.5.0)
lru_redux (1.1.0)
ltsv (0.1.2)
matrix (0.4.2)
mime-types (3.5.2)
mime-types-data (3.2024.0507)
minitest (5.22.3, 5.15.0)
msgpack (1.7.2)
multi_json (1.15.0)
mutex_m (0.2.0, default: 0.1.1)
net-ftp (0.1.4)
net-http (0.4.1, default: 0.3.0.1)
net-imap (0.2.4)
net-pop (0.1.1)
net-protocol (default: 0.1.2)
net-smtp (0.3.1.1)
netrc (0.11.0)
nkf (default: 0.1.1)
observer (default: 0.1.1)
oj (3.3.10)
open-uri (default: 0.2.0)
open3 (default: 0.1.1)
openssl (default: 3.0.1)
optparse (default: 0.2.0)
ostruct (default: 0.5.2)
pathname (default: 0.2.0)
power_assert (2.0.1)
pp (default: 0.3.0)
prettyprint (default: 0.1.1)
prime (0.1.2)
prometheus-client (4.2.2)
pstore (default: 0.1.1)
psych (default: 4.0.4)
public_suffix (5.0.5)
racc (default: 1.6.0)
rake (13.2.1, 13.0.6)
rbs (2.7.0)
rdoc (default: 6.4.1.1)
readline (default: 0.0.3)
readline-ext (default: 0.1.4)
recursive-open-struct (1.1.3)
reline (default: 0.3.1)
resolv (default: 0.2.1)
resolv-replace (default: 0.1.0)
rest-client (2.1.0)
rexml (3.2.5)
rinda (default: 0.1.1)
rss (0.2.9)
ruby-kafka (1.5.0)
ruby2_keywords (default: 0.0.5)
rubygems-update (3.5.10)
securerandom (default: 0.2.0)
serverengine (2.3.2)
set (default: 1.0.2)
shellwords (default: 0.1.0)
sigdump (0.2.5)
singleton (default: 0.1.1)
stringio (default: 3.0.1.2)
strptime (0.2.5)
strscan (default: 3.0.1)
syslog (default: 0.1.0)
systemd-journal (1.4.2)
tempfile (default: 0.1.2)
test-unit (3.5.3)
time (default: 0.2.2)
timeout (default: 0.2.0)
tmpdir (default: 0.1.2)
tsort (default: 0.1.0)
typeprof (0.21.3)
tzinfo (2.0.6)
tzinfo-data (1.2024.1)
un (default: 0.2.0)
uri (0.13.0, default: 0.12.2)
weakref (default: 0.1.1)
webrick (1.8.1)
yajl-ruby (1.4.3)
yaml (default: 0.2.0)
zlib (default: 2.1.1)

• ES version (optional) - 8.14.1
• ES template(s) (optional)