Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opentelemetry_envelope is not including all properties from input json #9732

Open
ssrslmcode opened this issue Dec 16, 2024 · 0 comments
Open

Comments

@ssrslmcode
Copy link

ssrslmcode commented Dec 16, 2024

Bug Report

Describe the bug
I am trying to setup fluent-bit to process log data into otel from backed and I use below configuration for fluent-bit and otel-collector

- Steps to reproduce the problem:

fluent-bit.yaml


service:
  flush: 1
  log_level: debug
  http_server: true
  http_listen: 0.0.0.0
  http_port: 2020
  daemon: off

pipeline:
  inputs:
    
    - name: dummy
      dummy: '{"level":"DEBUG", "thread":"1", "logger":"TestConnectionManager", "line":"42", "method":"ConnectionTestAlive", "message":"Connection Failure: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 25 - Connection string is not valid)"}'

      processors:
        logs:
          - name: opentelemetry_envelope
  
          - name: content_modifier
            context: otel_resource_attributes
            action: upsert
            key: service.name
            value: my-service-new
  
  filters:
    - name: record_modifier
      match: '*'
      record:
         - hostname ${HOSTNAME}
         - product Awesome_Tool

    - name: kubernetes
      match: 'demo-app'
      kube_url: https://kubernetes.default.svc:443
      kube_ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kube_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kube_tag_prefix: kube.var.log.containers.
      merge_log: off
      keep_log: off
      merge_log_key: log
      merge_log_trim: on
      k8s-logging.exclude: on
      labels: on
      annotations: on
      k8s-logging.parser: off
      regex_parser: k8s-attributes-parser

  outputs:
    - name : stdout
      match: '*'

    - name: opentelemetry
      match: '*'
      host: collector
      port: 4318
      logs_body_key: $message
      logs_severity_text_message_key: level
      add_label: app fluent-bit

parsers:
  - name: k8s-attributes-parser
    format: regex
    regex: (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$
    

otel-collector config

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"

exporters:
  file:
    path: /test.json
  debug:

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [file, debug]

Final otel collected data (test.json)

{
	"resourceLogs": [
		{
			"resource": {
				"attributes": [
					{
						"key": "service.name",
						"value": {
							"stringValue": "my-service-new"
						}
					}
				]
			},
			"scopeLogs": [
				{
					"scope": {},
					"logRecords": [
						{
							"timeUnixNano": "1734346978791329056",
							"severityText": "DEBUG",
							"body": {
								"stringValue": "Connection Failure: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 25 - Connection string is not valid)"
							},
							"traceId": "",
							"spanId": ""
						}
					]
				}
			]
		}
	]
}

Your Environment

  • Version used: Fluent-bit 3.2.2
  • Configuration: as mentioned in above
  • Environment name and version (e.g. Kubernetes? What version?): docker compose
  • Server type and version:
  • Operating System and version: windows
  • Filters and plugins: as mentioned in above

Additional context

  1. Is there any way to use record accessor like $log['method'] in content_modifier?
  2. Why it is not including other properties like thread, method, line from input (dummy)
  3. How to include additional properties like pod_name, container_name by assigning values from kubernetes filter
  4. How to assign value dynamically in content_modifier for otel_resource_attributes context, like to assign container or pod name for service.name
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant