[ES output] Cannot index event logs does not contain the meta field any more.

[belimawr]

Since we introduced the early encoding of events for the Elasticserach output, the format for the indexing errors logged has changed, at its current status only the fields of the event and the ES error message are logged, however we used to log the whole beat.Event, which included the Meta field.

The meta filed contains meaningful information to debug indexing issues like:

• input ID
• Raw index
• Stream ID

Old log entry:

{
  "log.level": "warn",
  "@timestamp": "2024-01-25T14:48:51.115+0100",
  "message": "Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2024, time.January, 25, 14, 48, 46, 614819591, time.Local), Meta:{\"input_id\":\"your-input-id\",\"raw_index\":\"logs-generic-default\",\"stream_id\":\"your-filestream-stream-id\"}, Fields:{\"agent\":{\"ephemeral_id\":\"a06806a9-f18d-4ffa-bee1-debcc15f7cf5\",\"id\":\"0ff4eb46-71e1-4c49-a921-3b984b303c0f\",\"name\":\"millennium-falcon\",\"type\":\"filebeat\",\"version\":\"8.13.0\"},\"data_stream\":{\"dataset\":\"generic\",\"namespace\":\"default\",\"type\":\"logs\"},\"ecs\":{\"version\":\"8.0.0\"},\"elastic_agent\":{\"id\":\"0ff4eb46-71e1-4c49-a921-3b984b303c0f\",\"snapshot\":false,\"version\":\"8.13.0\"},\"event\":{\"dataset\":\"generic\"},\"host\":{\"architecture\":\"x86_64\",\"containerized\":false,\"hostname\":\"millennium-falcon\",\"id\":\"851f339d77174301b29e417ecb2ec6a8\",\"ip\":[\"42.42.42.42\",,\"ec8a:fc90:d347:6316:116e:8a27:f731:08ff\"],\"mac\":[\"95-A2-37-0D-71-73\",],\"name\":\"millennium-falcon\",\"os\":{\"build\":\"rolling\",\"family\":\"arch\",\"kernel\":\"6.7.0-arch3-1\",\"name\":\"Arch Linux\",\"platform\":\"arch\",\"type\":\"linux\",\"version\":\"\"}},\"input\":{\"type\":\"filestream\"},\"log\":{\"file\":{\"device_id\":\"34\",\"inode\":\"172876\",\"path\":\"/tmp/flog.log\"},\"offset\":1061765},\"message\":\"154.68.172.7 - ritchie3302 [25/Jan/2024:14:10:52 +0100] \\\"HEAD /supply-chains/metrics/platforms HTTP/1.1\\\" 502 13383\"}, Private:(*input_logfile.updateOp)(0xc000fc6d20), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"index_closed_exception\",\"reason\":\"closed\",\"index_uuid\":\"0pq-XIYfSjuUQhTxlJKJjQ\",\"index\":\".ds-logs-generic-default-2024.01.22-000001\"}, dropping event!",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-default",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-default"
  },
  "log.origin": {
    "file.line": 461,
    "file.name": "elasticsearch/client.go",
    "function": "github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).bulkCollectPublishFails"
  },
  "log.type": "event",
  "ecs.version": "1.6.0",
  "log.logger": "elasticsearch"
}

New format:

{
  "log.level": "warn",
  "@timestamp": "2024-08-13T16:42:09.008-0400",
  "log.logger": "elasticsearch",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus",
    "file.name": "elasticsearch/client.go",
    "file.line": 490
  },
  "message": "Cannot index event '{\"@timestamp\":\"2024-08-13T20:42:05.928Z\",\"host\":{\"name\":\"millennium-falcon\"},\"agent\":{\"version\":\"8.16.0\",\"ephemeral_id\":\"6d195bff-27a4-40c4-9b3e-c3ecb068f06e\",\"id\":\"6a760df8-a3e6-4369-886a-3f499c792302\",\"name\":\"millennium-falcon\",\"type\":\"filebeat\"},\"log\":{\"file\":{\"device_id\":\"40\",\"inode\":\"51817\",\"path\":\"/tmp/flog.log\"},\"offset\":101},\"string\":10,\"message\":\"index failure\",\"int\":\"not a number\",\"input\":{\"type\":\"filestream\"},\"ecs\":{\"version\":\"8.0.0\"}}\n' (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:380] failed to parse field [int] of type [long] in document with id 'iVl6TZEBA82tHj8dCPpP'. Preview of field's value: 'not a number'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"For input string: \\\"not a number\\\"\"}}, dropping event!",
  "service.name": "filebeat",
  "log.type": "event",
  "ecs.version": "1.6.0"
}

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)