[Stack 8.16.0-SNAPSHOT] [iptables] Failing test daily: system test: journald in iptables.log

[elastic-vault-github-plugin-prod]

• Stack version: 8.16.0-SNAPSHOT
• Package: iptables
• Failing test: system test: journald
• DataStream: log
• Owners:
  • @elastic/sec-deployment-and-devices

Failure:

test case failed: one or more errors found in documents stored in logs-iptables.log-87275 data stream: [0] field "journald.custom.realtime_timestamp" is undefined

First build failed: https://buildkite.com/elastic/integrations/builds/14541

Latest 5 failed builds:

• https://buildkite.com/elastic/integrations/builds/15011
• https://buildkite.com/elastic/integrations/builds/15051
• https://buildkite.com/elastic/integrations/builds/15097
• https://buildkite.com/elastic/integrations/builds/15161
• https://buildkite.com/elastic/integrations/builds/15214

[aleksmaus]

The issue here is that the new field journald.custom.realtime_timestamp coming in from the input in 8.16:

          "journald": {
            "custom": {
              "realtime_timestamp": "1642033008518660"
            },
            "host": {
              "boot_id": "c2f79f985830406a9e08241d015eff05"
            }
          },

The suspect change is elastic/beats#40061

At the moment the thinking is that the proper "fix" for this issues should be handled in the input.
The fields such as:
__REALTIME_TIMESTAMP=, __SEQNUM= , __SEQNUM_ID=
should be dropped, instead of trying to map it in the integration package as the proposed PR above.

@belimawr @cmacknz what do you think?

[andrewkroh]

The double-underscore prefix fields (https://www.freedesktop.org/software/systemd/man/254/systemd.journal-fields.html#Address%20Fields) coming from the JSON export format should be handled like trusted fields by the input IMO. I feel like the journald.custom.* namespace should only be used by untrusted fields.

[belimawr]

I agree with @andrewkroh should be trusted fields.

I believe the reason it is being mapped in journald.custom.* is because it is not listed as one of our default mapped fields, but we could add it there.

We use _SOURCE_MONOTONIC_TIMESTAMP as the @timestamp fields, but systemd docs state the following about __REALTIME_TIMESTAMP:

The wallclock time (CLOCK_REALTIME) at the point in time the entry was received by the journal, in microseconds since the epoch UTC, formatted as a decimal string. This has different properties from "_SOURCE_REALTIME_TIMESTAMP=", as it is usually a bit later but more likely to be monotonic.

It does seem to be an important field to keep around.

[andrewkroh]

@belimawr Do you want the security team to handle this change to the journald input?

[belimawr]

@belimawr Do you want the security team to handle this change to the journald input?

If you folks have time, that would be nice :)

[andrewkroh]

@belimawr, yes, we can handle this. Before we start let's discuss the expected behavior to ensure we are aligned.

It does seem to be an important field to keep around.

I have been thinking about this and looking at systemd source. I'd be hesitant to add another date field to the journald documents due to storage costs. The docs are already very large due to the all the systemd metadata that normally comes in them.

To stay aligned with journalctl's default rendering¹, I think for @timestamp the input should prefer using _SOURCE_MONOTONIC_TIMESTAMP if present (it is optional), and then fallback to using the mandatory __REALTIME_TIMESTAMP value. And we never pass through any journald.realtime_timestamp field (we can always add it later if there is demand, but we start without it.)

How does that sound to you both, @belimawr and @aleksmaus?

Footnotes

1. https://github.com/systemd/systemd/blob/803a3924715f21ac9d3392793e6ec669349fbac9/src/shared/logs-show.c#L457 ↩

[aleksmaus]

How does that sound to you both, @belimawr and @aleksmaus?

Agreed on dropping that field If there is no use for it at the moment.

[belimawr]

How does that sound to you both, @belimawr and @aleksmaus?

I agree as well, it will stay consistent with the current implementation, which is good. I believe we should also document it in the input docs so users know where we're getting the @timestamp from.

[aleksmaus]

We use _SOURCE_MONOTONIC_TIMESTAMP as the @timestamp fields, but systemd docs state the following about __REALTIME_TIMESTAMP:

I just checked the code, it looks like it looks like you are using __REALTIME_TIMESTAMP for the timestamp
https://github.com/elastic/beats/blob/main/filebeat/input/journald/pkg/journalctl/reader.go#L294

		ts, isString := fields["__REALTIME_TIMESTAMP"].(string)
		if !isString {
			return JournalEntry{}, fmt.Errorf("'__REALTIME_TIMESTAMP': '%[1]v', type %[1]T is not a string", fields["__REALTIME_TIMESTAMP"])
		}

gets assigned:
https://github.com/elastic/beats/blob/main/filebeat/input/journald/pkg/journalctl/reader.go#L319

		return JournalEntry{
			Fields:             fields,
			RealtimeTimestamp:  unixTS,
			Cursor:             cursor,
			MonotonicTimestamp: monotonicTSInt,
		}, nil

get set to the message:
https://github.com/elastic/beats/blob/main/filebeat/input/journald/input.go#L277

	m := reader.Message{
		Ts:      time.UnixMicro(int64(data.RealtimeTimestamp)),
		Content: content,
		Bytes:   len(content),
		Fields:  fields,
		Private: checkpoint{
			Version:            cursorVersion,
			RealtimeTimestamp:  data.RealtimeTimestamp,
			MonotonicTimestamp: data.MonotonicTimestamp,
			Position:           data.Cursor,
		},
	}