Datasets system.auth and system.syslog are not available for Debian 12 under Data streams tab.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.11.0 BC3
BUILD: 67923
COMMIT: 714189fa2b0f0a4d9f3865a8fce08261211570c8

Host OS: Debian 12

Preconditions:

1. 8.11.0 BC3 Kibana cloud environment should be available.

Steps to reproduce:

1. Install agent on Debian 12 with agent policy having System integration.
2. Navigate to Data streams tab and observe no data available for system.auth and system.syslog datasets.

Screenshots:

Expected Result:
Datasets system.auth and system.syslog should be available for Debian 12 under Data streams tab.

What's working fine:

• Datasets system.auth and system.syslog are available for Debian 11 on both 8.10.4 and 8.11.0 BC3

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[amolnater-qasource]

JFI @pierrehilbert

[amolnater-qasource]

Hi Team,

In our further testing, we have also observed that on enabling system module for filebeat-8.11 on Debian 12, the data is not visible under Discover tab.

Screenshot:
system.yml:

Most likely, the issue is reproducible for beats because it is using the same missing datasets.

Please confirm if we need to report a separate issue for the same.

Thanks!!

[pierrehilbert]

Thx @amolnater-qasource for your testing here.
Debian 12 moved syslog to journalctl so that's normal that we can get this information anymore.
cc @cmacknz to have this under your radar.

[cmacknz]

We would probably have to start using the journald input on Debian 12 to fix this, https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-journald.html.

The biggest problem is that the journald input is in technical preview and we haven't been keeping up with bug fixes for it, so we can't just switch without fixing some of the larger problems. The list is at https://github.com/elastic/beats/issues?q=is%3Aissue+is%3Aopen+journald and some of the bugs are severe like elastic/beats#34077 which is a Filebeat crash.

[nimarezainia]

@cmacknz @pierrehilbert are we completely blocked on this? is it just the system.auth and system.syslog datasets that will be missing?

I'm not sure if we want to address all the journald inputs issues listed here https://github.com/elastic/beats/issues?q=is%3Aissue+is%3Aopen+journald - until at least OTEL is complete.

So what can we do in the mean time? missing those datasets seems like a regression. is there anyway to address those short of using journald input?

[pierrehilbert]

I don't think we have a simple that we can implement to solve the issue.
One solution from a user perspective would be to install rsyslog to have auth and syslog files as before.
sudo apt install rsyslog

[cmacknz]

One solution from a user perspective would be to install rsyslog to have auth and syslog files as before.
sudo apt install rsyslog

Forcing everyone to switch log daemons and/or duplicate logs to both places isn't a nice solution, at best it is a temporary work around.

is it just the system.auth and system.syslog datasets that will be missing?

I think it is any system log that used to be reported via syslog. This is at least system.auth and system.syslog.

Filestream can't read journald logs today because they are encoded in a binary format. So for this to work seamlessly we need to dynamically change the input type based on the host operating system version in every integration that wants logs from journald and syslog. Long term I think this is messy.

It would likely be far better for the journald binary format to be something that is natively built into filestream that we can detect automatically so integration authors and users don't have to care about this difference. I can see why we wrote a separate input but I don't like it from a maintenance or user experience perspective. This could be something we detect based on the file path. It also might make sense as a type of dynamically enabled parser.

With Debian switching to journald by default I don't think we have a choice but to invest more in making it easier to use journald.

The only complicating factor here is that agent itself runs just fine on Debian 12, integrations that were reading system log files just won't work anymore. That doesn't have to block people from running Elastic Defend or running agent in Debian 12 based containers. So perhaps the best compromise for now is to say we support it with the asterisk that the migration to journald is still in progress. I wish we were ahead of this but that might be the best path now.

[nimarezainia]

thank you @cmacknz for that explanation. I also think that we place a caveat on it. Journald is owned by our team so I think we should add this effort to the roadmap spreadsheet as it seems somewhat substantial.

[pierrehilbert]

Forcing everyone to switch log daemons and/or duplicate logs to both places isn't a nice solution, at best it is a temporary work around.

That was my point to be able to "unblock" users that would like to use Agent on Debian 12 and need to get their system log files as before, the time for us to put in place a proper solution.

[cmacknz]

I created elastic/beats#37086 to track doing this work. I didn't want to rewrite the description here, and I linked back to this issue.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)