Make Journald input resilient to Journald errors

[belimawr]

Currently if there is any error reading the next message from Journald, the input will stop working and never recover, effectively stopping ingestion and never recovering.

This happens because any error reading a new message or publishing a message

beats/filebeat/input/journald/input.go

Lines 163 to 173 in ffcd181

	for {
	entry, err := parser.Next()
	if err != nil {
	return err
	}
	event := entry.ToEvent()
	if err := publisher.Publish(event, event.Private); err != nil {
	return err
	}
	}

is returned by the Run method that was called in a goroutine that logs it and then exits

beats/filebeat/input/v2/compat/compat.go

Lines 119 to 135 in ffcd181

	go func() {
	defer r.wg.Done()
	log.Infof("Input '%s' starting", name)
	err := r.input.Run(
	v2.Context{
	ID: r.id,
	Agent: *r.agent,
	Logger: log,
	Cancelation: r.sig,
	},
	r.connector,
	)
	if err != nil && !errors.Is(err, context.Canceled) {
	log.Errorf("Input '%s' failed with: %+v", name, err)
	} else {
	log.Infof("Input '%s' stopped (goroutine)", name)
	}

We need to make the Journald input more resilient to errors we get when calling the host's journald via github.com/coreos/go-systemd/v22/sdjournal.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

Even after the merge of #40061 and the migration to using journalctl this issue is still relevant, if journalctl crashes the input finishes and the ingestion of journal messages stops.