Chapter 13 lab fluent not picking up annotations

[Alibirb]

In the chapter 13 lab, fluent-bit would not process annotations and run the nginx parser. Turning on "Annotations" in the configuration also did not seem to result in fluent-bit adding annotations to the log data, so it seems like it's failing to pick up the annotations?

I tried bringing everything down and up again multiple times, with no luck.

[jurlwin]

ok -- I have this working for me, now. Note for me that the parsing didn't really work anywhere in the chapter -- my issues went beyond the lab per se and I wasn't getting the dotnet-syslog parsing either (13.4 - parsing and filtering log entries), and so I'm going back and re-doing the chapter...as I want to get this right.

My issue is that I'm not running on dockerd. I"m running k3s on linux which uses containerd as the container portion, which adds logging fun. for me (and I haven't hit the lab again yet). My logs are all prefixed with a timestamp and some other stuff for CRI compatible logging. Everything before the <4> below was causing the Regex parsing to fail, which caused the annotations to appear not to work.

2022-11-28T20:59:12.395818741-05:00 stdout F <4>Numbers.Api.Controllers.RngController[0] Unhealthy! Failure ID: f1ec358f-2f4f-45d5-bcba-0fe4725f2930

googling got me to here once I figured out that the logging wasn't from me or something I explicitly did "wrong" and that the format is standard for containerd vs. docker: https://github.com/microsoft/fluentbit-containerd-cri-o-json-log

The fix - short version: change the [INPUT] parser from docker to cri and use the below CRI parser -- fixing the input parser works for these from my short testing as the downstream "log" doesn't have the timestamp, stream and log tag fields to mess up the later parsers.

[PARSER]
    Name        cri
    Format      regex
    Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[jurlwin]

I went back to this tonight -- so much for being a lunch time effort... the containerd logging issue would have bitten me too, but it still wasn't logging correctly as you note.

the issue now is that there is no RBAC setup in the lab, nor is it mentioned. I went about this by turning on debug in the fluent bit config - maybe warn would have still found it, but I don't recall -- and I found an authorization type error in the fluent bit pod logs saying it didn't have permission .. with a 403 error. by a few trial/error fixes to the deployment spec (changing the namespace from the original RBAC config in the fluent deployment yaml, and remembering to tag the spec with the serviceAccountName, I got rid of that error in the log and things started flowing through.

I think that will get you through at least parsing the log format for nginx.