[kinesis-firehose] Support simple_aggregation option to send multiple log events as a record

[kangaechu]

I changed the Firehose plugin from firehose (aws/amazon-kinesis-firehose-for-fluent-bit) to kinesis_firehose (core plugin) in my project, which caused a sharp increase in billing.

Upon investigation, we found that the firehose plugin supports simple_aggregation, so it was sending multiple records as a single event.

"{\"DeliveryStreamName\":\"fluent-bit--test\",\"Records\":[{\"Data\":\"(records encoded by BASE64)\"}]}"

However, kinesis_firehose does not support simple_aggregation, so events are created for each record.

"{\"DeliveryStreamName\":\"fluent-bit--test\",\"Records\":[
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"},
{\"Data\":\"（a record encoded by BASE64）\"}]}"

There were plans to implement simple_aggregation when the kinesis_firehose plugin was created, but as of August 2024, it has not yet been implemented.

aws/amazon-kinesis-firehose-for-fluent-bit#12 (comment)

Additional context

• Add an option to send multiple log events as a record aws/amazon-kinesis-firehose-for-fluent-bit#105
• Add an option to send multiple log events as a record aws/amazon-kinesis-firehose-for-fluent-bit#12

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.