Fluent Bit: Efficient Log Processing and Forwarding

Fluent Bit is a lightweight and high-performance log processor and forwarder. It's designed to collect, parse, filter, and store logs from various sources, making it an essential tool for modern observability stacks. Its small footprint and efficient design make it ideal for embedded systems, containers, and large-scale deployments.

Key Resources for Fluent Bit

• Official Docker Image: Access the official Fluent Bit Docker images for easy deployment.
• Docker Metadata in Docker Logs using Fluent Bit: Learn how to enrich your Docker logs with valuable metadata using Fluent Bit.
• Command line examples for fluent-bit and stdout/es: Find practical command-line examples for Fluent Bit, including outputting to stdout and Elasticsearch.
• Fluent Bit Guide by coralogix.com: A comprehensive guide to understanding and utilizing Fluent Bit for log management.
• Fluent Bit Official Documentation: The definitive source for Fluent Bit configuration, plugins, and advanced usage.
• Fluentd: Explore Fluentd, another popular log collector, and understand how Fluent Bit can complement it.

Basic Fluent Bit Integration Example

This section demonstrates a simple setup for running Fluent Bit to collect logs and output them in JSON lines format.

Running Fluent Bit with Docker

Start Fluent Bit as a Docker container, listening for logs via the forward input plugin and outputting to standard output.

$ docker run -p 127.0.0.1:24224:24224 fluent/fluent-bit:1.5 /fluent-bit/bin/fluent-bit -i forward -o stdout -p format=json_lines -f 1

Configuring Docker Containers to Use Fluent Bit

You can configure your Docker containers to send their logs directly to Fluent Bit using the fluentd log driver.

$ docker run --log-driver=fluentd -t ubuntu echo "Testing a log message"
Testing a log message

Example Fluent Bit Output

When configured as above, Fluent Bit will process and forward logs. Here's an example of what the output might look like when sending a log message from a Docker container:

{
  "date":1601638488,
  "container_id":"45eccdf719dc28629bded52c8b409d0b10d0efb6d4b72452fc369a256e31be97",
  "container_name":"/epic_tharp",
  "source":"stdout",
  "log":"Testing a log message\r"
}

Advanced Fluent Bit Configurations

Fluent Bit supports a wide range of input, filter, and output plugins, allowing for complex log routing and transformation pipelines. You can configure it to send logs to Elasticsearch, Splunk, AWS CloudWatch, and many other destinations. Explore the official documentation for detailed configuration options and best practices for your specific use case.