Send data from Fluent Bit to Axiom
This step-by-step guide will help you collect any data like metrics and logs from different sources, enrich them with filters, and send them to Axiom.
Fluent Bit
Fluent Bit is an open-source Log Processor and Forwarder that allows you to collect any data like metrics and logs from different sources, enrich them with filters, and send them to multiple destinations like Axiom.
Installation
Visit the Fluent Bit download page to install Fluent Bit on your system.
You’d need to specify the org-id header if you are using personal token, it’s best to use an API token to avoid the need to specify the org-id header.
Learn more about API and personal token
Configuration
Fluent Bit configuration file supports four types of sections:
- Service: Defines global properties of your service using different keys available for a specific version.
- Input: Defines the input plugin and base configuration of your file.
- Filter: Defines the input plugin and configure the pattern tags for your configuration.
- Output: Specify a destination that certain records should follow after a Tag match.
All sections are configured in your .conf
file.
Example
The example below shows fluent Bit configuration that sends data to Axiom:
Fluent Bit filters
Fluent Bit provides several filter plugins that can be used to modify the logs. These filters can be added to the configuration file in the [FILTER]
section.
Here’s how you can do it:
AWS ECS filter
For AWS ECS, you can use the grep
filter which enriches logs with Amazon ECS metadata:
Kubernetes Filter
The kubernetes
filter enriches logs with Kubernetes metadata:
WASM Filter
Fluent Bit allows the usage of WebAssembly (WASM) based filters.
Send logs from Docker Compose with Fluent Bit
This section outlines how to configure Fluent Bit with Docker Compose to forward logs to Axiom. It includes setting up fluent-bit.conf
for log processing and docker-compose.yaml
for deploying Fluent Bit as a container. The setup captures logs from various system metrics, logs, and forwards them to Axiom.
Create Fluent Bit configuration file (fluent-bit.conf)
Replace $DATASET
with your Axiom dataset name and $API_TOKEN
with your Axiom API token.
Create Docker Compose file (docker-compose.yaml)
Ensure the volumes
section correctly maps the fluent-bit.conf
file to /fluent-bit/etc/fluent-bit.conf
inside the container with read-only access.
To start the Fluent Bit container using the Docker Compose configuration you’ve set up, execute the docker-compose up -d
command.
Was this page helpful?