Fluentd
Fluentd is an open-source log collector that allows you to collect, aggregate, process, analyze, and route log files.
With Fluentd, you can collect logs from multiple sources and ship it instantly into Axiom
Installation
Visit the Fluentd download page to install Fluentd on your system.
You’d need to specify the org-id header if you are using personal token, it’s best to use an API token to avoid the need to specify the org-id header.
Learn more about API and personal token
Configuration
Fluentd lifecycle consist of five different components which are:
- Setup: Configure your
fluent.conf
file.
- Inputs: Define your input listeners.
- Filters: Create a rule to allow or disallow an event.
- Matches: Send output to Axiom when input data match and pair specific data from your data input within your configuration.
- Labels: Groups filters and simplifies tag handling.
When setting up Fluentd, the configuration file .conf
is used to connect its components.
Configuring Fluentd using the HTTP output plugin
The example below shows a Fluentd configuration that sends data to Axiom using the HTTP output plugin:
<source>
@type forward
port 24224
</source>
<match *.**>
@type http
endpoint https://api.axiom.co/v1/datasets/$DATASET_NAME/ingest
# Authorization Bearer should be an ingest token
headers {"Authorization": "Bearer <your-token>"}
json_array false
open_timeout 3
<format>
@type json
</format>
<buffer>
flush_interval 5s
</buffer>
</match>
Configuring Fluentd using the OpenSearch output plugin
The example below shows a Fluentd configuration that sends data to Axiom using the OpenSearch plugin:
<source>
@type tail
@id input_tail
<parse>
@type apache2
</parse>
path /var/log/*.log
tag td.logs
</source>
<match **>
@type opensearch
@id out_os
@log_level info
include_tag_key true
include_timestamp true
host "#{ENV['FLUENT_OPENSEARCH_HOST'] || 'api.axiom.co'}"
port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/datasets/$DATASET_NAME/elastic'}"
scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"
</match>
The example below shows a Fluentd configuration to hold logs in memory with specific flush intervals, size limits, and how to exclude specific logs based on patterns.
# Collect common system logs
<source>
@type tail
@id system_logs
<parse>
@type none
</parse>
path /var/log/*.log
pos_file /var/log/fluentd/system.log.pos
read_from_head true
tag system.logs
</source>
# Collect Apache2 logs (if they’re located in /var/log/apache2/)
<source>
@type tail
@id apache_logs
<parse>
@type apache2
</parse>
path /var/log/apache2/*.log
pos_file /var/log/fluentd/apache2.log.pos
read_from_head true
tag apache.logs
</source>
# Filter to exclude certain patterns (optional)
<filter **>
@type grep
<exclude>
key message
pattern /exclude_this_pattern/
</exclude>
</filter>
# Send logs to Axiom
<match **>
@type opensearch
@id out_os
@log_level info
include_tag_key true
include_timestamp true
host "#{ENV['FLUENT_OPENSEARCH_HOST'] || 'api.axiom.co'}"
port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/datasets/$DATASET_NAME/elastic'}"
scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"
<buffer>
@type memory
flush_mode interval
flush_interval 10s
chunk_limit_size 5M
retry_max_interval 30
retry_forever true
</buffer>
</match>
Collect and send PHP logs to Axiom
The example below shows a Fluentd configuration that sends PHP data to Axiom.
# Collect PHP logs
<source>
@type tail
@id php_logs
<parse>
@type multiline
format_firstline /^\[\d+-\w+-\d+ \d+:\d+:\d+\]/
format1 /^\[(?<time>\d+-\w+-\d+ \d+:\d+:\d+)\] (?<message>.*)/
</parse>
path /var/log/php*.log
pos_file /var/log/fluentd/php.log.pos
read_from_head true
tag php.logs
</source>
# Send PHP logs to Axiom
<match php.logs>
@type opensearch
@id out_os
@log_level info
include_tag_key true
include_timestamp true
host "#{ENV['FLUENT_OPENSEARCH_HOST'] || 'api.axiom.co'}"
port "#{ENV['FLUENT_OPENSEARCH_PORT'] || '443'}"
path "#{ENV['FLUENT_OPENSEARCH_PATH']|| '/v1/datasets/$DATASET_NAME/elastic'}"
scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'https'}"
ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
user "#{ENV['FLUENT_OPENSEARCH_USER'] || 'axiom'}"
password "#{ENV['FLUENT_OPENSEARCH_PASSWORD'] || 'xaat-xxxxxxxxxx-xxxxxxxxx-xxxxxxx'}"
index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'php-logs'}"
<buffer>
@type memory
flush_mode interval
flush_interval 10s
chunk_limit_size 5M
retry_max_interval 30
retry_forever true
</buffer>
</match>
Collect and send Scala logs to Axiom
The example below shows a Fluentd configuration that sends Scala data to Axiom
# Collect Scala logs
<source>
@type tail
@id scala_logs
<parse>
@type multiline
format_firstline /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}/
format1 /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) \[(?<thread>.*)\] (?<level>\w+) (?<class>[\w\.$]+) - (?<message>.*)/
</parse>
path /var/log/scala-app.log
pos_file /var/log/fluentd/scala.log.pos
read_from_head true
tag scala.logs
</source>
# Send Scala logs using HTTP plugin to Axiom
<match scala.logs>
@type http
endpoint "#{ENV['FLUENT_HTTP_ENDPOINT'] || 'https://api.axiom.co/v1/datasets/$DATASET_NAME/ingest'}"
headers {"Authorization": "Bearer #{ENV['FLUENT_HTTP_TOKEN'] || '<your-token>'}"}
<format>
@type json
</format>
<buffer>
@type memory
flush_mode interval
flush_interval 10s
chunk_limit_size 5M
retry_max_interval 30
retry_forever true
</buffer>
</match>
Send virtual machine logs to Axiom using the HTTP output plugin
The example below shows a Fluentd configuration that sends data from your virtual machine to Axiom using the apache
source type.
<source>
@type tail
@id input_tail
<parse>
@type apache2
</parse>
path /var/log/**/*.log
pos_file /var/log/fluentd/fluentd.log.pos
tag vm.logs
read_from_head true
</source>
<filter vm.logs>
@type record_transformer
<record>
hostname "#{Socket.gethostname}"
service "vm_service"
</record>
</filter>
<match vm.logs>
@type http
@id out_http_axiom
@log_level info
endpoint "#{ENV['AXIOM_URL'] || 'https://api.axiom.co'}"
path "/v1/datasets/${AXIOM_DATASET_NAME}/ingest"
ssl_verify "#{ENV['AXIOM_SSL_VERIFY'] || 'true'}"
<headers>
Authorization "Bearer ${AXIOM_API_TOKEN}"
Content-Type "application/json"
</headers>
<format>
@type json
</format>
<buffer>
@type memory
flush_mode interval
flush_interval 5s
chunk_limit_size 5MB
retry_forever true
</buffer>
</match>
The example below shows a Fluentd configuration that sends data from your virtual machine to Axiom using the nginx
source type.
<source>
@type tail
@id input_tail
<parse>
@type nginx
</parse>
path /var/log/nginx/access.log, /var/log/nginx/error.log
pos_file /var/log/fluentd/nginx.log.pos
tag nginx.logs
read_from_head true
</source>
<filter nginx.logs>
@type record_transformer
<record>
hostname "#{Socket.gethostname}"
service "nginx"
</record>
</filter>
<match nginx.logs>
@type http
@id out_http_axiom
@log_level info
endpoint "#{ENV['AXIOM_URL'] || 'https://api.axiom.co'}"
path "/v1/datasets/${AXIOM_DATASET_NAME}/ingest"
ssl_verify "#{ENV['AXIOM_SSL_VERIFY'] || 'true'}"
<headers>
Authorization "Bearer ${AXIOM_API_TOKEN}"
Content-Type "application/json"
</headers>
<format>
@type json
</format>
<buffer>
@type memory
flush_mode interval
flush_interval 5s
chunk_limit_size 5MB
retry_forever true
</buffer>
</match>