Send data from Vector to Axiom
This step-by-step guide will help you configure Vector to read and collect metrics from your sources using the Axiom sink.
Vector
Vector is a lightweight and ultra-fast tool for building observability pipelines. It has a built-in support for shipping logs to Axiom through the axiom
sink.
Installation
Follow the quickstart guide in the Vector documentation to install Vector, and to configure sources and sinks.
timestamp field
Logs and metrics sent via vector should use @timestamp
as the timestamp field like: {"@timestamp":"2022-04-14T21:30:30.658Z..."}
, not _time
. Axiom accepts many date strings and timestamps without knowing the format in advance, including Unix Epoch, RFC3339, and ISO 8601.
Configuration
Send data to Axiom with Vector using the file
method and the axiom
sink.
The example below configures Vector to read and collect logs from files and send them to Axiom:
-
Create a vector configuration file
vector.toml
with the following content: -
In the code above, replace the following:
- Replace
VECTOR_SOURCE_ID
with the Vector source ID. - Replace
PATH_TO_LOGS
with the path to the log files. For example,/var/log/**/*.log
. - Replace
SINK_ID
with the sink ID. - Replace
AXIOM_API_TOKEN
with the Axiom API token you have generated. - Replace
AXIOM_DATASET
with the name of the Axiom dataset where you want to send data.
- Replace
-
Run Vector to send logs to Axiom.
Example with data transformation
The example below deletes a field before sending the data to Axiom:
Replace FIELD_TO_REMOVE
with the field you want to remove.
Any changes to Vector’s file
method can make the code example above outdated. If this happens, please refer to the official Vector documentation on the file
method, and we kindly ask you to inform us of the issue using the feedback tool at the bottom of this page.
Send Kubernetes logs to Axiom
Send Kubernetes logs to Axiom using the Kubernetes source.
DATASET is the name of your dataset. When logs are sent from your vector, it’s stored in a dataset in Axiom.
See creating a dataset for more
TOKEN is used to ingest or query data to your dataset. API token can be generated from settings on Axiom dashboard.
See creating an API token for more
Send Docker logs to Axiom
To send Docker logs using the Axiom sink, you need to create a configuration file, for example, vector.toml
, with the following content:
Replace your_dataset_name
with the name of the dataset you want to send logs to in Axiom, and your_api_token
with your Axiom API token.
- Run Vector: Start Vector with the configuration file you just created:
Vector collects logs from Docker and forward them to Axiom using the Axiom sink. You can view and analyze your logs in your dataset.
Send AWS S3 logs to Axiom
To send AWS S3 logs using the Axiom sink, create a configuration file, for example, vector.toml
, with the following content:
Finally, run Vector with the configuration file using vector --config ./vector.toml
. This starts Vector and begins reading logs from the specified S3 bucket and sending them to the specified Axiom dataset.
Send Kafka logs to Axiom
To send Kafka logs using the Axiom sink, you need to create a configuration file, for example, vector.toml
, with the following code:
Finally, you can start Vector with your configuration file: vector --config /path/to/your/vector.toml
Send NGINX metrics to Axiom
To send NGINX metrics using Vector to the Axiom sink, first enable NGINX to emit metrics, then use Vector to capture and forward those metrics. Here is a step-by-step guide:
Step 1: Enable NGINX Metrics
Configure NGINX to expose metrics. This typically involves enabling the ngx_http_stub_status_module
module in your NGINX configuration.
- Open your NGINX configuration file (often located at
/etc/nginx/nginx.conf
) and in yourserver
block, add:
- Restart or reload NGINX to apply the changes:
This exposes basic NGINX metrics at the /metrics
endpoint on your server.
Step 2: Configure Vector
Configure Vector to scrape the NGINX metrics and send them to Axiom. Create a new configuration file (vector.toml
), and add the following:
Finally, you can start Vector with your configuration file: vector --config /path/to/your/vector.toml
Send Syslog logs to Axiom
To send Syslog logs using the Axiom sink, you need to create a configuration file, for example, vector.toml
, with the following code:
Send Prometheus metrics to Axiom
To send Prometheus scrape metrics using the Axiom sink, you need to create a configuration file, for example, vector.toml
, with the following code:
Check out the advanced configuration on Batch, Buffer configuration, and Encoding on Vector Documentation