Send data from Amazon CloudWatch to Axiom

This page explains how to send data from Amazon CloudWatch to Axiom.

Axiom CloudWatch Forwarder is a set of easy-to-use AWS CloudFormation stacks designed to forward logs from Amazon CloudWatch to Axiom. It includes a Lambda function to handle the forwarding and stacks to create Amazon CloudWatch log group subscription filters for both existing and future log groups.

Axiom CloudWatch Forwarder includes templates for the following CloudFormation stacks:

• Forwarder creates a Lambda function that forwards logs from Amazon CloudWatch to Axiom.
• Subscriber runs once to create subscription filters on Forwarder for Amazon CloudWatch log groups specified by a combination of names, prefix, and regular expression filters.
• Listener creates a Lambda function that listens for new log groups and creates subscription filters for them on Forwarder. This way, you don’t have to create subscription filters manually for new log groups.
• Unsubscriber runs once to remove subscription filters on Forwarder for Amazon CloudWatch log groups specified by a combination of names, prefix, and regular expression filters.

To determine the best method to send data from different AWS services, see Send data from AWS to Axiom.

The Axiom CloudWatch Forwarder is an open-source project and welcomes your contributions. For more information, see the GitHub repository.

Prerequisites

• Create an Axiom account.
• Create a dataset in Axiom where you send your data.
• Create an API token in Axiom with permissions to update the dataset you have created.

• Create an account on AWS Cloud.

Installation

To install the Axiom CloudWatch Forwarder, choose one of the following:

• Cloudformation stacks
• Terraform module

Install with Cloudformation stacks

1. Launch the Forwarder stack template on AWS. Copy the Forwarder Lambda ARN because it’s referenced in the Subscriber stack.
2. Launch the Subscriber stack template on AWS.
3. Launch the Listener stack template on AWS.

Install with Terraform module

1

Create a new Forwarder module in your Terraform file in the following way:

module "forwarder" {
  source           = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/forwarder"
  axiom_dataset    = "DATASET_NAME"
  axiom_token      = "API_TOKEN"
  prefix           = "axiom-cloudwatch-forwarder"
}

• Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.
• Replace DATASET_NAME with the name of the Axiom dataset where you want to send data.

Alternatively, create a dataset with the Axiom Terraform provider.

2

Create a new Subscriber module in your Terraform file in the following way:

module "subscriber" {
  source               = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/subscriber"
  prefix               = "axiom-cloudwatch-forwarder"
  forwarder_lambda_arn = module.forwarder.lambda_arn
  log_groups_prefix    = "/aws/lambda/"
}

3

Create a new Listener module in your Terraform file in the following way:

module "listener" {
  source               = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/listener"
  prefix               = "axiom-cloudwatch-forwarder"
  forwarder_lambda_arn = module.forwarder.lambda_arn
  log_groups_prefix    = "/aws/lambda/"
}

4

In your terminal, go to the folder of your main Terraform file, and then run terraform init.

5

Run terraform plan to check the changes, and then run terraform apply.

Filter Amazon CloudWatch log groups

The Subscriber and Unsubscriber stacks allow you to filter the log groups by a combination of names, prefix, and regular expression filters. If no filters are specified, the stacks subscribe to or unsubscribe from all log groups. You can also whitelist a specific set of log groups using filters in the CloudFormation stack parameters. The log group names, prefix, and regular expression filters included are additive, meaning the union of all provided inputs is matched.

Example

For example, you have the following list of log groups:

/aws/lambda/function-foo
/aws/lambda/function-bar
/aws/eks/cluster/cluster-1
/aws/rds/instance-baz

• To subscribe to the Lambda log groups exclusively, use a prefix filter with the value of /aws/lambda.
• To subscribe to EKS and RDS log groups, use a list of names with the value of /aws/eks/cluster/cluster-1,/aws/rds/instance-baz.
• To subscribe to the EKS log group and all Lambda log groups, use a combination of prefix and names list.
• To use the regular expression filter, write a regular expression to match the log group names. For example, \/aws\/lambda\/.* matches all Lambda log groups.
• To subscribe to all log groups, leave the filters empty.

Listener architecture

The optional Listener stack does the following:

• Creates an Amazon S3 bucket for AWS CloudTrail.
• Creates a trail to capture the creation of new log groups.
• Creates an event rule to pass those creation events to an Amazon EventBridge event bus.
• Sends an event via EventBridge to a Lambda function when a new log group is created.
• Creates a subscription filter for each new log group.

Remove subscription filters

To remove subscription filters for one or more log groups, launch the Unsubscriber stack template on AWS.

The log group filtering works the same way as the Subscriber stack. You can filter the log groups by a combination of names, prefix, and regular expression filters.

Alternatively, to turn off log forwarding to Axiom, create a new Unsubscriber module in your Terraform file in the following way:

module "unsubscriber" {
  source           = "axiomhq/axiom-cloudwatch-forwarder/aws//modules/unsubscriber"
  prefix           = "axiom-cloudwatch-forwarder"
  forwarder_lambda_arn = module.forwarder.lambda_arn
  log_groups_prefix    = "/aws/lambda/"
}