Monitors and notifiers

Learn how to setup monitors and notifiers to get alerts when queries match certain conditions.

This page introduces monitors and notifiers, and explains how you can use them to generate automated alerts from your event data.

A monitor is a background task that periodically runs a query that you define. For example, it counts the number of error messages in your logs over the previous 5 minutes. A notifier defines how Axiom notifies you about the monitor output. For example, Axiom can send you an email.

You can use the following types of monitor:

• Match monitors filter for key events and send them to you.
• Threshold monitors aggregate event data over time. When the results of the aggregation cross a threshold, Axiom sends you an alert.
• Anomaly monitors aggregate event data over time and look for values that are unexpected based the event history. When the results of the aggregation are too high or low compared to the expected value, Axiom sends you an alert.

Match monitors

Match monitors allow you to continuously filter your log data and send you matching events. Axiom sends a notification for each matching event. By default, the notification message contains the entire matching event in JSON format. When you define your match monitor using APL, you can control which event attributes to include in the notification message.

Axiom recommends using match monitors for alerting purposes only. A match monitor can send 10 notifications per minute and 500 notifications per day.

To create a match monitor, follow these steps:

1. Click the Monitors tab on the top of the page, and then click New monitor.
2. Click Match monitor.
3. Name your monitor and add a description.
4. Click Add notifier, and then select the notifiers that define how you want to receive notifications for this monitor. For more information, see Notifiers.
5. To define your query, use one of the following options:
   • To use the visual query builder, click Simple query builder. Select the filters, and then click Run query to preview the recent events that match your filters. To preview matching events over a specific period, select the time range.
   • To use Axiom Processing Language (APL), click Advanced query language. Write a query using the where operator to filter for events, and then click Run query to preview the results. To transform matching events before sending them to you, use the extend and the project operators. Don’t use aggregations in your query. For more information, see Introduction to APL.
6. When the preview displays the events that you want to match, click Create. You cannot create a match monitor if more than 500 events match your query within the past 24 hours.

You have created a match monitor, and Axiom alerts you about every event that matches the filters you set. Each notification contains the event details as shown in the preview.

Threshold monitors

Threshold monitors allow you to periodically aggregate your event data and compare the results of this aggregation to a threshold that you define. When the results cross the threshold, the monitor enters the alert state. The monitor remains in the alert state until the results no longer cross the threshold. A threshold monitor sends you a notification each time it enters or exits the alert state.

To create a threshold monitor, follow these steps:

1. Click the Monitors tab on the top of the page, and then click New monitor.
2. Click Threshold monitor.
3. Name your monitor and add a description.
4. Configure the monitor using the following options:
   • The threshold is the value to compare the results of the query to. This can be any numeric value.
   • The comparison operator is the rule to apply when comparing the results to the threshold. The possible values are above, above or equal, below, and below or equal.
   • The frequency is how often the monitor runs. This is a positive integer number of minutes.
   • The range is the time range for your query. This is a positive integer number of minutes. The end time is the time the monitor runs.
   • Alert on no data triggers the monitor when your query doesn’t return any data. Your query returns no data if no events match your filters and an aggregation used in the query is undefined. For example, you take the average of a field not present in any matching events.
   • You can group by attributes when defining your query. By default, your monitor enters the alert state if any of the values returned for the group-by attributes cross the threshold, and remains in the alert state until none of the values returned cross the threshold. To trigger the monitor separately for each group that crosses the threshold, enable Notify by group. At most one trigger notification is sent per monitor run. This option only has an effect if the monitor’s query groups by a non-time field.
5. Click Add notifier, and then select the notifiers that define how you want to receive notifications for this monitor. For more information, see Notifiers.
6. To define your query, use one of the following options:
   • To use the visual query builder, click Simple query builder. Click Visualize to select an aggregation method, and then click Run query to preview the results in a chart. The monitor enters the alert state if any points on the chart cross the threshold. Optionally, use filters to specify which events to aggregate, and group by fields to split the aggregation across the values of these fields.
   • To use Axiom Processing Language (APL), click Advanced query language. Write a query where the final clause uses the summarize operator, and then click Run query to preview the results. For more information, see Introduction to APL. If your query returns a chart, the monitor enters the alert state if any points on the chart cross the threshold. If your query returns a table, the monitor enters the alert state if any numeric values in the table cross the threshold. If your query uses the bin_auto function, Axiom displays a warning. To ensure that the monitor preview gives an accurate picture of future performance, use bin rather than bin_auto.
7. Click Create.

You have created a threshold monitor, and Axiom alerts you when the results from your query cross the threshold.

Anomaly monitors

Anomaly monitors allow you to aggregate your event data and compare the results of this aggregation to what can be considered normal for the query. When the results are too much above or below the value that Axiom expects based on the event history, the monitor enters the alert state. The monitor remains in the alert state until the results no longer deviate from the expected value. This can happen without the results returning to their previous level if they stabilize around a new value. An anomaly monitor sends you a notification each time it enters or exits the alert state.

To create an anomaly monitor, follow these steps:

1. Click the Monitors tab on the top of the page, and then click New monitor.
2. Click Anomaly monitor.
3. Name your monitor and add a description.
4. Configure the monitor using the following options:
   • The comparison operator is the rule to apply when comparing the results to the expected value. The possible values are above, below, and above or below.
   • The tolerance factor controls the sensitivity of the monitor. It defines how much deviation from the expected value to accept without triggering the monitor. It’s a range above and below the expected value. The higher the tolerance factor, the wider this range. When the results of the aggregation stay within this range, the monitor doesn’t trigger. When the results of the aggregation cross this range, the monitor triggers. The tolerance factor can be any positive numeric value.
   • The frequency is how often the monitor runs. This is a positive integer number of minutes.
   • The range is the time range for your query. This is a positive integer number of minutes. A longer time range allows the anomaly monitor to consider a larger number of datapoints when calculating the expected value.
   • Alert on no data triggers the monitor when your query doesn’t return any data. Your query returns no data if no events match your filters and an aggregation used in the query is undefined. For example, you take the average of a field not present in any matching events.
   • You can group by attributes when defining your query. By default, your monitor enters the alert state if any of the values returned for the group-by attributes deviate from the expected value, and remains in the alert state until none of the values returned deviates from the expected value. To trigger the monitor separately for each group that deviates from the expected value, enable Notify by group. At most one trigger notification is sent per monitor run. This option only has an effect if the monitor’s query groups by a non-time field.
   • Toggle Require seasonality to compare the results to seasonal patterns in your data. For example, your query produces a time series that increases at the same time each morning. Without accounting for seasonality, the monitor compares to recent results only. By toggling Require seasonality, the monitor compares the results to the same time of the previous day or week and only triggers if the results deviate from the expected seasonal pattern.
5. Click Add notifier, and then select the notifiers that define how you want to receive notifications for this monitor. For more information, see Notifiers.
6. To define your query, use one of the following options:
   • To use the visual query builder, click Simple query builder. Click Visualize to select an aggregation method, and then click Run query to preview the results in a chart. The monitor enters the alert state if any points on the chart deviate from the expected value. Optionally, use filters to specify which events to aggregate, and group by fields to split the aggregation across the values of these fields.
   • To use Axiom Processing Language (APL), click Advanced query language. Write a query where the final clause uses the summarize operator, and then click Run query to preview the results. For more information, see Introduction to APL. If your query returns a chart, the monitor enters the alert state if any points on the chart deviate from the expected value. If your query returns a table, the monitor enters the alert state if any numeric values in the table deviate from the expected value. If your query uses the bin_auto function, Axiom displays a warning. To ensure that the monitor preview gives an accurate picture of future performance, use bin rather than bin_auto.
7. Click Create.

You have created an anomaly monitor. Axiom alerts you when the results from your query are too high or too low compared to what’s expected based on the event history.

In the chart, the red dotted line displays the tolerance range around the expected value over time. When the results of the query cross this range, the monitor triggers.

Notifiers

Set up a notifier to determine the channel Axiom uses to alert you about what’s happening with your data. For example, Axiom can send you an email.

To create a notifier, follow these steps:

1. Click the Monitors tab on the top of the page, and then click Manage notifiers on the right.
2. Click New notifier on the top right.
3. Name your notifier.
4. Select and configure the channel Axiom uses to alert you. For more information, see the types of notifiers below.
5. Click Create.

By adding a notifier to a monitor, you receive a notification with the following message:

• When a match monitor matches an event, the message contains the full event if you created the monitor using the simple query builder, or the output of the APL query if you created the monitor using APL.
• When a threshold monitor changes state, the message includes a relevant value from the query results. If you enable Notify by group, the notification message also contains the relevant group value.

Email

Create an email notifier to notify a list of email addresses.

Slack

Create Slack notifiers to notify specific channels in your Slack organization using a webhook.

For more information on creating an incoming webhook in Slack, see the Slack documentation.

PagerDuty

Create a PagerDuty notifier to use all the incident management features of PagerDuty with Axiom.

1. In PagerDuty’s Events V2 API, create a new service named Axiom with the default settings. Copy the integration key. For more information, see the PagerDuty documentation
2. In Axiom, go to the Monitors tab.
3. Click Manage notifiers, and then click New notifier.
4. Click PagerDuty, and then enter the integration key.
5. Click Create.

Custom webhook

Create customizable webhook notifiers to connect to internal or external services.

The webhook URL receives an HTTP/HTTPS POST request with a content type of application/json together with any other headers you specify.

Use the following variables to define the content of your webhook:

• .Action has value Open when the notification corresponds to a match monitor matching or a threshold monitor triggering, and has value Closed when the notification corresponds to a threshold monitor resolving.
• .MonitorID is the unique identifier for the monitor associated with the notification.
• .Body is the message body associated with the notification. When the notification corresponds to a match monitor, this is the matching event data. When the notification corresponds to a threshold monitor, this provides information about the value that gave rise to the monitor triggering or resolving.
• .Description is the description of the monitor associated with the notification.
• .QueryEndTime is the end time applied in the monitor query that gave rise to the notification.
• .QueryStartTime is the start time applied in the monitor query that gave rise to the notification.
• .Timestamp is the time the notification was generated.
• .Title is the name of the monitor associated with the notification.
• .Value is the value that gave rise to the monitor triggering or resolving when the notification corresponds to a threshold monitor.

Use the Go template syntax to interact with these variables when defining a custom body for your webhook.

A sample webhook body is shown below. This applies the default template shown on creating a custom webhook to a sample notification generated from a threshold monitor triggering.

{
  "action": "Open",
  "event": {
    "monitorID": "CabI3w142069etTgd0",
    "body": "Current value of 57347 is above or equal to the threshold value of 0",
    "description": "",
    "queryEndTime": "2024-06-28 14:55:57.631364493 +0000 UTC",
    "queryStartTime": "2024-06-28 14:45:57.631364493 +0000 UTC",
    "timestamp": "2024-06-28 14:55:57 +0000 UTC",
    "title": "Axiom Monitor Test Triggered",
    "value": 57347,
    "matchedEvent": null
  }
}

Opsgenie

Create an Opsgenie notifier to use all the incident management features of OpsGenie with Axiom.

For more information on configuring an integration in Opsgenie, see the Opsgenie documentation.

Discord

Create a Discord notifier to notify specific channels in your Discord server. You need to provide Axiom with a Discord token and channel identifier to create a Discord notifier.

• To get a token, go to discord.com/developers/applications and create a new application.
• Click Bot > Add Bot > Reset Token to get your Discord token.
• Go to OAuth2 > URL Generator, check the Bot scope and the Send Messages permission, then copy and open the generated URL to add the bot to your server.
• To get the channel ID, go to User Settings > Advanced and enable developer mode. Then right-click a channel and click Copy ID.
• Confirm if the check box of the Discord Bot has the proper allow channel access permissions from your settings.

Discord Webhook

Create a Discord Webhook notifier to notify specific channels in your Discord server using a webhook.

For more information on creating a webhook in Discord, see the Discord documentation.

Microsoft Teams

Create a Microsoft Teams notifier to send a notification to a specific channel in your Microsoft Teams instance using a webhook.

For more information on creating an incoming webhook in Microsoft Teams, see the Microsoft documentation.

Snooze a monitor

Both monitors and notifiers can be snoozed by clicking the ‘alarm clock’ icon when configuring the monitor or notifier. Snoozing a monitor will prevent it from running until the snooze time is elapsed. Snoozing a notifier will prevent it from sending messages until the snooze time has elapsed.

Monitor examples

This section presents example monitor configurations for some common alerting use cases.

Notify on all occurrences of error

To receive a notification on all occurrences of an error, create a match monitor where the filter conditions match the events reporting the error.

To receive only certain attributes in the notification message, use the project operator.

Notify when error rate above threshold

To receive a notification when the error rate exceeds a threshold, create a threshold monitor with an APL query that identifies the rate of error messages.

For example, logs in your dataset ['sample_dataset'] have a status.code attribute that takes the value ERROR when a log is about an error. In this case, the following example query tracks the error rate every minute:

['sample_dataset']
| extend is_error = case(['status.code'] == 'ERROR', 1, 0)
| summarize avg(error) by bin(_time, 1m)

Other options:

• To trigger the monitor when the error rate is above or equal to 0.01, set the threshold value to 0.01 and the comparison operator to above or equal.
• To run the monitor every 5 minutes, set the frequency to 5.
• To keep the monitor in the alert state until 10 minutes have passed with the per-minute error rate remaining below your threshold value, set the range to 10.

Notify when number of error messages above threshold

To receive a notification when the number of error message of a given type exceeds a threshold, create a threshold monitor with an APL query that counts the different error messages.

For example, logs in your dataset ['sample_dataset'] have a error.message attribute. In this case, the following example query counts errors by type every 5 minutes:

['sample_dataset']
| summarize count() by ['error.message'], bin(_time, 5m)

Other options:

• To trigger the monitor when the count is above or equal to 10 for any individual message type, set the threshold to 10 and the comparison operator to above or equal.
• To run the monitor every 5 minutes, set the frequency to 5.
• To run the monitor the query with a range of 10 minutes, set the range to 10.

By default, the monitor enters the alert state when any of the counts returned by the query cross the threshold, and remains in the alert state until no counts cross the threshold. To alert separately for each message value instead, enable Notify by group.

Notify when response times spike

To receive a notification whenever your response times spike without having to rely on a single threshold, create an anomaly monitor with an APL query that tracks your median response time.

For example, you have a dataset ['my_traces'] of trace data with the following:

• Route information is in the route field.
• Duration information is in the duration field.
• For top-level spans, the parent_span_id field is empty.

The following query gives median response times by route in one-minute intervals:

['my_traces'] 
| where isempty(parent_span_id)
| summarize percentile(duration, 50) by ['route'], bin(_time, 1m)

Other options:

• To only trigger the monitor when response times are unusually high for a route, set the comparison operator to above.
• To run the monitor every 5 minutes, set the frequency to 5.
• To consider the previous 30 minutes of data when determining what sort of variation is expected for median response times for a route, set the range to 30.
• To notify separately for each route, enable Notify by group.