Set up Alerts in Gantry to:

1. Understand performance degradation, data drift, and quality issues in a single metric or two metrics relative to each other
2. Be notified of an issue either on a schedule or immediately

Create an alert

UI

Alerts can be created by clicking on Create Alert button on the Alerts page, or from a specific tile in workspaces.

There are two ways to configure an alert:

1. We can alert on a single metric going out of range of some absolute values:

2. We can alert on two metrics going out of range relative to each other:

Alerts can also be configured from a saved query. In this case, filters from the saved query will be imported and autofilled into the alert:

Configure notifications

Once alert criteria is configured the next page will prompt for notification information.

Gantry can be configured to send alerts to a slack channel. This is currently the best way to receive alerts from Gantry. To receive Slack messages, you'll need to configure a Slack webhook. Slack gives instructions on how to do that here.

These two screens ask for all the following information:

• Metric: What value should be alerted on. An example could be percent_true(correction_accepted).
• Evaluation Window: How long the metric is allowed to be out of range before an alert fires for it. For example, you might not care if users don't like the correction for a day, but if the trend continues for a week you might want to be made aware.
• Evaluation Delay: Set this to 0 if you want to be alerted in real time. Otherwise, this will control how long Gantry should wait after an alert is fired before notifying you. Modifying this value is mostly useful for measuring past or batch data.
• Filters: If you only want the alert to run on a subset of your data, you can filter the points of interest here.
• Range: If the metric or difference between metrics specified above is between the minimum and maximum values (or within a certain % tolerance), the alert will not fire.
• Name: The name you want to give the alert. When the alert fires, this name will be in the notification for it
• Add webhook: This is where you'll configure where you want the alert to be delivered. To understand how to use a webhook with Slack, see Slack documentation.
• Daily Notification Time: If you want a digest of you alerts at a specific time, specify it here. For example, you can ask to receive your alerts at 8am every day.

SDK

Alerts and their notifications can also be configured via the SDK. The following is an example of creating an alert based on two queries. It will trigger when the difference between the total records ingested today and yesterday is less than 0 or more than 2.

import gantry
import os
from dotenv import load_dotenv
import datetime
from gantry.alerts.client import AlertsAggregation
from gantry.automations.triggers import QueriesAggregationTrigger
from gantry.automations.actions import SendSlackMessage
from gantry.automations.automations import Automation
from gantry.query.time_window import RelativeTimeWindow
load_dotenv()

GANTRY_API_KEY = os.environ.get("GANTRY_API_KEY")
WEBHOOK_URL = os.environ.get("WEBHOOK_URL")
GANTRY_APP_NAME = "APP_NAME"

gantry.init(api_key=GANTRY_API_KEY)
app = gantry.get_application(GANTRY_APP_NAME)

# Today
time_window = RelativeTimeWindow(window_length = datetime.timedelta(days=1), offset=datetime.timedelta(days=1))
first_query = app.query(
    time_window,
)

# Yesterday
time_window = RelativeTimeWindow(window_length = datetime.timedelta(days=1))
second_query = app.query(
    time_window,
)

query_trigger_alert = QueriesAggregationTrigger(
    name = "queries-trigger",
    compare_aggregation = AlertsAggregation.TOTAL_DIFFERENCE,
    compare_fields = ["__time"],
    queries = [first_query, second_query],
    query_aggregation=[AlertsAggregation.TOTAL, AlertsAggregation.TOTAL],
    lower_bound = 0,
    upper_bound = 2,
    trigger_range_type="outside",
)

slack_action = SendSlackMessage(
    name="difference-between-total-records",
    webhook_url=WEBHOOK_URL
)

# Define automation object and put trigger and action together.
automation_alert = Automation(
    name="difference-between-total-records",
    trigger=query_trigger_alert,
    action=slack_action)

# Add automation to the application. This will automatically start the automation
app.add_automation(automation_alert)

Understanding when alerts evaluate

Since Gantry allows the ingestion of historical data, there are two parameters to configure for alerts: Evaluation window and Evaluation delay.

• Evaluation window determines the time window of data on which to compute the alert metric.
• Evaluation delay can be optionally configured to determine how long the system will wait for data to come in before evaluating a window. Evaluation delay should not be adjusted to create real-time alerts. It is intended to evaluate past or batch data.

For example, if the Evaluation window is 15 minutes and the Evaluation delay is 30 minutes, then at 12 pm, the alert will evaluate data between 11:15am and 11:30am.

When you create an alert, the hovering over the circle icon will help you understand the evaluation window:

Viewing and editing alerts

Alerts and their statuses can be viewed from the alert page:

To edit, delete, or disable an alert from the UI, click the horizontal ellipsis ⋯ at the end of the row:

Deleting an alert will ask for confirmation.