- Understand performance degradation, data drift, and quality issues in a single metric or two metrics relative to each other
- Be notified of an issue either on a schedule or immediately
Alerts can be created by clicking on
Create Alert button on the Alerts page, or from a specific tile in workspaces.
There are two ways to configure an alert:
- We can alert on a single metric going out of range of some absolute values:
- We can alert on two metrics going out of range relative to each other:
Alerts can also be configured from a saved query. In this case, filters from the saved query will be imported and autofilled into the alert:
Once alert criteria is configured the next page will prompt for notification information.
Gantry can be configured to send alerts to a slack channel. This is currently the best way to receive alerts from Gantry. To receive Slack messages, you'll need to configure a Slack webhook. Slack gives instructions on how to do that here.
These two screens ask for all the following information:
- Metric: What value should be alerted on. An example could be
- Evaluation Window: How long the metric is allowed to be out of range before an alert fires for it. For example, you might not care if users don't like the correction for a day, but if the trend continues for a week you might want to be made aware.
- Evaluation Delay: Set this to 0 if you want to be alerted in real time. Otherwise, this will control how long Gantry should wait after an alert is fired before notifying you. Modifying this value is mostly useful for measuring past or batch data.
- Filters: If you only want the alert to run on a subset of your data, you can filter the points of interest here.
- Range: If the metric or difference between metrics specified above is between the minimum and maximum values (or within a certain % tolerance), the alert will not fire.
- Name: The name you want to give the alert. When the alert fires, this name will be in the notification for it
- Add webhook: This is where you'll configure where you want the alert to be delivered. To understand how to use a webhook with Slack, see Slack documentation.
- Daily Notification Time: If you want a digest of you alerts at a specific time, specify it here. For example, you can ask to receive your alerts at 8am every day.
Alerts and their notifications can also be configured via the SDK. The following is an example of creating an alert based on two queries. It will trigger when the difference between the total records ingested today and yesterday is less than 0 or more than 2.
import gantry import os from dotenv import load_dotenv import datetime from gantry.alerts.client import AlertsAggregation from gantry.automations.triggers import QueriesAggregationTrigger from gantry.automations.actions import SendSlackMessage from gantry.automations.automations import Automation from gantry.query.time_window import RelativeTimeWindow load_dotenv() GANTRY_API_KEY = os.environ.get("GANTRY_API_KEY") WEBHOOK_URL = os.environ.get("WEBHOOK_URL") GANTRY_APP_NAME = "APP_NAME" gantry.init(api_key=GANTRY_API_KEY) app = gantry.get_application(GANTRY_APP_NAME) # Today time_window = RelativeTimeWindow(window_length = datetime.timedelta(days=1), offset=datetime.timedelta(days=1)) first_query = app.query( time_window, ) # Yesterday time_window = RelativeTimeWindow(window_length = datetime.timedelta(days=1)) second_query = app.query( time_window, ) query_trigger_alert = QueriesAggregationTrigger( name = "queries-trigger", compare_aggregation = AlertsAggregation.TOTAL_DIFFERENCE, compare_fields = ["__time"], queries = [first_query, second_query], query_aggregation=[AlertsAggregation.TOTAL, AlertsAggregation.TOTAL], lower_bound = 0, upper_bound = 2, trigger_range_type="outside", ) slack_action = SendSlackMessage( name="difference-between-total-records", webhook_url=WEBHOOK_URL ) # Define automation object and put trigger and action together. automation_alert = Automation( name="difference-between-total-records", trigger=query_trigger_alert, action=slack_action) # Add automation to the application. This will automatically start the automation app.add_automation(automation_alert)
Since Gantry allows the ingestion of historical data, there are two parameters to configure for alerts:
Evaluation window and
Evaluation windowdetermines the time window of data on which to compute the alert metric.
Evaluation delaycan be optionally configured to determine how long the system will wait for data to come in before evaluating a window.
Evaluation delayshould not be adjusted to create real-time alerts. It is intended to evaluate past or batch data.
For example, if the
Evaluation window is 15 minutes and the
Evaluation delay is 30 minutes, then at 12 pm, the alert will evaluate data between 11:15am and 11:30am.
When you create an alert, the hovering over the circle icon will help you understand the evaluation window:
Alerts and their statuses can be viewed from the alert page:
To edit, delete, or disable an alert from the UI, click the horizontal ellipsis
⋯ at the end of the row:
Deleting an alert will ask for confirmation.
Updated 4 months ago