Set up Astro alerts

Astro alerts provide an additional level of observability to Airflow's notification systems. You can configure an alert to notify you in Slack, PagerDuty, or through email when a DAG completes, if you have a DAG run failure, or if a task duration exceeds a specified time. You can also define whether alerts apply to a specific Deployment or across an entire Workspace or Organization.

Unlike Airflow callbacks and SLAs, Astro alerts do not require changes to DAG code. Follow this guide to set up your Slack, PagerDuty, or email to receive alerts from Astro and then configure your Deployment to send alerts.

Alert types​

Each Astro alert has a notification channel and a trigger type. The notification channel determines the format and destination of an alert and the trigger type defines what triggers the alert.

DAG and task alerts​

You can trigger an alert to a notification channel using one of the following trigger types:

• DAG duration: The alert triggers when a specified DAG takes longer than expected to complete.
• DAG failure: The alert triggers whenever the specified DAG fails.
• DAG success: The alert triggers whenever the specified DAG completes.
• Task duration: The alert triggers when a specified task takes longer than expected to complete.
• Task failure: The alert triggers whenever the specified Task fails.
• DAG Timeliness: The alert triggers when a given DAG does not have a successful DAG run within a defined time window.

Deployment health alerts​

Deployment health alerts are customizable Astro Alerts that notify you about Deployment health incidents and suggest specific remediation actions. You can use these alerts to proactively monitor when Deployment health issues arise. For example, you can create an alert for when the Airflow metadata database storage is unusually high. All available Deployment health alerts are enabled by default for new Deployments and can be individually created, deleted, and edited in the Alerts tab of existing Deployments. If you do not want Deployment health alerts for a new Deployment, disable Deployment Health Alerts in the Advanced section.

Astro enables the following Deployment health alerts by default for new Deployments. You can individually create and customize these alerts for any Deployment:

• Airflow Database Storage Unusually High: The alert triggers when the metadata database has tables that are larger than 50GiB (Info) or 75GiB (Warning).
• Deprecated Runtime Version: The alert triggers when your Deployment is using a deprecated Astro Runtime version.
• Job Scheduling Disabled: The alert triggers when the Airflow scheduler is configured to prevent automatic scheduling of new tasks using DAG schedules.
• Worker Queue at Capacity: The alert triggers when at least one worker queue in this Deployment is running the maximum number of tasks and workers.

Deployment health alert notifications​

For new Deployments, Astro creates the default Deployment health alerts with Email as the notification channel Type. By default, these alerts notify the Contact Emails of the Deployment, specified in the Advanced section of the Deployment creation page.

If Contact Emails is empty, Astro displays the fallback email(s) that receives alert notifications for the Deployment in the Advanced section of Deployment configuration. To change this notification channel, ensure the Contact Emails field of your Deployment is not blank by editing Deployment Details.

Create an alert in the Astro UI​

Prerequisites​

• An Astro project.
• An Astro Deployment. Your Deployment must run Astro Runtime 7.1.0 or later to configure Astro alerts, and it must also have OpenLineage enabled.
• A configured notification channel.

Step 1: Create your alert​

1. In the Astro UI, click Alerting > Alerts.

2. Click Add Alert.

3. Choose the Alert Type

   • DAG duration: Send an alert if a specified DAG takes longer than expected to complete.

   • DAG failure: Send an alert if a DAG fails.

   • DAG success: Send an alert when a DAG completes.

   • Task duration: Enter the Duration for how long a task should take to run before you send an alert to your notification channels.

   • Task failure: Send an alert if a Task fails.

   • Timeliness: Select the Days of Week that the alert should observe, the Verification Time when it should look for a DAG success, and the Lookback Period for how long it should look back for a verification time. For example, if an alert has a Verification Time of 3:00 PM UTC and a Lookback Period of 60 minutes, it will trigger whenever the given DAG does not produce a successful DAG run from 2:00 to 3:00 PM UTC.

   warning

   Timeliness alerts only support Standard Time, as opposed to Daylight Saving Time. If you want Local Time support (for example, for time zone aware DAGs), you must adjust the alert's UTC time when the time changes from Standard Time to Daylight Saving Time or from Daylight Saving Time to Standard Time.

4. Choose the alert Severity, either Info, Warning, Critical, or Error.

Step 2: Add alert rules​

1. Define the Workspace and Deployment

2. For DAG and Task Alerts, you can further customize the circumstances in which the alert is triggered by defining:

   • Attribute
     • For DAG-level alerts: DAG ID
     • For Task-level alerts: You can define rules for both the DAG ID and Task ID attributes.
   • Operator
     • is one of: Select specific DAGs or tasks the alert should apply to. You can also choose All DAGs to apply the alert to all DAGs in the Deployment and All Tasks to apply the alert to all tasks in the DAG.
     • contains: Define a substring that is in the DAG ID and/or Task ID for the DAGs and/or tasks that the alert should apply to.
   • Values/Substring

   The following example is a task-level alert for all tasks with a Task ID that contains the substring prod in all DAGs within a Deployment:

   

Step 3: Add notification channels for the alert​

Select the Notification Channels for the Alert. You can also create a new notification channel by clicking +Notification Channel. See Notification Channels for information on configuring and adding Notification Channels.

Step 4: (Optional) Change an alert name​

After you select a DAG that you want to apply an alert to, Astro automatically generates a name for your alert. However, you can choose to change the name of your alert.

1. Expand the Change alert names... section.

2. Edit the Alert Name.

3. Click Create Alert to save your changes.

Step 5: (Optional) Test your DAG failure alert​

Astro alerts work whether your DAG run is manual or scheduled, so you can test your configured Astro alerts by failing your DAG manually.

1. In the Astro UI, click DAGs.

2. Choose the DAG that has your alert configured.

3. Trigger a DAG run.

4. Select Mark as and choose Failed to trigger an alert for a DAG failure.

5. Check your Slack, PagerDuty, or Email alerts for your DAG failure alert. The alert includes information about the DAG, Workspace, Deployment, and data lineage associated with the failure as well as direct links to the Astro UI.

   

Configure and add alert notification channels​

You can send Astro alerts, including Deployment Health Alerts, to the following notification channels

• Slack
• PagerDuty
• Email
• DAG trigger
• Opsgenie

Notification channels scope​

When you create a notification channel, you can define whether it is available to a specific Deployment or available to an entire Workspace or Organization. The type of scope you can use for your notification channels depends on your user permissions.

You can view, create, and manage notification channels for your Workspace or Organization in the Notification channels page or when creating a Deployment alert.

Step 1: Configure your notification channel​

{
  "dagName": "fail_dag",
  "alertType": "PIPELINE_FAILURE",
  "alertId": "d75e7517-88cc-4bab-b40f-660dd79df216",
  "message": "[Astro Alerts] Pipeline failure detected on DAG fail_dag. \\nStart time: 2023-11-17 17:32:54 UTC. \\nFailed at: 2023-11-17 17:40:10 UTC. \\nAlert notification time: 2023-11-17 17:40:10 UTC. \\nClick link to investigate in Astro UI: https://cloud.astronomer.io/clkya6zgv000401k8zafabcde/dags/clncyz42l6957401bvfuxn8zyxw/fail_dag/c6fbe201-a3f1-39ad-9c5c-817cbf99d123?utm_source=alert\"\\n"
}

import datetime
from typing import Any

from airflow.models.dag import DAG
from airflow.operators.python import PythonOperator

with DAG(
    dag_id="register_incident",
    start_date=datetime.datetime(2023, 1, 1),
    schedule=None,
):

    def _register_incident(params: dict[str, Any]):
        failed_dag = params["dagName"]
        print(f"Register an incident in my system for DAG {failed_dag}.")

    PythonOperator(task_id="register_incident", python_callable=_register_incident)

Step 2: Add a notification channel in the Astro UI​

You can add notification channels in the Astro UI.

1. In the Astro UI, click Alerting > Notification channels.

2. Click Add Notification Channel.

3. Enter a name for your notification channel.

4. Choose the Channel Type.

5. Add the notification channel information.

   • Slack
   • PagerDuty
   • Email
   • DAG Trigger

   Paste the Webhook URL from your Slack workspace app. If you need to find a URL for an app you've already created, go to your Slack Apps page, select your app, and then choose the Incoming Webhooks page.

6. Choose the scope by defining what you want to make Notification channel available to:

7. Click Create notification channel.

Notification History​

To view a historical log of all previously triggered Alerts for your Organization, click Notification History under the Alerting section of the navigation. You can filter the notification log based on:

• Alert: Which Alert triggered the notfication
• Time Range: The time range the notification was triggered in
• Notification Channel: The notification channel that is defined for the Alert
• Status: Whether the notification was successfully sent or failed

To view the notification history for a particular Alert, click Alerts in the Alerting section of the navigation. Here, you can see the number of times an Alert sent a notification (in the "Sent" column) or failed to send a notification (in the "Failed" column). Click on the number in the Sent column to see notification history for that Alert.