Set up Astro alerts

​

Alert types

​

Dag and task alerts

• DAG Failure: The alert triggers whenever the specified Dag fails. From the alert notification, you can open the failed Dag run in the Astro UI to start an Otto investigation. To investigate Dag failures automatically when this alert fires, send it to a Dag Trigger notification channel that calls the Otto investigation API. See Investigate with Otto.
• DAG Success: The alert triggers whenever the specified Dag completes.
• DAG Timeliness: The alert triggers whenever the specified Dag doesn’t produce a successful Dag run by the given Verification Time including the Look Back Period.
• DAG Duration: The alert triggers whenever the specified Dag doesn’t produce a successful Dag run within the given Duration. If the Dag fails before the specified duration is exceeded, the alert is triggered.
• Task Failure: The alert triggers whenever the specified task fails.
• Task Duration: The alert triggers when a specified task takes longer than expected to complete successfully. If the task fails, the alert is triggered even if it fails before the specified duration is exceeded.

​

Deployment health alerts

• Airflow DB Storage Unusually High: The alert triggers when the metadata database has tables that are larger than 50GiB (Info) or 75GiB (Warning).
• Deprecated Runtime Version: The alert triggers when your Deployment is using a deprecated Astro Runtime version.
• Job Scheduling Disabled: The alert triggers when the Airflow scheduler is configured to prevent automatic scheduling of new tasks using Dag schedules.
• Worker Queue at Capacity: The alert triggers when at least one worker queue in this Deployment is running the maximum number of tasks and workers.

​

Deployment health alert notifications

​

Create an alert in the Astro UI

​

Prerequisites

• An Astro project.
• An Astro Deployment. Your Deployment must run Astro Runtime 7.1.0 or later to configure Astro alerts and it must have OpenLineage enabled.
• A configured notification channel.

​

Step 1: Create your alert

​

Step 2: Add alert rules

1. Define the Workspace and Deployment

2. For Dag and Task Alerts, you can further customize the circumstances in which the alert is triggered by defining:
   • Attribute
     • For Dag-level alerts: DAG ID
     • For Task-level alerts: You can define rules for both the DAG ID and Task ID attributes.
   • Operator
     • is one of: Select specific Dags or tasks the alert should apply to. You can also choose All dags to apply the alert to all Dags in the Deployment and All Tasks to apply the alert to all tasks in the Dag.
     • contains: Define a substring that is in the DAG ID and/or Task ID for the Dags and/or tasks that the alert should apply to.
   • Values/Substring The following example is a task-level alert for all tasks with a Task ID that contains the substring prod in all Dags within a Deployment:
     

​

Step 3: Add notification channels for the alert

​

Step 4: (Optional) Change an alert name

1. Expand the Change alert names… section.
2. Edit the Alert Name.
3. Click Create Alert to save your changes.

​

Step 5: (Optional) Test your Dag failure alert

1. In the Astro UI, click DAGs.
2. Choose the Dag that has your alert configured.
3. Trigger a Dag run.
4. Select Mark as and choose Failed to trigger an alert for a Dag failure.
   
5. Check your Slack, PagerDuty, or Email alerts for your Dag failure alert. The alert includes information about the Dag, Workspace, Deployment, and data lineage associated with the failure as well as direct links to the Astro UI.
   

​

Configure and add alert notification channels

• Slack
• PagerDuty
• Email
• Dag trigger
• Opsgenie

​

Notification channels scope

​

Step 1: Configure your notification channel

• Slack
• PagerDuty
• Email
• Dag Trigger

To set up alerts in Slack, you need to create a Slack app in your Slack workspace. After you’ve created your app, you can generate a webhook URL in Slack where Astro will send alerts.

1. Go to Slack API: Applications to create a new app in your organization’s Slack workspace.
2. Click From scratch when prompted to choose how you want to create your app.
3. Enter a name for your app, like astro-alerts, choose the Slack workspace where you want Astro to send your alerts, and then click Create App.

If you don’t have permission to install apps into your Slack workspace, you can still create the app, but you will need to request that an administrator from your team completes the installation.

4. Select Incoming webhooks.
5. On the Incoming webhooks page, click the toggle to turn on Activate Incoming Webhooks. See Sending messages using Incoming Webhooks.
6. In the Webhook URLs for your Workspace section, click Add new Webhook to Workspace.

If you don’t have permission to install apps in your Slack workspace, click Request to Add New Webhook to send a request to your organization administrator.

7. Choose the channel where you want to send your Astro alerts and click Allow.
8. After your webhook is created, copy the webhook URL from the new entry in the Webhook URLs for your Workspace table.

To set up an alert integration with PagerDuty, you need access to your organization’s PagerDuty Service. PagerDuty uses the Events API v2 to create a new integration that connects your Service with Astro.

1. Open your PagerDuty service and click the Integrations tab.

2. Click Add an integration.
3. Select Events API v2 as the Integration Type.
4. On your Integrations page, open your new integration and enter an Integration Name.
5. Copy the Integration Key for your new Astro alert integration.

No external configuration is required for the email integration. Astronomer recommends allowlisting astronomer.io with your email provider to ensure that no alerts go to your spam folder. Alerts are sent from postmaster@astronomer.io.

The Dag Trigger notification channel works differently from other notification channel types. Instead of sending a pre-formatted alert message, Astro makes a generic request through the DagRuns endpoint of the Airflow REST API to trigger any Dag in your Workspace. You can configure the triggered Dag to complete any action, such as sending a message to your own incident management system or writing data about an incident to a table.The alert payload includes the following parameters:

• conf: This parameter holds the alert payload:
• alertId: A unique alert ID.
• alertType: The type of alert triggered.
• dagName: The name of the Dag that the alert condition triggers.
• message: The detailed message with the cause of the alert.
• airflowDagRunId: The Airflow run ID of the Dag run that caused the alert. For example, scheduled__2023-11-17T17:30:00+00:00. This parameter is omitted when no Dag run is associated with the alert.
• note: By default, this is Triggering dag on Airflow <url>.

The following is an example alert payload that would be passed through the API to trigger the alert_dag if fail_dag fails:

{
  "dagName": "alert_dag",
  "alertType": "PIPELINE_FAILURE",
  "alertId": "d75e7517-88cc-4bab-b40f-660dd79df216",
  "airflowDagRunId": "scheduled__2023-11-17T17:30:00+00:00",
  "message": "[Astro Alerts] Pipeline failure detected on dag fail_dag. \\nStart time: 2023-11-17 17:32:54 UTC. \\nFailed at: 2023-11-17 17:40:10 UTC. \\nAlert notification time: 2023-11-17 17:40:10 UTC. \\nClick link to investigate in Astro UI: https://cloud.astronomer.io/clkya6zgv000401k8zafabcde/dags/clncyz42l6957401bvfuxn8zyxw/fail_dag/c6fbe201-a3f1-39ad-9c5c-817cbf99d123?utm_source=alert\"\\n"
}

These parameters are accessible in the triggered Dag using Dag params.

1. Create a Dag that you want to run when the alert is triggered. For example, you can use the following Dag to run arbitrary Python code when the alert is triggered:

from datetime import datetime
from typing import Any

from airflow.models.dag import DAG
from airflow.operators.python import PythonOperator

with DAG(
    dag_id="register_incident",
    start_date=datetime(2023, 1, 1),
    schedule=None,
):

    def _register_incident(params: dict[str, Any]):
        failed_dag = params["dagName"]
        print(f"Register an incident in my system for DAG {failed_dag}.")

    PythonOperator(task_id="register_incident", python_callable=_register_incident)

2. Deploy the Dag to any Deployment in the Workspace where you want to create the alert. The Dag that triggers the alert and the Dag that the alert runs can be in different Deployments, but they must be deployed in the same Workspace.
3. Create a Deployment API token for the Deployment where you deployed the Dag that the alert will run. Copy the token to use in the next step.

​

Step 2: Add a notification channel in the Astro UI

1. In the Astro UI, click Alerting > Notification Channels.
2. Click New Notification Channel.
3. Enter a name for your notification channel.
4. Choose the Channel Type.
5. Add the notification channel information.

• Slack
• PagerDuty
• Email
• Dag Trigger

Paste the Webhook URL from your Slack workspace app. If you need to find a URL for an app you’ve already created, go to your Slack Apps page, select your app, and then choose the Incoming Webhooks page.

Paste the Integration Key from your PagerDuty Integration and select the Severity of the alert.

Enter the email addresses that should receive the alert.

Select the Deployment where your Dag is deployed, then select the Dag. Enter the Deployment API token that you created in Step 1.

6. Choose the scope by defining what you want to make Notification channel available to:
7. Click Create notification channel.

​

Notification History

• Alert: Which Alert triggered the notification
• Time Range: The time range the notification was triggered in
• Notification Channel: The notification channel that is defined for the Alert
• Status: Whether the notification was successfully sent or failed