Skip to main content
Event-driven scheduling is a sub-type of data-aware scheduling where a DAG is triggered when messages are posted to a message queue. This is useful for scenarios where you want to trigger a DAG based on events that occur outside of Airflow, such as data delivery to an external system or IoT sensor events, and is key to inference execution pipelines. In this guide you will learn about the concepts used in event-driven scheduling, common usage patterns, and how to implement an example using Amazon SQS and an example using Apache Kafka.

Build Event-Driven Workflows with Apache Airflow®: On-Demand Webinar

Watch the on-demand webinar on building and operating event-driven pipelines with the latest Airflow features.

Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

Concepts

There are a number of concepts that are important to understand when using event-driven scheduling.
  • Data-aware scheduling: Data-aware or data-driven scheduling refers to all ways you can schedule a DAG based on updates to assets. Updates to assets outside of event-driven scheduling occur by tasks in the same Airflow instance completing successfully, manually through the Airflow UI, or through a call to the Airflow REST API. See Assets and data-aware scheduling in Airflow.
  • Event-driven scheduling: Event-driven scheduling is a sub-type of data-aware scheduling where a DAG is run based on messages posted to a message queue. This message in the queue is triggered by an event that occurs outside of Airflow.
  • Message queue: A message queue is a service that allows you to send and receive messages between different systems. Examples of message queues include Amazon SQS, RabbitMQ, and Apache Kafka. Event-driven scheduling in Airflow 3.0 is supported for Amazon SQS with support for other message queues planned for future releases.
  • Trigger: A trigger is an asynchronous Python function running in the Airflow triggerer component. Triggers that inherit from BaseEventTrigger can be used in AssetWatchers for event-driven scheduling. The trigger is responsible for polling the message queue for new messages, when a new message is found, a TriggerEvent is created. The message is deleted from the queue.
  • AssetWatcher: An AssetWatcher is a class in Airflow that watches one or more triggers for events. When a trigger fires a TriggerEvent, the AssetWatcher updates the asset it is associated with, creating an AssetEvent. The payload of the trigger is attached to the AssetEvent in its extra dictionary.
  • AssetEvent: An Asset is an object in Airflow that represents a concrete or abstract data entity. For example, an asset can be a file, table in a database, or not tied to any specific data. An AssetEvent represents one update to an asset. In the context of event-driven scheduling, the AssetEvent represents one message having been detected in the message queue.
In Airflow versions before 3.4, asset events that occurred in quick succession were sometimes batched into one Dag run. In Airflow 3.4 and later, each AssetEvent creates its own Dag run by default, so a Dag scheduled on an asset with an AssetWatcher runs once per AssetEvent. To explicitly control batching behavior, see Batch asset events.

When to use event-driven scheduling

Basic and advanced data-aware scheduling are great for use cases where updates to assets occur within Airflow or can be accomplished through a call to the Airflow REST API. However, there are scenarios where you need a DAG to run based on events in external systems. Two common patterns exist for event-driven scheduling:
  1. Data delivery to an external system: Data is delivered to an external system, such as manually by a domain expert, and a data-ready event is sent to a message queue. The DAG in Airflow is scheduled based on this message event and runs an extract-transform-load (ETL) pipeline that processes the data in the external system.
  2. IoT sensor events: An Internet of Things (IoT) device sends a sensor event to a message queue. A DAG in Airflow is scheduled based on this message event and consumes the message to evaluate the sensor value. If the evaluation determines that an alert is warranted, an alert event is published to another message queue.
Two patterns using event-driven scheduling.
One common use case for event-driven scheduling is inference execution, where the DAG that is triggered involves a call to a machine learning model. Airflow can be used to orchestrate inference execution pipelines of all types, including in Generative AI applications. A key change enabling inference execution in Airflow 3.0 is that a DAG can be triggered with None provided as the logical_date, meaning simultaneous triggering of multiple DAG runs is possible.

Example: Amazon SQS

This example shows how to configure a DAG to run as soon as a message is posted to an Amazon SQS queue.
  1. Create an Amazon SQS queue. See Amazon Simple Queue Service Documentation for instructions.
  2. Add the Airflow Common Messaging provider and Airflow Amazon provider to your Airflow instance. When using the Astro CLI, you can add the providers to your requirements.txt file:
  3. Set the connection to your Amazon SQS queue in your Airflow instance. Note that the connection needs to include the region_name in the extra field. Replace <your access key>, <your secret key>, and <your region> with your AWS credentials and region. For other authentication options see the Airflow Amazon provider documentation.
    Your AWS user requires at least sqs:ReceiveMessage and sqs:DeleteMessage permissions.
  4. Create a new file in the dags folder of your Airflow project and add the following code. Replace the SQS_QUEUE with the URL to your message queue.
    This DAG is scheduled to run as soon as the sqs_queue_asset asset is updated. This asset uses one AssetWatcher with the name sqs_watcher that watches one MessageQueueTrigger. This trigger is polling for new messages in the provided SQS queue. The process_message task gets the triggering asset events from the Airflow context and prints the message body from the triggering message. The process_message task is a placeholder for your own task that processes the message.
  5. Create a new message in the SQS queue. It will trigger the DAG to run and the process_message task will print the message body.

Example: Apache Kafka

To use Apache Kafka as the message queue for event-driven scheduling, follow these steps:
  1. Create an Apache Kafka topic. See the Apache Kafka documentation for instructions.
  2. Add the Airflow Common Messaging provider and Airflow Apache Kafka provider to your Airflow instance. When using the Astro CLI, you can add the providers to your requirements.txt file:
  3. Set the connection to your Apache Kafka topic in your Airflow instance. Below is an example connection JSON, depending on your Kafka setup you may need to provide additional values in the extra dictionary.
  4. Create a new file in the include folder of your Airflow project called kafka_trigger.py and add the following code. This function will be applied to the Kafka message as soon as it is received and return the value of the message to be processed by the Dag.
  5. Create a new file in the dags folder of your Airflow project and add the following code. Replace the placeholders in the KAFKA_QUEUE with your information.
    This DAG is scheduled to run as soon as the kafka_topic_asset asset is updated. This asset uses one AssetWatcher with the name kafka_watcher that watches one MessageQueueTrigger. This trigger is polling for new messages in the provided Kafka topic. The process_message task gets the triggering asset events from the Airflow context and prints the event information from the trigger event that includes the message. The process_message task is a placeholder for your own task that processes the message.
  6. Create a new message in the Kafka topic. It will trigger the DAG to run and the process_message task will print the event information.

Write a custom event trigger

If no existing trigger supports your event source, you can write your own by subclassing BaseEventTrigger. A custom event trigger needs to:
  • Implement serialize(), which returns the import path of the trigger class and the keyword arguments needed to re-create it. The triggerer calls this method to store the trigger in the metadata database.
  • Implement run(), an async generator that yields a TriggerEvent when it detects an event. Airflow turns each TriggerEvent into an AssetEvent, and the payload becomes the extra of that event.
In Airflow 3.3 and later, the triggerer sets self.asset_state_store before it calls run(). This attribute is an accessor for the asset state store of the asset the trigger watches. Use it to persist a watermark, such as the timestamp of the last record the trigger processed, so the trigger doesn’t fire again for the same data each time it runs.
self.asset_state_store is None in __init__ and serialize(), and is only available inside run(). Plain BaseTrigger subclasses, which are used to defer tasks, don’t have access to the asset state store.

Example: Watch a SQL table for new records

The following trigger polls a SQL table for rows with a timestamp later than a stored watermark. When it finds new rows, it fires one TriggerEvent per row and moves the watermark to the latest timestamp it saw.

Prerequisites

The following is required for this example:
  • Airflow 3.3 or later.
  • The Airflow Common SQL provider and the provider for your database installed in your Airflow environment.
  • An Airflow connection to your database, with a table that has a timestamp column that increases as new records arrive.

Define the trigger

Create a file in the include folder of your Airflow project called sql_triggers.py and add the following code:
This trigger does the following:
  • serialize() returns only the arguments needed to rebuild the trigger. It doesn’t include state such as the watermark, which lives in the asset state store.
  • run() reads the watermark key with aget. On the first poll the key doesn’t exist, so the trigger selects every row in the table.
  • The query runs in a worker thread with asyncio.to_thread because the hook is synchronous. All triggers on a triggerer share one event loop, so blocking calls delay every other trigger.
  • After the trigger yields events, it stores the latest timestamp with aset and returns. After the asset event is created, the triggerer starts the trigger again, and the next poll begins at the new watermark.
  • run() uses the async methods aget and aset. The synchronous get and set methods also work, but they block the event loop for the full round trip to the API server.

Use the trigger in a Dag

Create an asset with an AssetWatcher that uses the trigger, then schedule a Dag on the asset:
The trigger fires one event for each new row. In Airflow 3.4 and later, each event starts its own Dag run. The process_order task reads the row from the payload key of the AssetEvent extra. To read or reset the watermark, open the asset in the Airflow UI and go to the Asset State Store tab. Deleting the watermark entry makes the trigger process every row in the table again on its next poll.