Datasets and Data-Aware Scheduling in Airflow

Datasets in Airflow are a powerful feature that allow you to define explicit dependencies between DAGs and the data they rely on. By leveraging datasets, you can establish clearer relationships between workflows, optimize scheduling for DAGs accessing the same data, and improve coordination across teams. This approach enhances pipeline efficiency, transparency, and overall data orchestration.

In this webinar, we’ll explore how to take full advantage of datasets and data-aware scheduling, including:

• Implementing datasets effectively, from pipeline design to advanced scheduling techniques
• Monitoring dataset-driven pipelines using the Airflow UI and Astro Observe
• Upcoming enhancements in Airflow 3.0, including updates to datasets and additional functionality