June 20, 2024

Exploring Airflow 2.9 Features with Astronomer's 29 Days of Airflow 2.9 Series

Radhika Gulati Product Marketing Manager Astronomer
Constance Martineau Senior Product Manager Astronomer

Last month, the Astronomer team led a comprehensive exploration of the features in Airflow 2.9 through our "29 Days of Airflow 2.9" social media posts. Each day, we highlighted a significant enhancement in Airflow 2.9, exploring new functionalities designed to streamline and improve data engineering workflows. Airflow 2.9 is packed with new features designed to make the platform more efficient, intuitive, and flexible.

As we close this series, let's take a comprehensive look at the exciting new features explored, and the big themes that reflect their impact on the Airflow ecosystem. For a deeper dive into some of the features below, check out our blog post: Introducing Airflow 2.9.

1. Dataset & Data-Aware Scheduling Revamp

Enhanced dataset handling and scheduling features make workflows more dynamic and data-driven. New functionalities include manual triggers for dataset updates, standardized dataset URI formats for better interoperability, and advanced scheduling tools like conditional logic for dataset triggering. These enhancements ensure precise control and deepened data context, elevating the capabilities of data-aware scheduling in complex environments.

Day 3 & Day 7: Manual triggers for dataset updates and advanced dataset condition specifications for precise control.
Day 10: Standardization of dataset URI formats for better interoperability.
Day 20: Addition of a POST endpoint for dataset events, allowing programmable triggers.
Day 26, Part 1 & Part 2: Introduction of conditional logic for dataset triggering and management of dataset-triggered queues.
Day 27, Part 1 & Part 2: Features such as DatasetOrTimeSchedule and logical operators for dataset conditions.
Day 29, Part 2 & Part 3: Introduction of new metrics with tagging for detailed performance tracking and dataset_expression in grid DAG details for deeper data context.

2. Dynamic Task Mapping Improvements

Focused on improving task flexibility and data flow within workflows, this theme introduces support for multiple XCom outputs in the BaseOperator, expanding data exchange capabilities. Additional features such as mutable display names and custom instance names for mapped tasks in the UI enhance clarity and adaptability in task management, streamlining communication and workflow customization.

Day 12: Support for multiple XCom outputs in the BaseOperator, facilitating more flexible and robust data exchange between tasks.
Day 21: Display of custom instance names for mapped tasks in the UI, enhancing clarity for complex workflows.

3. UI Enhancements

Aiming to streamline user interaction and increase intuitiveness, significant UI improvements include the integration of Matomo for robust analytics and enhanced grid views for better data visibility. CLI support for bulk operations like pausing and resuming DAGs further boosts operational efficiency, making the Airflow interface more responsive and easier to manage.

Day 4: Integration of Matomo as an analytics tool for robust data insights.
Day 8, Part 1 & Part 2: Enhancements in grid view for dataset events and task log grouping for improved data visibility and simplified log analysis.
Day 11 & Day 17: Visualizing task failures and custom task prioritization rules, personalizing UI elements, and automatic configuration setups for adaptable environments.
Day 22: Mutable display names for DAGs and tasks, adding flexibility and clear communication.
Day 28, Part 1: Integration of run duration tracking in React for a responsive monitoring interface.

4. Workflow Optimization: Flexible Customization + Improved Efficiency

This theme delivers tools for tailored workflow management and enhanced operational efficiency. It includes real-time task duration calculations, customizable cron timetables, and the new @task.bash TaskFlow decorator for simplified task creation. Enhanced configuration options for DAG processor outputs and task prioritization rules support adaptable environments, fostering better governance and data management. Additionally, CLI enhancements for bulk operations like pausing and resuming DAGs complement these features by providing robust control mechanisms for managing workflows efficiently from a command line perspective, further optimizing workflow management.

Day 1 & Day 2: Real-time task duration calculations and visualization of average duration marks for dynamic monitoring.
Day 5 & Day 19: Custom task prioritization rules and automatic configuration setups for adaptable environments.
Day 6: Customizing the default cron timetable for tailored scheduling.
Day 9: Addition of color formatting for ANSI characters in logs to improve readability.
Day 25: Making datasets hashable for better workflow management and debugging.
Day 13, Day 14, & Day 29, Part 1: Enhancements in governance, compliance, and data management across distributed environments.
Day 15: Extending XCom capabilities with an object storage backend, supporting larger data volumes and complex data types.
Day 16: Addition of an on_skipped_callback function to the BaseOperator for enhanced task lifecycle management.
Day 18: Introduction of the @task.bash TaskFlow decorator for simplified task creation using Bash commands.
Day 23 & Day 24: New configuration variables for controlling DAG processor outputs and improving log management.
Day 28, Part 2 & Part 3: CLI support for bulk pause and resume of DAGs and showing abandoned tasks in Grid View for enhanced workflow management.

Experience Airflow 2.9

We are excited to see how these new capabilities will be leveraged to drive efficiency and innovation in data engineering projects worldwide. As always, Astronomer remains committed to supporting and contributing to the growth of Apache Airflow® and its vibrant community.

Thank you for joining us in this campaign, and we look forward to continuing to support your efforts in harnessing the full potential of Airflow 2.9 and beyond!

Curious to see how these Airflow 2.9 enhancements can streamline your data workflows? Try them out on Astro, where we provide same-day support for every new version of Airflow. Start your free 14-day trial of Astro today and explore how our platform makes it easier to adopt and leverage the latest Airflow features for your data engineering needs.