Apache Airflow® 3 Development Update

We’ve talked about Apache Airflow 3.0® being a huge milestone for the Airflow project before—most notably at the Airflow Summit in September 2024. Now, I’m thrilled to announce the release of the first beta build of Airflow 3.0! This is the result of a massive effort from the global Airflow community, and I’m excited to share more details about this milestone.

A Major Release, Four Years in the Making

Airflow 3.0 is the biggest release in Airflow’s history—2.0 was released in 2020, and the last 4 years have seen incremental updates and releases every quarter with version 2.10 released in Q4 2024. With over 30 million monthly downloads (up over 30x since 2020) and 77,000 organizations (up from 25,000 in 2020) now using Airflow, we’ve seen an incredible growth in popularity since 2.0. 3.0 is the first major release in over four years and we’re targeting release candidates for March 2025, with general availability (GA) expected by mid-April 2025. This beta marks a crucial step in that journey, and we’re eager for feedback from the developer community.

Highly requested UX features

DAG Versioning

One of the most highly requested features in Airflow has been DAG versioning. This was broken out into two AIPs, which are part of the beta release.

1. Improve DAG history (AIP-65)

2. DAG Bundles and Parsing (AIP-66)

This includes improved DAG history, available through the UI, and is fully feature-complete from a server perspective. DAG bundling and parsing are also finalized, allowing users to download bundles and retrieve DAGs from Git, NFS, or other sources. All of this is part of the beta release. The UI work is partially completed in the beta and will be fleshed out before GA.

Backfills improvements

3. Another long-standing user request has been better support for backfills. While often discussed in the context of machine learning, backfills also apply to traditional ETL and ELT use cases. This was defined as part of “Scheduler-managed backfills” (AIP-78). In the beta release, backfills are now integrated with the scheduler for improved scalability and diagnostics.

Run anywhere, at any time, in any Language

Run anywhere, in any language

4. A foundational goal of Airflow 3 is allowing execution in any environment, in any language. A key component of this is the Task Execution Interface (AIP-72), enabling the evolution of Airflow into a client-server architecture, which represents one of the most significant architectural shifts in Airflow’s history. This foundational feature enables multi-cloud deployments and multi-language support. This feature is part of the beta release and supports Celery, Kubernetes, and Local Executors.

5. The Edge Executor (AIP-69) is an incremental feature built on top of the Task Execution Interface work. Initial incarnations have been released in experimental mode based on Airflow 2x and this executor will evolve with the Task Execution Interface. Significant progress has already been made on this including using the new Airflow 3 API server.

Data Assets and Event driven scheduling

The evolution of Data awareness support in Airflow from Datasets into Data Assets was broken out into several AIPs as detailed below. These are all part of the beta release.

6. The fundamental evolution of Datasets into Data Assets has been done as part of “Introducing Data Assets” (AIP-74).

7. A significant enhancement around Data Assets is the New Asset-Centric Syntax (AIP-75) for defining Assets easily with DAGs using the Python decorator syntax, which is part of this release.

8. External event driven scheduling (AIP-82)

External event driven scheduling is based on the foundation Data Assets work described above. The initial scope as defined in the AIP is complete, and we are extending this with more integration capabilities as part of an upcoming beta release. More details to follow here.

Inference execution and hyper-parameter tuning

9. Remove Execution date unique constraint from DAG run (AIP-83)

Removing the requirement for execution date to be unique for each DAG run, enables Airflow to directly support a variety of new uses such as Model training and Gen AI inference without requiring hacks and workarounds by ML and AI engineers using Airflow. This was much more difficult than initially expected and took a couple of retries, but is now complete and is part of this release.

Security and usability improvements

UI Modernization

Modernizing the entire Airflow UI to be based on React and the FastAPI is a huge project and is broken out into several AIPs as detailed below.

10. Internal API for UI Operations (AIP-84)

The foundation for the new UI is the REST API and a set of internal APIs both of which are now based on FastAPI. This has had tremendous community contributions and is part of the beta release. While this API is close to completion, we anticipate further refinements as the UI evolves.

11. Modern Web Application (AIP-38)

The UI in Airflow 3.0 has been significantly improved, with several UX enhancements making workflows smoother. Below, we’ve included some screenshots to showcase the improvements—feedback on these changes is especially welcome.

12. Remove Flask AppBuilder as a Core dependency

Getting rid of Flask AppBuilder as a core dependency has been desirable from a security and maintenance perspective, but also problematic from a backwards compatibility perspective. FAB has been moved into a provider package, so that it can be used for backwards compatibility specifically for Plugins and significantly de-risks the Airflow 3.0 release.

Security

Security improvements have also been a major focus. Task isolation implemented as part of the client-server evolution explained in the previous section has been frequently requested by Airflow users and is part of the beta release.

Additionally, the introduction of the new internal API for UI operations forms the basis for a centralized more secure pattern.

13. Enhanced security in CLI via Integration of API (AIP-81)

Improving the CLI and reducing the maintenance burden by having the CLI use the Airflow APIs, rather than direct access is an important evolution for Airflow. This is making good progress leveraging the foundational work being done on the API above.

Community Contributions and Next Steps

The level of community engagement around Airflow 3.0 has been truly inspiring. Over 100 developers have contributed each month, demonstrating incredible commitment to pushing this release forward. While there is still work to be done—such as packaging enhancements and expanding multi-language support—we’re excited to share this beta and gather feedback from Airflow developers.

This beta release should not be used in production environments just yet and is intended for testing by Airflow developers. We are looking forward to sharing more updates as we approach shipping release candidates (RCs) and GA in the coming months.

Thank you for being part of this journey—please share your feedback so we can continue improving and delivering a high-quality release!

You can see the DAG version on the right hand side edge!