Skip to main content

Airflow 3 features

Remote Execution on Astro​

Remote Execution in Airflow 3 allows tasks to run securely in customer-managed environments without opening inbound connections. Only essential scheduling and health data leave the execution plane.

Remote Execution Agents on Astro enable tasks to run in user-managed hardware or private clouds with only outbound connections to Astro’s Orchestration Plane. Sensitive data stays local, ideal for regulated or multi-regional deployments.

Learn more about Execution modes on Astro.

DAG Versioning​

Airflow 3’s Dag Versioning ensures that each pipeline run references its exact code snapshot, enabling complete historical traceability. Teams can rapidly audit, compare with and debug by viewing the historic DAG code/structure of each run, eliminating confusion and accelerating compliance checks.

Backfills​

Backfills solve one of the most common and time-consuming challenges in data orchestration: reliably reprocessing historical or newly available data. Previously, backfills in Airflow had to be triggered from a command-line process that could easily terminate if the session was lost, leaving longer reruns vulnerable to interruption and without robust monitoring. In Airflow 3, backfills become first-class citizens managed by the scheduler itself, enabling asynchronous API triggers, real-time monitoring through the UI, and the ability to pause or cancel jobs mid-run. This unified approach not only saves teams from manual scripting and fragile workarounds, but it also gives them confidence that large-scale historical recalculations—often critical for machine learning retraining and data integrity checks—will run consistently, even if they take hours or days to complete.

UI Modernization​

Airflow 3 introduces a modern, React-based UI that unifies logs, task details, and dynamic DAG updates in a clean, intuitive interface.

Event-driven Scheduling​

Event-driven scheduling in Airflow 3 lets pipelines react to near real-time data changes or external triggers, rather than relying solely on fixed time-based schedules. This means a DAG can automatically start running as soon as a file appears in a storage bucket, a table is updated in a data warehouse, or another system sends an event signal. By removing the need for constant polling or hard-coded cron schedules, event-driven pipelines can process data the instant it arrives. This not only saves resources and shortens end-to-end processing time, but also enables more dynamic, near–real-time workflows that are crucial for modern data science, streaming analytics, and AI/ML applications.

Inference Execution​

Airflow 3.0 introduces several enhancements to support AI Inference Execution:

  • Ad-hoc scheduling: Airflow 3.0 allows DAGs to be run independently of any data interval, which is crucial for supporting inference execution. This feature enables on-demand execution of inference tasks without being constrained by predefined schedules.
  • Synchronous DAG execution: The new version supports simultaneous execution of the same DAG, allowing for synchronous inference runs. This is particularly useful for scenarios where multiple inference requests need to be processed concurrently.
  • API-triggered execution: Airflow 3.0 introduces the ability to trigger DAGs via API calls, enabling multiple instances to be initiated simultaneously for inference tasks. This feature facilitates experimentation and allows for dynamic, near real-time inference processing.
  • Event-driven scheduling: The new version supports automatic triggering of DAGs based on external events or data availability. This can be particularly useful for inference pipelines that need to react to new data or model updates in near real-time.
  • Language-agnostic Task Execution Interface: Airflow 3.x lays the groundwork to run tasks in any language. This enables users to implement inference tasks in the most suitable language for their models, without expensive code refactoring such as using C++, Golang, Java, etc. for more efficient execution

Collectively, these enhancements make Airflow 3 more capable of handling diverse inference scenarios, from batch processing to on-demand execution, while offering improved flexibility and performance for AI and ML workflows.

Was this page helpful?