Execution mode

Overview​

Starting with Airflow 3.x Deployments on Astro, you can choose Remote Execution mode to securely run your tasks in your own environments, whether it is on-prem, hybrid, or edge, without opening inbound connections. Only essential scheduling and health data leave the execution plane.

You can still use the default Hosted execution mode for Astronomer-Hosted execution and orchestration for hands-off infrastructure management. Hosted execution mode on Astro is the quickest and easiest way to run data pipelines on Airflow.

Hosted execution​

Hosted execution mode is Astronomer-Hosted execution and orchestration for hands-off infrastructure management. Hosted is the default execution mode and supports dag versioning for Airflow 3 Deployments and autoscaling workers out of the box.

Hosted execution offers a convenient, secure, and stable way to run Airflow workloads without having to manage workers in your environment.

Remote Execution​

Remote Execution on Astro is built on a decoupled architecture that separates the orchestration layer from the execution layer so sensitive data never leaves local infrastructure, and your code remains in secure repositories.

• Orchestration Plane (Managed by Astronomer): The centralized control layer that contains the Scheduler, Web/API Server, metadata DB, and at it's core, the Remote Execution API. The Remote Execution API securely manages all communication with your Remote Execution Agents—assigning tasks, tracking agent health, and maintaining observability without accessing your data or code.

• Execution Plane (Customer-Controlled): Your infrastructure—on-prem or in the cloud—where tasks actually run. This is where your code, data, task logs, and secrets stay. Inside this environment are Remote Execution Agents, lightweight services that pull tasks from the Remote Execution API, run them locally, and report status and metrics back to Astro. The Remote Execution Agent includes a DAG Processor, Triggerer, and Worker(s). This means that DAG parsing, task execution (deferrable and non-deferrable), sourcing of secrets at runtime, and logging all happen in the execution plane. Remote Execution Agent can be configured for different workloads, offering full flexibility without compromising security, and you can deploy them on platforms and hardware of your choosing.