Skip to main content
Use this document to configure resource usage for a Deployment’s executor, webserver/apiserver, scheduler and triggerer components.

​
Select an executor

The Airflow executor works closely with the Airflow scheduler to determine what resources complete tasks as they queue. The main difference between executors is their available resources and how they utilize those resources to distribute work. Astro Private Cloud supports three executors: Though it largely depends on your use case, Astronomer recommends the Local executor for development environments and the Celery or Kubernetes executors for production environments operating at scale. For a detailed description of each executor, see Airflow executors explained.

​
Scale core resources

Apache Airflow requires four primary components:
  • The Webserver or API server (Airflow 3)
  • The Scheduler
  • The Executor (and the workers it runs)
  • The Triggerer
To scale these resources, adjust the corresponding slider in the Astro Private Cloud UI to increase its available resources. Read the following sections to help you determine which core resources to scale and when.

​
Airflow Webserver and API server

In Airflow 2, the webserver component renders the Airflow UI, providing access to Dag monitoring, task logs, and configuration. Starting with Airflow 3, the apiserver replaces the webserver for serving the Airflow UI. If loading pages or functions in the Airflow UI is slow or unresponsive, increase the resources allocated to the webserver (Airflow 2) or apiserver (Airflow 3).

​
Scheduler

The Airflow scheduler is responsible for monitoring task execution and triggering downstream tasks once dependencies have been met. If you experience delays in task execution, which you can track via the Gantt Chart view of the Airflow UI, Astronomer recommends increasing the resources allocated towards the scheduler.
To set alerts that notify you via email when your Airflow scheduler is underprovisioned, configure an Airflow alert.

​
Scheduler count

Airflow 2.0 comes with the ability for users to run multiple schedulers concurrently to ensure high-availability, zero recovery time, and faster performance. You can provision up to 4 schedulers on any Deployment. Each individual scheduler will be provisioned with the resources specified in Scheduler Resources. For example, if you set the CPU figure in Scheduler Resources to 5 CPUs and set Scheduler Count to 2, your Airflow Deployment will run with 2 Airflow schedulers using 5 CPUs each for a total of 10 CPUs. To increase the speed at which tasks are scheduled and ensure high-availability, Astronomer recommends provisioning 2 or more Airflow schedulers for production environments.

​
Dag Processor

Complex, dynamically-generated Dags, sub-optimal Dag parsing practices, or a growing business that requires a larger data pipeline can strain Dag processing and threaten your Airflow scheduler’s availability. Deployments can support high-scale environments more reliably by separating the Dag processor from the scheduler. You can now configure the number of Dag processors for the Deployment from the UI and the Houston API. If you want to enable and provision resources for standalone Dag processors, you can set the dagProcessorEnabled feature flag to true in your Houston API configuration in the config.yaml file:
By adjusting the Dag Processor slider in the Astro Private Cloud UI, you can provision up to 3 Dag Processors on any Deployment. You can also configure CPU and Memory with the Dag Processor Resources sliders.
Airflow 3 requires the Dag Processor component to be enabled. Set dagProcessorEnabled: true in your Deployment configuration. If this flag is disabled, Airflow 3 Deployments will fail to start or process Dags correctly.

​
Triggerer

Airflow 2.2 introduces the triggerer, which is a component for running tasks with deferrable operators. Like the scheduler, the triggerer is highly-available: If a triggerer shuts down unexpectedly, the tasks it was deferring can be recovered and moved to another triggerer. By adjusting the Triggerer slider in the Astro Private Cloud UI, you can provision up to 2 triggerers on any Deployment running Airflow 2.2+. To take advantage of the Triggerer’s high availability, we recommend provisioning 2 triggerers for production Deployments.

​
(Kubernetes executor only) Set extra capacity

On Astronomer, resources required for the KubernetesPodOperator or the Kubernetes Executor are set as Extra Capacity. The Kubernetes executor and KubernetesPodOperator each spin up an individual Kubernetes pod for each task that needs to be executed, then spin down the pod after that task is completed. The amount of CPU and Memory allocated to Extra Capacity maps to resource quotas on the Kubernetes Namespace in which your Airflow Deployment lives on Astro Private Cloud. More specifically, Extra Capacity represents the maximum possible resources that could be provisioned to a pod at any given time. Resources allocated to Extra Capacity do not affect scheduler or webserver/apiserver performance and do not represent actual usage.

​
(Celery executor only) Configure workers

To optimize for flexibility and availability, the Celery executor works with a set of independent Celery workers across that it can delegate tasks. On Astro Private Cloud, you can configure your Celery workers to fit your use case.

​
Worker count

By adjusting the Worker Count slider, users can provision up to 20 Celery workers on any Airflow Deployment. If you would like users to have the ability to set more than 20 workers on any Deployment, you can change this limit by applying a config change. Each individual worker will be provisioned with the resources specified in Worker Resources. If you set the CPU figure in Worker Resources to 5 CPUs and set Worker Count to 3, for example, your Airflow Deployment will run with 3 Celery workers using 5 CPUs each for a total of 15 CPUs.

​
Worker termination grace period

On Astro Private Cloud, Celery workers restart after every code deploy to your Airflow Deployment. This makes sure that workers execute with the most up-to-date code. To minimize disruption during task execution, however, APC supports the ability to set a Worker Termination Grace Period. If a deploy is triggered while a Celery worker is executing a task and Worker Termination Grace Period is set, the worker will continue to process that task up to a certain number of minutes before restarting itself. By default, the grace period is ten minutes.