Skip to main content
Astronomer Units (AUs) RemovedAstronomer Units (AUs) are no longer supported. All Deployments must now specify CPU and memory directly when configuring resources. This change provides more clarity, flexibility, and aligns with Kubernetes-native resource management.
When you create a new Astro Private Cloud (APC) Deployment, you now specify the exact amount of CPU and memory that power its core components. For example:
  • You might need to allocate significantly more memory than CPU to your worker Pods if you run memory-intensive tasks.
  • At the same time, your scheduler may need more memory than CPU.
Deployment Admins can:
  • Assign exact CPU and memory values to each component.
  • Ensure that Pods use these values as both requests and limits, providing predictable scheduling and resource enforcement.
In Astro Private Cloud, you configure resources by setting CPU and memory directly for each Apache Airflow component in the UI or with the APC API.

Set CPU and memory resources in the Astro Private Cloud UI

Configure worker resources

  1. Navigate to your Deployment in the APC UI.
  2. Open the Settings page for that Deployment.
  3. In the Execution Environment section, under Workers, set the desired CPU and memory values. These values will be applied to all worker Pods.
  4. Click Deploy Changes.

Configure core component resources

  1. Navigate to your Deployment in the APC UI.
  2. Open the Settings page for that Deployment.
  3. In the Core Resources section, set CPU and memory for the following components individually:
    • Webserver / API Server (Airflow 3 only)
    • Scheduler
    • Triggerer
    • Dag Processor
When setting CPU resources for the Scheduler or Workers using milliCPU (mCPU) values, the values must be multiples of 100 (such as 100, 200, 300, etc). Non-multiples of 100 may cause Deployment failures.
  1. Click Deploy Changes.
All values set here are applied as both requests and limits for Pods, ensuring consistent scheduling.
For memory-intensive or CPU-intensive workloads, adjust individual components accordingly. For example, you can give workers more memory for data-heavy tasks.

Set Deployment component resources with APC API

You can also set resource specifications with the APC API.

Set defaults in the config

You can set custom defaults, limits, and minimums for Airflow components by defining them in the config file. When you define a component using the following example, Deployments are created with those resource values, and the UI reflects the same defaults.

Set resource quotas per Deployment

The upsertDeployment mutation takes a quotas argument that sets a minimum resource quota for one Deployment. Starting in APC 2.2, APC uses the larger of each value you set and the quota it calculates from the Deployment’s components, so a value that’s too small doesn’t cause Pods to be refused. For the format of the argument and how it combines with other quota settings, see Raise the quota for one Deployment with the quotas argument.

Configure Deployment-level limits for resource usage

Astro Private Cloud limits the resources that all Pods in a Deployment can use by creating and managing a LimitRange and ResourceQuota in the Deployment’s namespace. APC calculates both from the resource requirements of the Deployment’s components. To give KubernetesExecutor and KubernetesPodOperator task Pods room in a Helm-mode Deployment, set the Deployment’s Extra Capacity. resourceManagement.maxExtraCapacity sets the largest Extra Capacity a Deployment can have, and resourceManagement.maxPodCapacity sets the largest size of any single Pod.
For every setting that changes the quota and limit range, see Deployment resource quotas and limit ranges.

Configurable components

KubernetesExecutor task Pod sizes are created on an as-needed basis and don’t have persisting resource requirements. Their resource requirements are configured at the task level.
Configurable components include:

Airflow scheduler

Airflow Dag processor

To enable a standalone Dag processor, set the airflowComponents.dagProcessor.enabled feature flag to true at the cluster level. In the APC UI, go to your Clusters page, select your cluster, click Edit in the Deployment Configuration section, and add the following override to the Configuration Override field:
For details on using the UI for configuration, see Override base configuration.
You can configure extra containers for the Dag processor in the values.yaml file. For example:

Airflow webserver

StatsD

Database connection pooler (PgBouncer)

Celery diagnostic web interface (Flower)

Redis

Celery workers

Triggerer