- You might need to allocate significantly more memory than CPU to your worker Pods if you run memory-intensive tasks.
- At the same time, your scheduler may need more memory than CPU.
- Assign exact CPU and memory values to each component.
- Ensure that Pods use these values as both requests and limits, providing predictable scheduling and resource enforcement.
Set CPU and memory resources in the Astro Private Cloud UI
Configure worker resources
- Navigate to your Deployment in the APC UI.
- Open the Settings page for that Deployment.
- In the Execution Environment section, under Workers, set the desired CPU and memory values. These values will be applied to all worker Pods.
- Click Deploy Changes.
Configure core component resources
- Navigate to your Deployment in the APC UI.
- Open the Settings page for that Deployment.
- In the Core Resources section, set CPU and memory for the following components individually:
- Webserver / API Server (Airflow 3 only)
- Scheduler
- Triggerer
- Dag Processor
- Click Deploy Changes.
All values set here are applied as both requests and limits for Pods, ensuring consistent scheduling.
For memory-intensive or CPU-intensive workloads, adjust individual components accordingly. For example, you can give workers more memory for data-heavy tasks.
Set Deployment component resources with APC API
You can also set resource specifications with the APC API.Set defaults in the config
You can set custom defaults, limits, and minimums for Airflow components by defining them in the config file. When you define a component using the following example, Deployments are created with those resource values, and the UI reflects the same defaults.Set resource quotas per Deployment
TheupsertDeployment mutation takes a quotas argument that sets a minimum resource quota for one Deployment. Starting in APC 2.2, APC uses the larger of each value you set and the quota it calculates from the Deployment’s components, so a value that’s too small doesn’t cause Pods to be refused. For the format of the argument and how it combines with other quota settings, see Raise the quota for one Deployment with the quotas argument.
Configure Deployment-level limits for resource usage
Astro Private Cloud limits the resources that all Pods in a Deployment can use by creating and managing aLimitRange and ResourceQuota in the Deployment’s namespace. APC calculates both from the resource requirements of the Deployment’s components.
To give KubernetesExecutor and KubernetesPodOperator task Pods room in a Helm-mode Deployment, set the Deployment’s Extra Capacity. resourceManagement.maxExtraCapacity sets the largest Extra Capacity a Deployment can have, and resourceManagement.maxPodCapacity sets the largest size of any single Pod.
Configurable components
Configurable components include:Airflow scheduler
Airflow Dag processor
To enable a standalone Dag processor, set the For details on using the UI for configuration, see Override base configuration.
airflowComponents.dagProcessor.enabled feature flag to true at the cluster level. In the APC UI, go to your Clusters page, select your cluster, click Edit in the Deployment Configuration section, and add the following override to the Configuration Override field:You can configure extra containers for the Dag processor in the
values.yaml file. For example: