Customize Deployment CPU and management resources per component

Astronomer Units (AUs) RemovedStarting in Astro Private Cloud 1.0, Astronomer Units (AUs) are no longer supported. All Deployments must now specify CPU and memory directly when configuring resources. This change provides more clarity, flexibility, and aligns with Kubernetes-native resource management.

When you create a new Astro Private Cloud Deployment, you now specify the exact amount of CPU and memory that power its core components. For example:

You might need to allocate significantly more memory than CPU to your worker Pods if you run memory-intensive tasks.
At the same time, your Scheduler may need more memory than CPU.

Deployment Admins can:

Assign exact CPU and memory values to each component.
Ensure that Pods use these values as both requests and limits, providing predictable scheduling and resource enforcement.

In Astro Private Cloud 1.0, you configure resources by setting CPU and memory directly for each Airflow component in the UI or with the Houston API.

Set CPU and Memory Resources in the Astro Private Cloud UI

Configure Worker resources

Navigate to your Deployment in the APC UI.
Open the Settings page for that Deployment.
In the Execution Environment section, under Workers, set the desired CPU and memory values. These values will be applied to all worker Pods.
Click Deploy Changes.

Configure core component resources

Navigate to your Deployment in the APC UI.
Open the Settings page for that Deployment.
In the Core Resources section, set CPU and memory for the following components individually:
- Webserver / API Server (Airflow 3 only)
- Scheduler
- Triggerer
- Dag Processor

When setting CPU resources for the Scheduler or Workers using milliCPU (mCPU) values, the values must be multiples of 100 (such as 100, 200, 300, etc). Non-multiples of 100 may cause Deployment failures.

Click Deploy Changes.

All values set here are applied as both requests and limits for Pods, ensuring consistent scheduling.

For memory-intensive or CPU-intensive workloads, adjust individual components accordingly. For example, you can give workers more memory for data-heavy tasks.

Set Deployment component resources with Houston API

You can also set resources specifications with the Houston API.

Setting defaults in the config

You can set custom defaults, limits, and minimums for Airflow components by defining them in the config file. When you define a component using the example below, Deployments are created with those resource values, and the UI reflects the same defaults.

astronomer:
  houston:
    config:
      components:
        - name: scheduler
          custom:
            default:
              cpu: 1000
              memory: 2000
            minimum:
              cpu: 500
              memory: 1000
            limit:
              cpu: 6000
              memory: 12000
          extra:
            - name: replicas
              default: 1
              minimum: 1
              limit: 4
              minAirflowVersion: "2.0.0"

Set resource quotas per deployment

Astro Private Cloud lets you override default quota calculations by specifying resource quotas directly in the Houston API’s Deployment payload. Use the quotas object to set custom CPU and memory requests/limits for your deployment. If you do not provide quotas, Astronomer uses the platform’s default quota logic.

Default quotas config

Include the requests and limits in your quotas parameter of the deployment upsert API payload. If you set quotas, make sure your values are not less than required platform minimums and do not exceed the allowed platform maximums. The following example shows the quotas object and the platform defaults for CPU and memory requests and limits.

"quotas": {
  "requests": {
    "cpu": 1,
    "memory": "1920Mi"
  },
  "limits": {
    "cpu": 2,
    "memory": "1920Mi"
  }
}

Configuration options

Use custom quotas if you need to guarantee or constrain Deployment resources beyond the system-determined logic. If you do not set quotas, the deployment will use platform default resource constraints.

Deployments will fail due to insufficient quotas if you set resource quotas to less than or greater than the Astronomer platform-provided minimum or maximum limits.Typical platform defaults:

CPU: 10 vCPU
Memory: 28272Mi (~28Gi)

Configuration Name	Component	Description	Default Value (if not set)	Accepted Values
`quotas`	Deployment	Optional JSON object with custom resource quotas	Not set–platform logic is used	JSON object (`requests` and `limits`)
`quotas.requests.cpu`	Deployment	CPU quota guaranteed (requested)	Platform default	Number (integer or float)
`quotas.requests.memory`	Deployment	Memory quota guaranteed (requested)	Platform default	String, (`"1920Mi"`, `"2Gi"`)
`quotas.limits.cpu`	Deployment	CPU quota maximum (limit)	Platform default	Number (integer or float)
`quotas.limits.memory`	Deployment	Memory quota maximum (limit)	Platform default	String, ( `"1920Mi"`, `"28272Mi"`)

Configure Deployment-level limits for resource usage

Astro Private Cloud limits the amount of resources that can be used by all Pods in a Deployment by creating and managing a LimitRange and ResourceQuota for the namespace associated with each Deployment. These values are automatically adjusted to account for the resource requirements of various components. You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the LimitRange and ResourceQuota. Add resources by configuring astronomer.houston.config.deployments.maxExtraCapacity and astronomer.houston.config.deployments.maxExtraPodCapacity to account for the requirements of KubernetesExecutor and KubernetesPodOperator tasks.

astronomer:
  houston:
    config:
      deployments:
        maxExtraCapacity:
          cpu: 40000 # in milliCPUs (m)
          memory: 153600 # in MiB (Mi)
        maxPodCapacity:
          cpu: 3500 # in milliCPUs (m)
          memory: 13440 # in MiB (Mi)

Configurable Components

KubernetesExecutor task Pod sizes are created on an as-needed basis and don’t have persisting resource requirements. Their resource requirements are configured at the task level.

Configurable components include:

Airflow Scheduler

- name: scheduler
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
      minAirflowVersion: "2.0.0"

Airflow Dag processor

To enable a standalone Dag processor, set the dagProcessorEnabled feature flag to true in your Houston API configuration in the config.yaml file:

astronomer:
  houston:
    config:
      deployments:
        dagProcessorEnabled: true

You can configure extra containers for the Dag processor in the values.yaml file. For example:

houston:
  config:
    deployments:
      helm:
        airflow:
          dagProcessor:
            extraContainers:
              - name: <extra-container>
                env:
                  - name: DAGS_LOCATION
                    value: /dags
                  - name: SECRET_LOCATION
                    value: /secrets
                  - name: SYNC_INTERVAL
                    value: "15"
                image: ubuntu:latest
                command: ["/bin/bash", "-c", "--"]
                args: ["<arg-1>; <arg-2>; <arg-3>;"]

- name: dagProcessor
  custom:
    default:
      cpu: 1000
      memory: 3840
    minimum:
      cpu: 1000
      memory: 3840
    limit:
      cpu: 3000
      memory: 11520
  extra:
    - name: replicas
      default: 0
      minimum: 0
      limit: 3

Airflow Webserver

- name: webserver
 custom:
   default:
     cpu: 1000
     memory: 2000
   minimum:
     cpu: 500
     memory: 1000
   limit:
     cpu: 6000
     memory: 12000

StatsD

- name: statsd
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 3000
      memory: 11520

Database Connection Pooler (PgBouncer)

- name: pgbouncer
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768

Celery Diagnostic Web Interface (Flower)

- name: flower
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768

Redis

- name: redis
 custom:
   default:
     cpu: 200
     memory: 768
   minimum:
     cpu: 200
     memory: 768
   limit:
     cpu: 200
     memory: 768

Celery Workers

- name: workers
 custom:
   default:
     cpu: 1000
     memory: 3840
   minimum:
     cpu: 100
     memory: 384
   limit:
     cpu: 3000
     memory: 11520
 extra:
   - name: terminationGracePeriodSeconds
     default: 600
     minimum: 0
     limit: 36000
   - name: replicas
     default: 1
     minimum: 1
     limit: 20

Triggerer

- name: triggerer
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 0
      limit: 4
      minAirflowVersion: "2.2.0"

Overview

Architecture

Upgrade to Astro Private Cloud

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

Houston API

Platform Observability

Release notes

Version compatibility

Reference

Customize Deployment CPU and management resources per component

Set CPU and Memory Resources in the Astro Private Cloud UI

Configure Worker resources

Configure core component resources

Set Deployment component resources with Houston API

Setting defaults in the config

Set resource quotas per deployment

Default quotas config

Configuration options

Configure Deployment-level limits for resource usage

Configurable Components

Airflow Scheduler

Airflow Dag processor

Airflow Webserver

StatsD

Database Connection Pooler (PgBouncer)

Celery Diagnostic Web Interface (Flower)

Redis

Celery Workers

Triggerer

​Set CPU and Memory Resources in the Astro Private Cloud UI

​Configure Worker resources

​Configure core component resources

​Set Deployment component resources with Houston API

​Setting defaults in the config

​Set resource quotas per deployment

​Default quotas config

​Configuration options

​Configure Deployment-level limits for resource usage

​Configurable Components

​Airflow Scheduler

​Airflow Dag processor

​Airflow Webserver

​StatsD

​Database Connection Pooler (PgBouncer)

​Celery Diagnostic Web Interface (Flower)

​Redis

​Celery Workers

​Triggerer

Set CPU and Memory Resources in the Astro Private Cloud UI

Configure Worker resources

Configure core component resources

Set Deployment component resources with Houston API

Setting defaults in the config

Set resource quotas per deployment

Default quotas config

Configuration options

Configure Deployment-level limits for resource usage

Configurable Components

Airflow Scheduler

Airflow Dag processor

Airflow Webserver

StatsD

Database Connection Pooler (PgBouncer)

Celery Diagnostic Web Interface (Flower)

Redis

Celery Workers

Triggerer