Customize Deployment CPU and management resources per component

When you create a new Astronomer Deployment, you can specify the exact amount of CPU and memory that power its core components using the Custom Resources resource strategy. Compared to using Astronomer Units (AU), which represent a fixed amount of CPU and memory, specifying custom resources gives you greater flexibility to define how your Deployments run. For example, you might need to allocate significantly more memory than CPU to your worker Pods if you need to run memory-intensive tasks, but at the same time you need more memory than CPU in your scheduler. In this scenario, using AUs isn’t sufficient because each component needs a different CPU to memory ratio. When a Deployment uses the Custom Resources resource strategy, Deployment Admins can specify the exact amount of CPU and memory they want each component to have without any scaling limitations. The resources you specify are used both as the limit and request values for the Pods running your components.

Set custom resource usage in the Software UI

To switch from using AU to using custom resource specifications:

In the Software UI, open your Deployment.
In the Settings tab, change your Resource Strategy to Custom Resources.
For each Software component, adjust the CPU and memory sliders to match your resource requirements. You can change the unit of the resource by clicking the dropdown menu that shows your current unit.

When setting CPU resources for the Scheduler or Workers using milliCPU (mCPU) values, the values must be multiples of 100 (such as 100, 200, 300, etc). Non-multiples of 100 may cause Deployment failures.

To create a Deployment through the Houston API that uses the custom resource strategy be default, set astroUnitsEnabled: false in your Deployment creation mutation.

Disable Astronomer Units (AUs) from Deployment resource configurations

To limit users to only configure Deployments using custom resources, set the following configuration in your values.yaml file:

astronomer:
  houston:
    config:
      deployments:
        resourceProvisioningStrategy:
          astroUnitsEnabled: false

Then, save this configuration and push it to your platform. See Apply a Platform Config Change.

Set custom resources with Houston API

You can also switch from using AUs to custom resources specifications with the Houston API.

Use Custom Resource Defaults

You can set custom defaults, limits and minimums for Airflow components and enable the use of the custom defaults with the feature flag componentsCustomDefaultsEnabled. To enforce using custom resources only, you must enable custom defaults and disable Astro Units.

astronomer:
  houston:
    config:
      componentsCustomDefaultsEnabled: true
      resourceProvisioningStrategy:
        astroUnitsEnabled: false

When componentsCustomDefaultsEnabled is set to false and Astro Units are enabled, users must use the values defined in the config file for the AU defaults. This means when you use the following code example to define a component, Deployments are created respecting the AU resource definitions and the UI uses these defaults.

astronomer:
  houston:
    config:
      componentsCustomDefaultsEnabled: false
      resourceProvisioningStrategy:
        astroUnitsEnabled: true
      components:
        - name: scheduler
          au:
            default: 5
            limit: 50
            minimum: 1
          extra:
            - name: replicas
              default: 1
              minimum: 1
              limit: 4
              minAirflowVersion: "2.0.0"

When componentsCustomDefaultsEnabled is set to true and Astro Units is enabled or disabled, users must use the values defined in the config file for the custom defaults. This means when you use the following code example to define a component, Deployments are created respecting the custom definitions and the UI uses these defaults.

astronomer:
  houston:
    config:
      componentsCustomDefaultsEnabled: true
      resourceProvisioningStrategy:
        astroUnitsEnabled: true
      components:
        - name: scheduler
          custom:
            default:
              cpu: 1000
              memory: 2000
            minimum:
              cpu: 500
              memory: 1000
            limit:
              cpu: 6000
              memory: 12000
          extra:
            - name: replicas
              default: 1
              minimum: 1
              limit: 4
              minAirflowVersion: "2.0.0"

Set resource quotas per deployment

Astronomer Software lets you override default quota calculations by specifying resource quotas directly in the Houston API’s Deployment payload. Use the quotas object to set custom CPU and memory requests/limits for your deployment. If you do not provide quotas, Astronomer uses the platform’s default quota logic.

Default quotas config

Include the requests and limits in your quotas parameter of the deployment create or update API payload. If you set quotas, make sure your values are not less than required platform minimums and do not exceed the allowed platform maximums. The following example shows the quotas object and the platform defaults for CPU and memory requests and limits.

"quotas": {
  "requests": {
    "cpu": 1,
    "memory": "1920Mi"
  },
  "limits": {
    "cpu": 2,
    "memory": "1920Mi"
  }
}

Configuration options

Use custom quotas if you need to guarantee or constrain Deployment resources beyond the system-determined logic. If you do not set quotas, the deployment will use platform default resource constraints.

Deployments will fail due to insufficient quotas if you set resource quotas to less than or greater than the Astronomer platform-provided minimum or maximum limits.Typical platform defaults:

CPU: 10 vCPU
Memory: 28272Mi (~28Gi)

Configuration Name	Component	Description	Default Value (if not set)	Accepted Values
`quotas`	Deployment	Optional JSON object with custom resource quotas	Not set–platform logic is used	JSON object (`requests` and `limits`)
`quotas.requests.cpu`	Deployment	CPU quota guaranteed (requested)	Platform default	Number (integer or float)
`quotas.requests.memory`	Deployment	Memory quota guaranteed (requested)	Platform default	String, (`"1920Mi"`, `"2Gi"`)
`quotas.limits.cpu`	Deployment	CPU quota maximum (limit)	Platform default	Number (integer or float)
`quotas.limits.memory`	Deployment	Memory quota maximum (limit)	Platform default	String, ( `"1920Mi"`, `"28272Mi"`)

Configure Deployment-level limits for resource usage

Astronomer Software limits the amount of resources that can be used by all Pods in a Deployment by creating and managing a LimitRange and ResourceQuota for the namespace associated with each Deployment. These values are automatically adjust to account for the resource requirements of various components. You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the LimitRange and ResourceQuota. Add resources by configuring astronomer.houston.config.deployments.maxExtraCapacity and astronomer.houston.config.deployments.maxExtraPodCapacity to account for the requirements of KubernetesExecutor and KubernetesPodOperator tasks.

astronomer:
  houston:
    config:
      deployments:
        maxExtraCapacity:
          cpu: 40000 # in milliCPUs (m)
          memory: 153600 # in MiB (Mi)
        maxPodCapacity:
          cpu: 3500 # in milliCPUs (m)
          memory: 13440 # in MiB (Mi)

Configurable Components

KubernetesExecutor task Pod sizes are created on an as-needed basis and don’t have persisting resource requirements. Their resource requirements are configured at the task level.

Configurable components include:

Airflow Scheduler

- name: scheduler
  au:
    default: 5
    limit: 50
    minimum: 1
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
      minAirflowVersion: "2.0.0"

Airflow DAG processor

To enable a standalone DAG processor, set the dagProcessorEnabled feature flag to true in your Houston API configuration in the config.yaml file:

astronomer:
  houston:
    config:
      deployments:
        dagProcessorEnabled: true

You can configure extra containers for the DAG processor in the values.yaml file. For example:

houston:
  config:
    deployments:
      helm:
        airflow:
          dagProcessor:
            extraContainers:
              - name: <extra-container>
                env:
                  - name: DAGS_LOCATION
                    value: /dags
                  - name: SECRET_LOCATION
                    value: /secrets
                  - name: SYNC_INTERVAL
                    value: "15"
                image: ubuntu:latest
                command: ["/bin/bash", "-c", "--"]
                args: ["<arg-1>; <arg-2>; <arg-3>;"]

- name: dagProcessor
  custom:
    default:
      cpu: 1000
      memory: 3840
    minimum:
      cpu: 1000
      memory: 3840
    limit:
      cpu: 3000
      memory: 11520
  extra:
    - name: replicas
      default: 0
      minimum: 0
      limit: 3

Airflow Webserver

- name: webserver
 au:
   default: 5
   limit: 50
   minimum: 1
 custom:
   default:
     cpu: 1000
     memory: 2000
   minimum:
     cpu: 500
     memory: 1000
   limit:
     cpu: 6000
     memory: 12000

StatsD

- name: statsd
  au:
    default: 2
    minimum: 2
    limit: 30
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 3000
      memory: 11520

Database Connection Pooler (PgBouncer)

- name: pgbouncer
  au:
    default: 2
    minimum: 2
    limit: 2
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768

Celery Diagnostic Web Interface (Flower)

- name: flower
  au:
    default: 2
    minimum: 2
    limit: 2
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768

Redis

- name: redis
  au:
    default: 2
    minimum: 2
    limit: 2
 custom:
   default:
     cpu: 200
     memory: 768
   minimum:
     cpu: 200
     memory: 768
   limit:
     cpu: 200
     memory: 768

Celery Workers

- name: workers
 au:
   default: 10
   minimum: 1
   limit: 30
 custom:
   default:
     cpu: 1000
     memory: 3840
   minimum:
     cpu: 100
     memory: 384
   limit:
     cpu: 3000
     memory: 11520
 extra:
   - name: terminationGracePeriodSeconds
     default: 600
     minimum: 0
     limit: 36000
   - name: replicas
     default: 1
     minimum: 1
     limit: 20

Triggerer

- name: triggerer
  au:
    default: 5
    limit: 50
    minimum: 1
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 0
      limit: 4
      minAirflowVersion: "2.2.0"

Overview

Get started

Install

Develop

Administration

Observability

Troubleshoot

Release notes

Reference

Customize Deployment CPU and management resources per component

Set custom resource usage in the Software UI

Disable Astronomer Units (AUs) from Deployment resource configurations

Set custom resources with Houston API

Use Custom Resource Defaults

Set resource quotas per deployment

Default quotas config

Configuration options

Configure Deployment-level limits for resource usage

Configurable Components

Airflow Scheduler

Airflow DAG processor

Airflow Webserver

StatsD

Database Connection Pooler (PgBouncer)

Celery Diagnostic Web Interface (Flower)

Redis

Celery Workers

Triggerer

​Set custom resource usage in the Software UI

​Disable Astronomer Units (AUs) from Deployment resource configurations

​Set custom resources with Houston API

​Use Custom Resource Defaults

​Set resource quotas per deployment

​Default quotas config

​Configuration options

​Configure Deployment-level limits for resource usage

​Configurable Components

​Airflow Scheduler

​Airflow DAG processor

​Airflow Webserver

​StatsD

​Database Connection Pooler (PgBouncer)

​Celery Diagnostic Web Interface (Flower)

​Redis

​Celery Workers

​Triggerer

Set custom resource usage in the Software UI

Disable Astronomer Units (AUs) from Deployment resource configurations

Set custom resources with Houston API

Use Custom Resource Defaults

Set resource quotas per deployment

Default quotas config

Configuration options

Configure Deployment-level limits for resource usage

Configurable Components

Airflow Scheduler

Airflow DAG processor

Airflow Webserver

StatsD

Database Connection Pooler (PgBouncer)

Celery Diagnostic Web Interface (Flower)

Redis

Celery Workers

Triggerer