Skip to main content
Version: 0.37

Customize Deployment CPU and management resources per component

When you create a new Astronomer Deployment, you can specify the exact amount of CPU and memory that power its core components using the Custom Resources resource strategy.

Compared to using Astronomer Units (AU), which represent a fixed amount of CPU and memory, specifying custom resources gives you greater flexibility to define how your Deployments run. For example, you might need to allocate significantly more memory than CPU to your worker Pods if you need to run memory-intensive tasks, but at the same time you need more memory than CPU in your scheduler. In this scenario, using AUs isn't sufficient because each component needs a different CPU to memory ratio.

When a Deployment uses the Custom Resources resource strategy, Deployment Admins can specify the exact amount of CPU and memory they want each component to have without any scaling limitations. The resources you specify are used both as the limit and request values for the Pods running your components.

Set custom resource usage in the Software UI​

To switch from using AU to using custom resource specifications:

  1. In the Software UI, open your Deployment.
  2. In the Settings tab, change your Resource Strategy to Custom Resources.
  3. For each Software component, adjust the CPU and memory sliders to match your resource requirements. You can change the unit of the resource by clicking the dropdown menu that shows your current unit.
info

To create a Deployment through the Houston API that uses the custom resource strategy be default, set astroUnitsEnabled: false in your Deployment creation mutation.

Disable Astronomer Units (AUs) from Deployment resource configurations​

To limit users to only configure Deployments using custom resources, set the following configuration in your values.yaml file:

astronomer:
  houston:
    config:
      deployments:
        resourceProvisioningStrategy:
          astroUnitsEnabled: false

Then, save this configuration and push it to your platform. See Apply a Platform Config Change.

Set custom resources with Houston API​

You can also switch from using AUs to custom resources specifications with the Houston API.

Use Custom Resource Defaults​

You can set custom defaults, limits and minimums for Airflow components and enable the use of the custom defaults with the feature flag componentsCustomDefaultsEnabled.

To enforce using custom resources only, you must enable custom defaults and disable Astro Units.

astronomer:
  houston:
    config:
      componentsCustomDefaultsEnabled: true
      resourceProvisioningStrategy:
        astroUnitsEnabled: false

When componentsCustomDefaultsEnabled is set to false and Astro Units are enabled, users must use the values defined in the config file for the AU defaults. This means when you use the following code example to define a component, Deployments are created respecting the AU resource definitions and the UI uses these defaults.

astronomer:
  houston:
    config:
      componentsCustomDefaultsEnabled: false
      resourceProvisioningStrategy:
        astroUnitsEnabled: true
      components:
        - name: scheduler
          au:
            default: 5
            limit: 50
            minimum: 1
          extra:
            - name: replicas
              default: 1
              minimum: 1
              limit: 4
              minAirflowVersion: "2.0.0"

When `componentsCustomDefaultsEnabled` is set to `true` and Astro Units is enabled or disabled, users must use the values defined in the config file for the custom defaults. This means when you use the following code example to define a component, Deployments are created respecting the custom definitions and the UI uses these defaults.

```yaml
astronomer:
  houston:
    config:
      componentsCustomDefaultsEnabled: true
      resourceProvisioningStrategy:
        astroUnitsEnabled: true
      components:
        - name: scheduler
          custom:
            default:
              cpu: 1000
              memory: 2000
            minimum:
              cpu: 500
              memory: 1000
            limit:
              cpu: 6000
              memory: 12000
          extra:
            - name: replicas
              default: 1
              minimum: 1
              limit: 4
              minAirflowVersion: "2.0.0"

## Configure Deployment-level limits for resource usage

Astronomer Software limits the amount of resources that can be used by all Pods in a Deployment by creating and managing a `LimitRange` and `ResourceQuota` for the namespace associated with each Deployment.

These values are automatically adjust to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the `LimitRange` and `ResourceQuota`. Add resources by configuring `astronomer.houston.config.deployments.maxExtraCapacity` and `astronomer.houston.config.deployments.maxExtraPodCapacity` to account for the requirements of KubernetesExecutor and KubernetesPodOperator tasks.
```yaml
astronomer:
  houston:
    config:
      deployments:
        maxExtraCapacity:
          cpu: 40000 # in milliCPUs (m)
          memory: 153600 # in MiB (Mi)
        maxPodCapacity:
          cpu: 3500 # in milliCPUs (m)
          memory: 13440 # in MiB (Mi)

Configurable Components​

tip

KubernetesExecutor task Pod sizes are created on an as-needed basis and don't have persisting resource requirements. Their resource requirements are configured at the task level.

Configurable components include:

Airflow Scheduler​

- name: scheduler
  au:
    default: 5
    limit: 50
    minimum: 1
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
      minAirflowVersion: "2.0.0"

Airflow Webserver​

- name: webserver
 au:
   default: 5
   limit: 50
   minimum: 1
 custom:
   default:
     cpu: 1000
     memory: 2000
   minimum:
     cpu: 500
     memory: 1000
   limit:
     cpu: 6000
     memory: 12000

StatsD​

- name: statsd
  au:
    default: 2
    minimum: 2
    limit: 30
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 3000
      memory: 11520

Database Connection Pooler (PgBouncer)​

- name: pgbouncer
  au:
    default: 2
    minimum: 2
    limit: 2
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768

Celery Diagnostic Web Interface (Flower)​

- name: flower
  au:
    default: 2
    minimum: 2
    limit: 2
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768

Redis​

- name: redis
  au:
    default: 2
    minimum: 2
    limit: 2
 custom:
   default:
     cpu: 200
     memory: 768
   minimum:
     cpu: 200
     memory: 768
   limit:
     cpu: 200
     memory: 768

Celery Workers​

- name: workers
 au:
   default: 10
   minimum: 1
   limit: 30
 custom:
   default:
     cpu: 1000
     memory: 3840
   minimum:
     cpu: 100
     memory: 384
   limit:
     cpu: 3000
     memory: 11520
 extra:
   - name: terminationGracePeriodSeconds
     default: 600
     minimum: 0
     limit: 36000
   - name: replicas
     default: 1
     minimum: 1
     limit: 20

Triggerer​

- name: triggerer
  au:
    default: 5
    limit: 50
    minimum: 1
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 0
      limit: 4
      minAirflowVersion: "2.2.0"

Was this page helpful?