Configure component size limits

Astronomer Software allows you to customize the minimum and maximum sizes of most Astronomer platform and Airflow components.

You can configure the CPU and memory resources of Airflow components through the Astro UI or with the Houston API, in either Astro Units (AU’s) or Custom Resources, when CPU and memory are not a ratio of each other.

Astro Units (AU’s)

You can increase or decrease resources on Astro by units of size know as AU’s (Astro Units). Adding or removing an AstroUnit changes a Deployment’s size by changing the following:

• the size of a component by one-tenth of a v-core
• the size of the component by 384Mi of memory
• the limit of connections to the Postgres server by half of a connection
• the limit of connections used internally by Airflow to the database connection pooler by 5
• the size of the deployment-namespace’s overall cumulative limits (ResourceQuota and LimitRange) by one-tenth of a v-core and 834Mi of memory.
• the limit of Pods in the namespace by one Pod (ResourceQuota)

Configure Deployment-level limits for individual Pod sizes

You can use astronomer.houston.config.deployments.maxPodAu to configure the maximum size any individual pod can be.

1
astronomer:
2
  houston:
3
    config:
4
      deployments:
5
        maxPodAu: 35 # default is 35

Configure Deployment-level limits for resource usage

Astronomer Software limits the amount of resources that can be used by all pods in a Deployment by creating and managing a LimitRange and ResourceQuota for the namespace associated with each Deployment.

These values are automatically adjust to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the LimitRange and ResourceQuota. Add resources by configuring astronomer.houston.config.deployments.maxExtraAu to account for the requirements of KubernetesExecutor and KubernetesPodOperator tasks.

1
astronomer:
2
  houston:
3
    config:
4
      deployments:
5
        maxExtraAu: 400 # default is 400

Configure the sizes of individual Deployment-level components

Components represent different parts of the Astronomer Software Deployment. You can customize the default configuration for a component by defining it in astronomer.houston.config.deployments.components.

A list of configurable components and options is provided in Configurable Components.

When defining components, you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.

For example, to increase the maximum size a Celery worker task from 30 AU (3 Vcpu/11.5Gi) to 50AU (3 Vcpu/192Gi), add the full Celery worker component definition to astronomer.houston.config.deployments.components in your values.yaml with a higher limit (defined in AUs):

1
astronomer:
2
  houston:
3
    config:
4
      deployments:
5
        components:
6
        - name: workers
7
          au:
8
            default: 10
9
            minimum: 1
10
            limit: 50 # default was 30
11
          extra:
12
            - name: terminationGracePeriodSeconds
13
              default: 600
14
              minimum: 0
15
              limit: 36000
16
            - name: replicas
17
              default: 1
18
              minimum: 1
19
              limit: 10
20
        # any additional component configurations go here
21
        # - name: another-component
22
        #   au:
23
        #     default: 10
24
        #     ...

Configurable Components

Configurable components include:

Airflow Scheduler

1
- name: scheduler
2
  au:
3
    default: 5
4
    minimum: 5
5
    limit: 30
6
  extra:
7
    - name: replicas
8
      default: 1
9
      minimum: 1
10
      limit: 4
11
      minAirflowVersion: 2.0.0

Airflow Webserver

1
- name: webserver
2
  au:
3
    default: 5
4
    minimum: 5
5
    limit: 30

StatsD

1
- name: statsd
2
  au:
3
    default: 2
4
    minimum: 2
5
    limit: 30

Database Connection Pooler (PgBouncer)

1
- name: pgbouncer
2
  au:
3
    default: 2
4
    minimum: 2
5
    limit: 2

Celery Diagnostic Web Interface (Flower)

1
- name: flower
2
  au:
3
    default: 2
4
    minimum: 2
5
    limit: 2

Redis

1
- name: redis
2
  au:
3
    default: 2
4
    minimum: 2
5
    limit: 2

Celery Workers

1
- name: workers
2
  au:
3
    default: 10
4
    minimum: 1
5
    limit: 30
6
  extra:
7
    - name: terminationGracePeriodSeconds
8
      default: 600
9
      minimum: 0
10
      limit: 36000
11
    - name: replicas
12
      default: 1
13
      minimum: 1
14
      limit: 10

Triggerer

1
- name: triggerer
2
  au:
3
    default: 5
4
    minimum: 5
5
    limit: 30
6
  extra:
7
    - name: replicas
8
      default: 1
9
      minimum: 0
10
      limit: 2
11
      minAirflowVersion: 2.2.0

 Manage Kubernetes worker CPU and memory with global platform config

You can use workers.resources.enabled for the KubernetesExecutor to manage worker Pod CPU and memory allocations with a platform configuration. When set to false, Astronomer Software ignores worker Pod CPU or memory allocations set in the API or UI. This allows you to control Pod resources with your own Kubernetes policies or admission controllers.

API/UI manages Pod resources (Default KubernetesExecutor configuration)

To configure the default behavior where the API and UI manage worker Pod CPU and memory resources for the KubernetesExecutor, use the following configuration:

1
deployments:
2
  executors:
3
    - name: KubernetesExecutor
4
      enabled: true
5
      components:
6
        - scheduler
7
        - webserver
8
        - statsd
9
        - pgbouncer
10
        - triggerer
11
        - dagProcessor
12
      defaultExtraAu: 10
13
      defaultExtraCapacity:
14
        cpu: 1000
15
        memory: 3840
16
      workers:
17
        ephemeralStorage:
18
          disabled: true
19
        resources:
20
          enabled: true

Disable API/UI resource configuration

If you configure your KubernetesExecutor so that CPU and memory requests and limits are set outside of Astronomer Software, you must disable resource configuration set with the Astronomer Software API/UI. To disable the Astronomer Software UI/API configuration, add the resources.enabled: false flag to your values.yaml file. This ensures that Software applies the worker resource settings exclusively from your preferred source, such as Pod mutation hooks or Pod configuration files. The following configuration sets them as an empty dict.

1
deployments:
2
  executors:
3
   - name: KubernetesExecutor
4
      enabled: true
5
      components:
6
        - scheduler
7
        - webserver
8
        - statsd
9
        - pgbouncer
10
        - triggerer
11
        - dagProcessor
12
      defaultExtraAu: 10
13
      defaultExtraCapacity:
14
        cpu: 1000
15
        memory: 3840
16
      workers:
17
        ephemeralStorage:
18
          disabled: true
19
        resources:
20
          enabled: false

Existing deployments are unchanged unless you update this setting.

Configure the size of KubernetesExecutor task pods

Kubernetes Executor task pods are defined at the task level when the DAG passes resource requests as part of executor_config into the Operator. When not defined, these tasks default to using 1 AU, which is one-tenth of a Vcpu/384 Mi of memory. This means that when you define resource requests or limits for CPU and memory, ensure the maximum pod size is large enough to avoid errors during pod creation.

The following example demonstrates how to configure resource limits and requests:

# import kubernetes.client.models as k8s
from kubernetes.client import models as k8s
# define an executor_config with the desired resources
my_executor_config={
  "pod_override": k8s.V1Pod(
      spec=k8s.V1PodSpec(
          containers=[
              k8s.V1Container(
                  name="base",
                  resources=k8s.V1ResourceRequirements(
                      requests={
                          "cpu": "50m",
                          "memory": "384Mi"
                      },
                      limits={
                          "cpu": "1000m",
                          "memory": "1024Mi"
                      }
                  )
              )
          ]
      )
  )
}
# pass in executor_config=my_executor_config to any Operator
#@task(executor_config=my_executor_config)
#def some_task():
 #   ...
#task = PythonOperator(
#    task_id="another_task",
#    python_callable=my_fun,
#    executor_config=my_executor_config
#)

Note that KubernetesExecutor task pods are limited to the LimitRanges and quotas defined within the pod namespace.

Customize Astro Units (AU’s)

Configurable options include:

• astronomer.houston.config.deployments.astroUnit.cpu - the amount an AU contributes to the size of a component, in thousandths of a vCPU, and the size of the LimitRange or ResourceQuota, defaults to 100.
• astronomer.houston.config.deployments.astroUnit.memory - the amount an AU contributes to the size of a component, in Mi, and the size of the LimitRange or ResourceQuota, set to 384 by default.
• astronomer.houston.config.deployments.astroUnit.pods - the amount an AU contributes to the maximum amount of pods permitted by ResourceQuota, set to 1 by default.
• astronomer.houston.config.deployments.astroUnit.actualConns - the amount an AU contributes to the limit of connections to the Postgres server, set to 0.5 by default.
• astronomer.houston.config.deployments.astroUnit.airflowConns - The amount an AU contributes to the limit of connections used internally by Airflow to the database connection pooler, set to 5 by default.

The following code example shows how to set the resources per individual AU:

1
astronomer:
2
  houston:
3
    config:
4
      deployments:
5
        astroUnit:
6
          # values are listed with their default values
7
          cpu: 100
8
          memory: 384
9
          pods: 1
10
          actualConns: 0.5 # upstream connections to postgres
11
          airflowConns: 5  # downstream connections to airflow