Deployment resource quotas and limit ranges

Astro Private Cloud (APC) runs each Deployment in its own Kubernetes namespace and manages two Kubernetes objects in that namespace:

A ResourceQuota named <release-name>-resource-quota. It caps the number of Pods in the namespace, and the total CPU, memory, and ephemeral storage that those Pods can request and use.
A LimitRange named <release-name>-limit-range. It caps the size of any single Pod, and gives a default CPU and memory size to containers that don’t declare one.

APC calculates both objects from the Deployment’s configuration and reapplies them every time the Deployment updates.

Operator-mode Deployments get a ResourceQuota and LimitRange starting in APC 2.2. The rule that combines your configured quota values with the calculated quota, and the calculated ephemeral storage quota, also start in 2.2. For the behavior of APC 2.0 and 2.1, see Behavior in earlier versions.

How the quota affects a Deployment

The ResourceQuota applies to every Pod in the Deployment’s namespace, not only to Airflow components. It counts:

The Airflow components: the scheduler, webserver or API server, triggerer, Dag processor, Celery workers, StatsD, PgBouncer, Redis, and Flower.
Pods that KubernetesPodOperator tasks start in the Deployment’s namespace.
Kubernetes executor task Pods.
Any other workload that runs in the namespace, for example in a pre-created namespace.

If a new Pod would take the namespace over any quota value, Kubernetes refuses to create it and the Pod’s controller records a FailedCreate event whose message contains exceeded quota. Running Pods aren’t affected. Kubernetes counts each resource as follows:

CPU and memory: Every container must declare a CPU and memory request and limit. The LimitRange gives a default to containers that declare none. See Configure the limit range.
Ephemeral storage: A container that doesn’t declare ephemeral storage counts as zero.
Requests and limits: APC sets requests.<resource> and limits.<resource> to the same value.

How APC calculates the quota

APC calculates a value for each quota key from the components the Deployment runs, called the calculated quota.

Quota key	What APC counts
`pods`	Two times the number of Pods of every component, plus the Pods that Extra Capacity allows.
`requests.cpu`, `limits.cpu`	Two times the CPU of each component multiplied by its replicas, plus an allowance for sidecars, plus Extra Capacity.
`requests.memory`, `limits.memory`	Two times the memory of each component multiplied by its replicas, plus an allowance for sidecars, plus Extra Capacity.
`requests.ephemeral-storage`, `limits.ephemeral-storage`	Two times the ephemeral storage of each component’s Pod, its main container and sidecars, multiplied by its replicas.

The calculation follows these rules:

Components: APC counts the components of the Deployment’s executor. It counts the triggerer and Dag processor only when they’re enabled, and it doesn’t count Flower in operator mode.
Rollout headroom: APC doubles each component’s resources so old and new Pods can run together during a rolling update.
Worker queues: In a Deployment with worker queues, APC counts each queue at its ceiling. The ceiling is the queue’s maximum worker count when the queue can autoscale, and its fixed worker count otherwise. APC sums every queue’s ceiling, because all queues can reach it at the same time.
Sidecar allowance: APC adds a fixed allowance for the sidecar containers and jobs it runs beside the Airflow components, such as log groomers and metrics exporters. It also adds the resources of the authentication sidecar, logging sidecar, Dag deploy server, and Airflow database cleanup job when enabled. The allowance is sized from resourceManagement.sidecars and resourceManagement.astroUnit.
Ephemeral storage: APC reads each container’s ephemeral storage from the larger of its request and its limit. It counts each component’s main container and the containers APC runs beside it: the log groomer, metrics exporter, logging sidecar, Dag downloader, authentication sidecar, and any extra containers in the component’s Helm configuration. Only the main containers have ephemeral storage by default, so the sidecars count only when you give them some. In a Deployment with worker queues, APC counts the worker sidecars on every worker Pod.

The calculated quota is the most the Deployment can claim at one time, usually much more than it uses.

How configured values combine with the calculated quota

You can configure a quota value in two places: the helm.airflow.quotas configuration, which you can set at the platform, cluster, Workspace, or Deployment tier, and the quotas argument of a single Deployment. Starting in APC 2.2, APC sets each quota key to the largest of these values:

The calculated quota.
The value for that key in helm.airflow.quotas, after config governance resolves the tiers.
The value for that key in the Deployment’s quotas argument.

As a result:

A configured value can raise a quota but can’t lower it below the calculated quota. A smaller value doesn’t cause Pod refusals; APC uses the calculated quota.
When a configured value wins, including when it equals the calculated quota, APC writes it exactly as you wrote it. For example, 50Gi stays 50Gi.
The pods, CPU, and memory keys always appear in the ResourceQuota.
The ephemeral storage keys appear only when a configured value sets them. APC raises an ephemeral storage limit that’s too small, but it never adds one. The APC Helm chart sets both ephemeral storage keys to 50Gi by default, so every Deployment has an ephemeral storage quota of at least 50 GiB by default.
APC ignores a configured value that isn’t a valid Kubernetes quantity, for example 50 Gi or 4 cores, and uses the calculated quota for that key. The APC API logs a warning that names the key.

Examples of the resulting limits.ephemeral-storage for one Deployment:

Configured value	Calculated quota	Resulting quota
`50Gi` (chart default)	`16384Mi`	`50Gi`
`50Gi`	`139264Mi`	`139264Mi`
`1Gi`	`16384Mi`	`16384Mi`
Not set	Any	No ephemeral storage key

Where you can change the quota

The table lists every setting that changes the ResourceQuota or LimitRange. The Astro CLI can’t set or show quota values. It can change only the executor and the number of triggerer replicas, which change what APC calculates.

Setting	What it changes	APC UI	APC API	Astro CLI	Helm values and config governance
`helm.airflow.quotas`	Raises quota keys for a platform, cluster, Workspace, or Deployment	Configuration override editors	`updateCluster`, `updateWorkspaceDeploymentsConfig`, `updateDeploymentConfig`	Not available	`astronomer.houston.config.deployments.helm.airflow.quotas`
`quotas` argument	Raises quota keys for one Deployment	Not available	`upsertDeployment`	Not available	Not available
Extra Capacity	Adds CPU, memory, and Pods to the quota (Helm mode only)	Settings > Execution Environment > Extra Capacity	`upsertDeployment` `properties.extra_capacity`	Not available	Bounds: `resourceManagement.maxExtraCapacity` and the executor’s `defaultExtraCapacity`
Component CPU, memory, and replicas	Changes the calculated quota	Settings > Execution Environment, Core Resources, and Ancillary Resources	`upsertDeployment`	`--executor`, `--triggerer-replicas`	Bounds: `resourceManagement.components`
Worker queues	Changes the calculated quota	Settings > Worker Queues	`upsertDeployment` `workerQueues`	Not available	Bounds: `resourceManagement.components`, `workerQueues`
Component ephemeral storage	Changes the calculated ephemeral storage quota	Configuration override editors	Configuration override mutations	Not available	`astronomer.houston.config.deployments.helm.airflow.<component>.resources`
Git-sync ephemeral storage	Raises the scheduler and worker ephemeral storage (Helm mode only)	DAG Deployment > Ephemeral Storage Overwrite Gigabytes	`upsertDeployment` `dagDeployment.ephemeralStorage`	Not available	Not available
`resourceManagement.maxPodCapacity`	LimitRange maximum Pod size	Configuration override editors	Configuration override mutations	Not available	`astronomer.houston.config.deployments.resourceManagement.maxPodCapacity`
`resourceManagement.astroUnit`	LimitRange container default, and the sidecar allowance	Configuration override editors	Configuration override mutations	Not available	`astronomer.houston.config.deployments.resourceManagement.astroUnit`
`resourceManagement.resourceQuotas.enabled`	Turns quota and limit range management on or off	Configuration override editors	Configuration override mutations	Not available	`astronomer.houston.config.deployments.resourceManagement.resourceQuotas.enabled`

Raise the quota for a scope with config governance

Set helm.airflow.quotas to give every Deployment in a scope a minimum quota. Use it to give KubernetesPodOperator Pods and Kubernetes executor task Pods room in an operator-mode Deployment, where Extra Capacity isn’t available, or to keep an ephemeral storage quota larger than the calculated quota. The keys are the ResourceQuota keys, written as one flat key each, for example limits.cpu. APC combines these keys with the calculated quota:

pods
requests.cpu and limits.cpu
requests.memory and limits.memory
requests.ephemeral-storage and limits.ephemeral-storage

APC passes any other key in helm.airflow.quotas, for example count/configmaps, to the ResourceQuota as written without validating it. An invalid value can make the Deployment update fail. Set the requests and limits for a resource to the same value so the ResourceQuota keeps requests.<resource> equal to limits.<resource>.

APC UI
APC API
Helm values

The cluster, Workspace, and Deployment configuration override editors take YAML that starts at the deployments level, so omit astronomer, houston, config, and deployments.

Open the override editor for the scope you want:
- For a cluster, see Override base configuration.
- For a Workspace, go to Workspace Settings > Configuration Overrides.
- For a Deployment, open the Deployment and go to Configuration Overrides.
Click Edit.
Add the quota values, and keep the rest of the editor’s content. For a Workspace or a Deployment, saving replaces the stored override with the editor’s content.
helm: airflow: quotas: requests.ephemeral-storage: 200Gi limits.ephemeral-storage: 200Gi
Save the change, then review and confirm each change in the dialog that appears.

Pass the override as a JSON variable. GraphQL field names can’t contain the . and - characters in the quota keys, so you can’t write these keys inline in the mutation.The following example sets the quota for one Deployment. applyToDeployment: true applies the change immediately.

mutation SetDeploymentQuota($deploymentUuid: Uuid!, $override: JSON!) {
  updateDeploymentConfig(
    deploymentUuid: $deploymentUuid
    deploymentsConfigOverride: $override
    reason: "Raise the ephemeral storage quota"
    applyToDeployment: true
  ) {
    id
    config
  }
}

{
  "deploymentUuid": "<deployment-id>",
  "override": {
    "helm": {
      "airflow": {
        "quotas": {
          "requests.ephemeral-storage": "200Gi",
          "limits.ephemeral-storage": "200Gi"
        }
      }
    }
  }
}

deploymentsConfigOverride replaces the whole stored override for the Deployment. Include every key of the existing override that you want to keep. To set the quota for a Workspace or a cluster, pass the same helm object to updateWorkspaceDeploymentsConfig or updateCluster. See Config governance.

To set a minimum quota for every Deployment on the platform, add the values to your control plane values.yaml and apply the change with a Helm upgrade. See Apply a config change.

astronomer:
  houston:
    config:
      deployments:
        helm:
          airflow:
            quotas:
              requests.ephemeral-storage: 50Gi
              limits.ephemeral-storage: 50Gi

When the change applies

APC recalculates and applies the quota every time a Deployment updates: when you deploy code, change its settings, environment variables, or worker queues, or set its quotas argument.

A change at the platform, cluster, or Workspace tier applies to each Deployment the next time that Deployment updates.
A change at the Deployment tier applies immediately when you use updateDeploymentConfig with applyToDeployment: true. A change that you save in the APC UI applies the next time the Deployment updates.

Raise the quota for one Deployment with the quotas argument

The upsertDeployment mutation takes a quotas argument, settable only with the APC API, that raises the quota for one Deployment. Unlike helm.airflow.quotas, it nests each resource under requests and limits:

Field	Required	Format
`requests.cpu`, `limits.cpu`	Yes	Cores, for example `"2"`, or millicores, for example `"2000m"`
`requests.memory`, `limits.memory`	Yes	Memory quantity, for example `"8Gi"` or `"8192Mi"`
`requests.ephemeral-storage`, `limits.ephemeral-storage`	No	Kubernetes quantity, for example `"200Gi"`, `".5Ti"`, or `"2e11"`. Decimal kilo is a lowercase `k`.
`pods`	No	Whole number

Each request must be less than or equal to its limit. The mutation rejects a quotas argument that’s missing a required field or that has a value in the wrong format.

mutation SetQuotaArgument($deploymentUuid: Uuid!, $quotas: JSON) {
  upsertDeployment(deploymentUuid: $deploymentUuid, quotas: $quotas) {
    id
  }
}

{
  "deploymentUuid": "<deployment-id>",
  "quotas": {
    "pods": 60,
    "requests": {
      "cpu": "40",
      "memory": "160Gi",
      "ephemeral-storage": "200Gi"
    },
    "limits": {
      "cpu": "40",
      "memory": "160Gi",
      "ephemeral-storage": "200Gi"
    }
  }
}

APC stores the quotas argument with the Deployment and uses it on every later update. A later quotas argument replaces the values for the keys it sends and keeps the stored values for the others. You can’t delete a stored quotas argument. To stop it from raising the quota, send "0" for each key you set. APC then uses the calculated quota for those keys.

Give task Pods room with extra capacity

Extra Capacity adds CPU, memory, and Pods to the quota of a Helm-mode Deployment so KubernetesPodOperator Pods and Kubernetes executor task Pods have room to run. It sizes the most those Pods can claim at one time. For more, see Kubernetes executor: Set extra capacity. APC adds the Extra Capacity CPU and memory to the CPU and memory quota, and adds Pods in proportion to the larger of the two. Set Extra Capacity in one of these ways:

In the APC UI, open the Deployment, go to Settings > Execution Environment, and set the Extra Capacity CPU and memory.
With the APC API, pass properties: { extra_capacity: { cpu: <millicores>, memory: <mebibytes> } } to upsertDeployment.

Extra Capacity must be within these bounds:

Minimum: the executor’s defaultExtraCapacity. The default is 1,000 millicores and 3,840 MiB for the Kubernetes executor, and zero for the Celery and Local executors.
Maximum: resourceManagement.maxExtraCapacity. The default is 40,000 millicores and 153,600 MiB. A Deployment that already stores a larger value keeps that value as its own maximum.

Values outside the bounds return Minimum Extra CPU not met, Minimum Extra Memory not met, Maximum Extra CPU exceeded, or Maximum Extra Memory exceeded. In operator mode, Extra Capacity is always zero and doesn’t appear in the APC UI. To give task Pods room in an operator-mode Deployment, set a minimum quota with helm.airflow.quotas or the quotas argument.

Change what APC calculates

Change any of these to change the calculated quota:

Component CPU, memory, and replicas: Set them in the Deployment’s Settings or with the APC API. The CPU and memory you set are both the request and the limit. See Customize Deployment CPU and memory resources.
Worker queues: Each queue counts its CPU, memory, and ephemeral storage per worker, multiplied by its ceiling. A queue’s ephemeral storage defaults to 2,048 MiB. See Configure worker queues.
Executor: The executor decides which components run. Change it in the Deployment’s Settings, with the APC API, or with the --executor flag of astro deployment update.
Deployment mode: Flower and Extra Capacity count only in Helm mode.
Component ephemeral storage: Set helm.airflow.<component>.resources.requests.ephemeral-storage and helm.airflow.<component>.resources.limits.ephemeral-storage with config governance. The APC Helm chart sets a 1 GiB request and a 2 GiB limit for the scheduler, webserver, API server, and workers. In a Deployment with worker queues, each queue’s own ephemeral storage replaces the worker value.
Sidecar ephemeral storage: Ephemeral storage that you set on a sidecar adds to its component’s Pods. The sidecars read it from helm.airflow.<component>.logGroomerSidecar.resources, helm.airflow.pgbouncer.metricsExporterSidecar.resources, helm.airflow.<component>.extraContainers, logging.loggingSidecar.resources, dagDeploy.client.resources, and authSideCar.resources.
Git-sync ephemeral storage: In a Helm-mode Deployment that uses git-sync, Ephemeral Storage Overwrite Gigabytes on the DAG Deployment card, or dagDeployment.ephemeralStorage in upsertDeployment, raises the scheduler and worker ephemeral storage. The range is 2 to 10 GiB.
Optional features: Enabling the authentication sidecar, logging sidecar, Dag-only deploys, or Airflow database cleanup adds their resources to the CPU and memory quota.

Configure the limit range

APC builds the LimitRange from two settings:

LimitRange entry	Setting	Default	Effect
Maximum Pod size	`resourceManagement.maxPodCapacity`	3,500 millicores and 13,440 MiB	Kubernetes refuses a Pod whose containers together set a larger CPU or memory limit, including `KubernetesPodOperator` Pods and Kubernetes executor task Pods.
Default container size	`resourceManagement.astroUnit`	100 millicores and 384 MiB	Kubernetes gives this request and limit to a container that declares no CPU or memory.

The LimitRange doesn’t set ephemeral storage. You can set both settings at any config governance tier. maxPodCapacity also caps the CPU and memory of a worker queue, and APC checks the scheduler and webserver limits against it. astroUnit also sizes the sidecar allowance in the quota and the default PgBouncer connection pool sizes.

astronomer:
  houston:
    config:
      deployments:
        resourceManagement:
          maxPodCapacity:
            cpu: 6000
            memory: 23040

APC builds the LimitRange entries itself, so set these two settings instead of helm.airflow.limits.

Turn off quota and limit range management

Set resourceManagement.resourceQuotas.enabled to false to stop APC from managing the ResourceQuota and LimitRange. You can set it at any config governance tier, so you can turn management off for the platform, a cluster, a Workspace, or a single Deployment.

astronomer:
  houston:
    config:
      deployments:
        resourceManagement:
          resourceQuotas:
            enabled: false

The next time each affected Deployment updates, APC removes its ResourceQuota and LimitRange, in both Helm and operator mode. APC also stops checking Extra Capacity against its bounds and component limits against maxPodCapacity for those Deployments.

Without a ResourceQuota and LimitRange, Pods in the Deployment’s namespace can use unlimited resources on the nodes they run on, which can make the cluster unstable. If you turn management off, manage your own ResourceQuota and LimitRange objects.

APC 2.x doesn’t read the APC 1.x setting disableManageResourceQuotasAndLimitRanges. The upgrade to 2.x converts it to resourceManagement.resourceQuotas.enabled. If you still have disableManageResourceQuotasAndLimitRanges in your Helm values, APC ignores it and manages quotas. See Upgrade to Astro Private Cloud 2.0 from 1.x. If you use namespace pools and create the namespace RBAC yourself, grant the data plane internal API the create, delete, get, list, patch, and watch verbs on resourcequotas and limitranges in each namespace. Without them, APC can’t apply or remove the objects.

View the quota

APC shows the quota in these places:

APC UI: Open the Deployment and go to Metrics. The Usage Quotas card shows Pods Usage, CPU Usage, and Memory Usage as a share of the quota. It doesn’t show ephemeral storage.

Kubernetes: Read the objects in the Deployment’s namespace:

kubectl get resourcequota <release-name>-resource-quota -n <namespace> -o yaml
kubectl get limitrange <release-name>-limit-range -n <namespace> -o yaml

The Kubernetes API server rewrites quantities in canonical form, so 51200Mi appears as 50Gi.

APC API: The effectiveConfig field of a Deployment shows the resolved helm.airflow.quotas and resourceManagement values. The config field shows the stored quotas argument, and properties shows extra_capacity. See Effective configuration.
Alerts: The AirflowPodQuota alert fires when a Deployment uses more than 95% of its Pod quota for 10 minutes. The KubeQuotaExceeded alert fires when a namespace uses more than 90% of any quota value for 15 minutes. See Platform and deployment alerts.

The Astro CLI doesn’t show quota values.

Permissions

A custom role can change a setting if it has the listed permission.

Action	Permission	Default roles
Set the `quotas` argument or Extra Capacity	`deployment.config.upsert`	Deployment Editor, Deployment Admin, System Admin
Set a Deployment configuration override	`deployment.deployments.config.update`	Deployment Editor, Deployment Admin, System Editor, System Admin
Apply a Deployment configuration override immediately	`deployment.deployments.config.update` and `deployment.config.upsert`	Deployment Editor, Deployment Admin, System Admin
Set a Workspace configuration override	`workspace.deployments.config.update`	Workspace Editor, Workspace Admin, System Editor, System Admin
Set a cluster configuration override	`cluster.config.update`	Cluster Admin, System Admin
Change platform Helm values	Not applicable	Platform operator with access to the control plane Helm release
View the Usage Quotas card	`deployment.metrics.get`	Every Deployment role and every System role

None of the settings on this page are blocklisted at the Workspace or Deployment tier. Anyone who can set a Workspace or Deployment configuration override can raise the quota for that scope or turn quota management off for it.

Troubleshoot

Kubernetes refuses a Pod with exceeded quota

Find the event in the Deployment’s namespace:

kubectl get events -n <namespace> --field-selector reason=FailedCreate

The message names the quota key, the amount the Pod requested, the amount already used, and the limit. Starting in APC 2.2, the calculated quota covers every Airflow component and every worker queue at its ceiling. A refusal usually means that something APC doesn’t count needs room, for example KubernetesPodOperator Pods or workloads you run in the namespace yourself. Raise the quota with Extra Capacity in Helm mode, or with helm.airflow.quotas or the quotas argument in either mode.

Kubernetes refuses a Pod that’s larger than the maximum Pod size

The message says the maximum usage per Pod is lower than the Pod’s limit, meaning the Pod is larger than resourceManagement.maxPodCapacity. Make the Pod smaller, or raise maxPodCapacity as described in Configure the limit range.

A configured quota value has no effect

Check the following:

The value is smaller than the calculated quota. APC uses the larger value.
The value isn’t a valid Kubernetes quantity. Check the APC API logs for a warning that names the key.
The Deployment hasn’t updated since you saved the change. See When the change applies.
In helm.airflow.quotas, the key is nested instead of flat. Write limits.cpu: "40", not limits: { cpu: "40" }.

Pods stay in the Pending state

The quota admitted the Pods, but no node has enough free CPU, memory, or ephemeral storage. Run kubectl describe pod <pod-name> -n <namespace> to see the scheduler’s reason.

Behavior in earlier versions

In APC 2.0 and 2.1:

Operator-mode Deployments have no ResourceQuota or LimitRange.
The ephemeral storage quota is the configured helm.airflow.quotas value, 50Gi by default, whatever the Deployment’s components need.
APC ignores pods, CPU, and memory values in helm.airflow.quotas and uses the calculated quota.
The quotas argument replaces the calculated pods, CPU, and memory quota, even when it’s smaller. APC ignores its ephemeral storage keys.

Starting in APC 2.2, every quota key is the largest of the calculated quota, helm.airflow.quotas, and the quotas argument, in both Helm and operator mode, and operator-mode Deployments get both objects.

​How the quota affects a Deployment

​How APC calculates the quota

​How configured values combine with the calculated quota

​Where you can change the quota

​Raise the quota for a scope with config governance

​When the change applies

​Raise the quota for one Deployment with the quotas argument

​Give task Pods room with extra capacity

​Change what APC calculates

​Configure the limit range

​Turn off quota and limit range management

​View the quota

​Permissions

​Troubleshoot

​Kubernetes refuses a Pod with exceeded quota

​Kubernetes refuses a Pod that’s larger than the maximum Pod size

​A configured quota value has no effect

​Pods stay in the Pending state

​Behavior in earlier versions

How the quota affects a Deployment

How APC calculates the quota

How configured values combine with the calculated quota

Where you can change the quota

Raise the quota for a scope with config governance

When the change applies

Raise the quota for one Deployment with the quotas argument

Give task Pods room with extra capacity

Change what APC calculates

Configure the limit range

Turn off quota and limit range management

View the quota

Permissions

Troubleshoot

Kubernetes refuses a Pod with exceeded quota

Kubernetes refuses a Pod that’s larger than the maximum Pod size

A configured quota value has no effect

Pods stay in the Pending state

Behavior in earlier versions