Autoscale Remote Execution Agent workers on queue depth

Airflow 3This feature is only available for Airflow 3.x Deployments.

Each Remote Execution Agent worker Pod runs a fixed number of concurrent tasks. The syncSlots value sets that number, and each concurrent task uses one slot. A worker that has no free slot does not accept more tasks, and the extra tasks stay in the queue. Tasks that wait on an external system, such as a warehouse query or an API call, use very little CPU and memory. These tasks fill every slot on a worker while CPU utilization stays low. A Horizontal Pod Autoscaler (HPA) that scales on CPU or memory does not add replicas in this state, so the queue grows and task latency increases. This document shows how to scale worker Pods on queue depth instead. Queue depth is the number of tasks in the queued and running states.

Choose a scaling method

The Remote Execution Agent Helm chart includes support for two approaches:

Method	How it works	Choose it when…
KEDA	The chart creates a KEDA `ScaledObject` that reads each worker’s queue backlog from the Sentinel service. KEDA creates and owns the HPA.	…KEDA can be installed in the cluster.
prometheus-adapter	An adapter republishes the queue metric on the Kubernetes custom metrics API. The chart creates an HPA.	…you cannot install KEDA or do not wish to. …your platform already runs Prometheus and prometheus-adapter. …you cannot enable the Sentinel service.

Astronomer recommends you use KEDA autoscaling whenever possible. This method needs no Prometheus and no metrics adapter, all configuration stays in values.yaml, and it is the only method that supports scale-to-zero. You can only use one method (HPA or KEDA) for each worker deployment. Learn more about:

Autoscale with KEDA.
Autoscale with HPA and prometheus-adapter.

Slot capacity and queue depth

The task capacity of a worker deployment is syncSlots multiplied by the number of replicas. A deployment with syncSlots: 20 and two replicas can run 40 tasks at the same time. Use desired capacity to configure autoscaling:

Set maxReplicaCount to your peak number of concurrent tasks divided by syncSlots, rounded up. For a peak of 200 concurrent tasks and syncSlots: 20, set maxReplicaCount: 10.
If you are using HPA, set the metric target explicitly to the same value as the worker’s syncSlots. (You do not need to configure this explicitly with KEDA: the chart configures it for you.)
Set minReplicaCount to the capacity you want available at all times. The minimum value of minReplicaCount is 1 with HPA and 0 with KEDA.

Raising syncSlots is the other way to add capacity for I/O-bound tasks. A slot holds a task process, so more slots need more memory on the worker Pod. Test a higher value against your own tasks before you use it in production.

Prerequisites

KEDA-based autoscaling requires:

Helm chart version 2.4.0 or later.
Remote Execution Agent version 1.8.4 or later.
Sentinel enabled with sentinel.enabled: true. See Sentinel for Remote Execution Agents.
KEDA must be installed in the cluster. See Deploying KEDA.

Autoscaling with HPA and prometheus-adapter requires:

Remote Execution Agent version 1.7.0 or later.
A Prometheus instance that scrapes the agent worker Pods. See Scrape metrics from Remote Execution Agents.
prometheus-adapter, which serves the metric through the Kubernetes custom metrics API.

Autoscale with KEDA (recommended)

Simply set keda.enabled: true on a worker configuration, and the chart will create a KEDA ScaledObject for that worker’s Deployment, pre-configured to scale the worker based on queue depth. The backlog metric is provided by the Sentinel, so even when workers are scaled to zero KEDA can still query the backlog and will scale back up when work arrives.

The chart does not install KEDA or its CRDs, and it does not check that they are present. Install KEDA before you enable this feature. Without the CRDs, helm upgrade will fail because the cluster does not recognize the ScaledObject resource. With the CRDs but no running KEDA operator, the chart applies the ScaledObject and nothing acts on it.

Enable KEDA on the worker

Configure the worker in values.yaml:

values.yaml

sentinel:
  enabled: true

workers:
  - name: default-worker
    queues: "default"
    syncSlots: 20
    keda:
      enabled: true
      minReplicaCount: 0
      maxReplicaCount: 10
    terminationGracePeriodSeconds: 600

minReplicaCount defaults to 0, which lets the worker scale to zero when its queues are empty. Set it higher to keep idle capacity ready for the first tasks of a run.maxReplicaCount defaults to 10. This default caps the Deployment at 10 × syncSlots concurrent tasks, so set it from your own peak concurrency if needed.

To avoid task interruptions, you must set terminationGracePeriodSeconds to a value longer than the longest-running task in your workflow. If you have tasks that can run for 8h, set terminationGracePeriodSeconds: 28800.

Consult the chart’s default values.yaml (e.g. via helm show values astronomer/astro-remote-execution-agent) for additional configuration options you can use to tweak KEDA’s behavior.

Apply the configuration and check the scaling

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
kubectl get scaledobject -n re

The READY and ACTIVE columns of the ScaledObject report whether KEDA can read the backlog metric. ACTIVE is False for an idle worker, which is the state in which KEDA holds the Deployment at minReplicaCount.

Autoscale with HPA and prometheus-adapter

This method uses the astro_agent_client_queue_stats metric, which each agent worker exposes on its /metrics endpoint, and it needs Prometheus and prometheus-adapter in the cluster. Only one adapter can serve custom.metrics.k8s.io in a cluster, so this method is unavailable if another component already occupies that API.

Scrape the workers with the Deployment name in the job label

The adapter rule in the next step maps the job label to a Kubernetes Deployment, so job must hold the name of the worker Deployment. The Helm chart creates one Service for each worker, labeled deploymentName: <resourceNamePrefix>-worker-<name>, which a ServiceMonitor can copy into job:

worker-servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: astro-agent-workers
  namespace: re
spec:
  jobLabel: deploymentName
  selector:
    matchLabels:
      app: astro-agent
      component: worker
  endpoints:
    - port: http
      path: /metrics

If you use a standalone Prometheus with static scrape configs, relabel the target so that job holds the same value.

Publish the queue metric through prometheus-adapter

Add the following rule to your prometheus-adapter configuration:

prometheus-adapter-values.yaml

rules:
  - seriesQuery: '{__name__="astro_agent_client_queue_stats",container!="POD",namespace!="",pod!=""}'
    resources:
      overrides:
        job:
          resource: deployment
        namespace:
          resource: namespace
    metricsQuery: sum by (job) (max by (job, queue, state) (<<.Series>>{state=~"queued|running", <<.LabelMatchers>>}))
    name:
      matches: ".*astro_agent_client_queue_stats.*"
      as: "astro_agent_client_queued_or_running_tasks"

The counts come from the Astro orchestration plane in the worker heartbeat response, so every Pod of a worker Deployment reports the same values. max by (job, queue, state) removes the duplicate series, and sum by (job) adds the queued and running series together into one queue-depth value for each worker Deployment.

Confirm that the custom metric is available

Query the custom metrics API for the worker Deployment. Replace re with your namespace and astro-worker-default-worker with your Deployment name:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/re/deployments.apps/astro-worker-default-worker/astro_agent_client_queued_or_running_tasks"

The response contains a value field with the current queue depth. An error means that the adapter rule or the job label does not match. Fix this before you enable the HPA.

Enable the HPA on the worker

Configure the worker in values.yaml:

values.yaml

workers:
  - name: default-worker
    queues: "default"
    syncSlots: 20
    hpa:
      enabled: true
      minReplicaCount: 2
      maxReplicaCount: 10
      metric:
        enabled: true
        name: astro_agent_client_queued_or_running_tasks
        target:
          type: AverageValue
          averageValue: "20"
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Pods
              value: 1
              periodSeconds: 60
        scaleDown:
          stabilizationWindowSeconds: 300
    terminationGracePeriodSeconds: 600

The averageValue target of 20 matches syncSlots: 20, so the HPA runs one replica for each 20 tasks in the queue. See Metric target types for why queue depth needs an AverageValue target.The chart ignores replicas when hpa.enabled is true.

Apply the configuration and check the HPA

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
kubectl get hpa -n re

The TARGETS column shows the current queue depth against the target. If it shows <unknown>, run kubectl describe hpa -n re to see which metric the HPA cannot read.

Metric target types

Kubernetes computes the replica count differently for each target type. Use AverageValue for queue depth.

Target type	Replica count	Use it for
`AverageValue`	`metric / target`, rounded up	A metric that counts work for the whole Deployment, such as queue depth
`Value`	`metric / target × current replicas`, rounded up	A metric that already describes a single replica

A Value target multiplies the ratio by the current replica count. With a queue-depth metric, the replica count therefore multiplies again in each scaling cycle for as long as the queue stays above the target, until the HPA reaches maxReplicaCount. For the full algorithm, see the Kubernetes HPA documentation.

Combine queue depth with resource metrics

Queue depth measures outstanding work, not the load on a Pod. Tasks that use a lot of memory can still exhaust a worker before its slots are full. With KEDA, add a cpu or memory trigger to the generated ScaledObject with keda.extraTriggers, which takes a list of KEDA trigger objects:

values.yaml

    keda:
      enabled: true
      maxReplicaCount: 10
      extraTriggers:
        - type: memory
          metricType: Utilization
          metadata:
            value: "80"

For the full list of triggers, see the KEDA scalers reference. With an HPA, add resource metrics with hpa.extraMetrics, which takes Kubernetes HPA metric objects:

values.yaml

    hpa:
      enabled: true
      minReplicaCount: 2
      maxReplicaCount: 10
      metric:
        enabled: true
        name: astro_agent_client_queued_or_running_tasks
        target:
          type: AverageValue
          averageValue: "20"
      extraMetrics:
        - type: Resource
          resource:
            name: memory
            target:
              type: Utilization
              averageUtilization: 80

In both cases the HPA evaluates every metric and uses the highest replica count that any of them recommends. Resource metrics need resources.requests on the worker, because the HPA calculates utilization against the request.

Scale down without task failures

A worker that stops during a task fails that task. These settings protect running work:

terminationGracePeriodSeconds on the worker gives a Pod time to finish its tasks before Kubernetes stops it. The default is 600. Set it higher than your longest task if your tasks run for more than 10 minutes.
keda.cooldownPeriod makes KEDA wait after the backlog last justified more than minReplicaCount replicas before it scales back down. The default is 300.
hpa.behavior.scaleDown.stabilizationWindowSeconds makes the HPA use the highest replica count it recommended over the trailing window, so a brief dip in queue depth does not remove a worker. The HPA examples on this page use 300.

A queue of short tasks makes queue depth dip and recover between scaling cycles, which is what these windows absorb.

Overview

Get started

Develop DAGs

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Autoscale Remote Execution Agent workers on queue depth

Choose a scaling method

Slot capacity and queue depth

Prerequisites

Autoscale with KEDA (recommended)

Autoscale with HPA and prometheus-adapter

Metric target types

Combine queue depth with resource metrics

Scale down without task failures

​Choose a scaling method

​Slot capacity and queue depth

​Prerequisites

​Autoscale with KEDA (recommended)

​Autoscale with HPA and prometheus-adapter

​Metric target types

​Combine queue depth with resource metrics

​Scale down without task failures

​Related documentation

Choose a scaling method

Slot capacity and queue depth

Prerequisites

Autoscale with KEDA (recommended)

Autoscale with HPA and prometheus-adapter

Metric target types

Combine queue depth with resource metrics

Scale down without task failures

Related documentation