Autoscale Remote Execution Agent workers on queue depth

Airflow 3This feature is only available for Airflow 3.x Deployments.

Each Remote Execution Agent worker Pod runs a fixed number of concurrent tasks. The syncSlots value sets that number, and each concurrent task uses one slot. A worker that has no free slot does not accept more tasks, and the extra tasks stay in the queue. Tasks that wait on an external system, such as a warehouse query or an API call, use very little CPU and memory. These tasks fill every slot on a worker while CPU utilization stays low. A Horizontal Pod Autoscaler (HPA) that scales on CPU or memory does not add replicas in this state, so the queue grows and task latency increases. This document shows how to scale worker Pods on queue depth instead. Queue depth is the number of tasks in the queued and running states, which each agent worker reports in the astro_agent_client_queue_stats metric. It covers two setups: prometheus-adapter with the HPA that the Helm chart creates, and KEDA for clusters that do not run prometheus-adapter.

Slot capacity and queue depth

The task capacity of a worker Deployment is syncSlots multiplied by the number of replicas. A Deployment with syncSlots: 20 and two replicas runs 40 tasks at the same time. Use this capacity to size the autoscaler:

Set maxReplicaCount to your peak number of concurrent tasks divided by syncSlots, rounded up. For a peak of 200 concurrent tasks and syncSlots: 20, set maxReplicaCount: 10.
Set the metric target to syncSlots. The autoscaler then adds one replica for each syncSlots tasks of work in the queue.
Set minReplicaCount to the capacity you want available before the autoscaler reacts. Each scaling decision takes at least one metric interval, and a new worker Pod takes time to start.

Raising syncSlots is the other way to add capacity for I/O-bound tasks. A slot holds a task process, so more slots need more memory on the worker Pod. Test a higher value against your own tasks before you use it in production.

Prerequisites

A Remote Execution Agent that runs Agent Client version 1.7.0 or later, which exposes the /metrics endpoint. See Register and configure agents.
A Prometheus instance that scrapes the agent worker Pods. See Scrape metrics from Remote Execution Agents.
Permission to install cluster components and to run helm upgrade against the agent release.
One of the following:
- prometheus-adapter, which serves the metric through the Kubernetes custom metrics API.
- KEDA, which reads Prometheus directly and creates its own HPA.

Choose a scaling method

Method	How it works	Choose it when
prometheus-adapter	An adapter publishes the queue metric on the Kubernetes custom metrics API. The Helm chart creates the HPA from `workers[].hpa`.	Your platform runs prometheus-adapter, or you can install it. All autoscaling configuration stays in `values.yaml`.
KEDA	A `ScaledObject` queries Prometheus directly. KEDA creates and owns the HPA.	Your platform has no custom metrics API, or another component already occupies it. Only one adapter can serve `custom.metrics.k8s.io` in a cluster.

Do not use both methods for the same worker Deployment. Two autoscalers that target one Deployment overwrite each other’s decisions.

Scale with prometheus-adapter

Scrape the workers with the Deployment name in the job label

The adapter rule in the next step maps the job label to a Kubernetes Deployment, so job must hold the name of the worker Deployment. The Helm chart creates one Service for each worker, labeled deploymentName: <resourceNamePrefix>-worker-<name>, which a ServiceMonitor can copy into job:

worker-servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: astro-agent-workers
  namespace: re
spec:
  jobLabel: deploymentName
  selector:
    matchLabels:
      app: astro-agent
      component: worker
  endpoints:
    - port: http
      path: /metrics

If you use a standalone Prometheus with static scrape configs, relabel the target so that job holds the same value.

Publish the queue metric through prometheus-adapter

Add the following rule to your prometheus-adapter configuration:

prometheus-adapter-values.yaml

rules:
  - seriesQuery: '{__name__="astro_agent_client_queue_stats",container!="POD",namespace!="",pod!=""}'
    resources:
      overrides:
        job:
          resource: deployment
        namespace:
          resource: namespace
    metricsQuery: sum by (job) (max by (job, queue, state) (<<.Series>>{state=~"queued|running", <<.LabelMatchers>>}))
    name:
      matches: ".*astro_agent_client_queue_stats.*"
      as: "astro_agent_client_queued_or_running_tasks"

The counts come from the Astro orchestration plane in the worker heartbeat response, so every Pod of a worker Deployment reports the same values. max by (job, queue, state) removes the duplicate series, and sum by (job) adds the queued and running series together into one queue-depth value for each worker Deployment.

Confirm that the custom metric is available

Query the custom metrics API for the worker Deployment. Replace re with your namespace and astro-worker-default-worker with your Deployment name:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/re/deployments.apps/astro-worker-default-worker/astro_agent_client_queued_or_running_tasks"

The response contains a value field with the current queue depth. An error means that the adapter rule or the job label does not match. Fix this before you enable the HPA.

Enable the HPA on the worker

Configure the worker in values.yaml:

values.yaml

workers:
  - name: default-worker
    queues: "default"
    syncSlots: 20
    hpa:
      enabled: true
      minReplicaCount: 2
      maxReplicaCount: 10
      metric:
        enabled: true
        name: astro_agent_client_queued_or_running_tasks
        target:
          type: AverageValue
          averageValue: "20"
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Pods
              value: 1
              periodSeconds: 60
        scaleDown:
          stabilizationWindowSeconds: 300
    terminationGracePeriodSeconds: 600

The averageValue target of 20 matches syncSlots: 20, so the HPA runs one replica for each 20 tasks in the queue. See Metric target types for why queue depth needs an AverageValue target.The chart ignores replicas when hpa.enabled is true.

Apply the configuration and check the HPA

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
kubectl get hpa -n re

The TARGETS column shows the current queue depth against the target. If it shows <unknown>, run kubectl describe hpa -n re to see which metric the HPA cannot read.

Scale with KEDA

KEDA queries Prometheus directly, so the cluster needs no custom metrics adapter. KEDA creates the HPA for the worker Deployment and owns it.

Turn off the HPA in the Helm chart

Set hpa.enabled: false for the worker, so that the chart and KEDA do not create two autoscalers for one Deployment:

values.yaml

workers:
  - name: default-worker
    queues: "default"
    syncSlots: 20
    replicas: 2
    hpa:
      enabled: false

Keep replicas equal to the minReplicaCount you set in the next step. Each helm upgrade writes replicas back to the Deployment, and KEDA restores the scaled count at its next poll.

Create a ScaledObject for the worker Deployment

worker-scaledobject.yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: astro-worker-default-worker
  namespace: re
spec:
  scaleTargetRef:
    name: astro-worker-default-worker
  minReplicaCount: 2
  maxReplicaCount: 10
  pollingInterval: 30
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300
  triggers:
    - type: prometheus
      metricType: AverageValue
      metadata:
        serverAddress: http://prometheus-operated.monitoring.svc.cluster.local:9090
        threshold: "20"
        query: |-
          sum(max by (queue, state) (astro_agent_client_queue_stats{namespace="re",job="astro-worker-default-worker",state=~"queued|running"}))

scaleTargetRef.name is the name of the worker Deployment, which the chart builds as <resourceNamePrefix>-worker-<worker name>.threshold matches syncSlots, and the default AverageValue metric type divides the query result by the threshold. KEDA therefore runs one replica for each 20 tasks in the queue.The query must return a single value. Adjust the label matchers to your own scrape configuration, and add a matcher for the worker Deployment when one Prometheus instance scrapes several of them. Read the labels from a worker’s /metrics endpoint before you write the query.

Apply the ScaledObject and check the scaling

kubectl apply -f worker-scaledobject.yaml
kubectl get scaledobject -n re
kubectl get hpa -n re

The READY and ACTIVE columns of the ScaledObject report whether KEDA can read the metric. KEDA names the HPA that it creates keda-hpa-<scaled-object-name>.

Do not set minReplicaCount: 0 for an agent worker. The queue metric comes from the worker Pods, so a Deployment that scales to zero exposes no metric and never scales back up.

Other KEDA scalers

The prometheus trigger fits any platform that keeps the agent metrics in Prometheus or in a Prometheus-compatible service, such as Amazon Managed Service for Prometheus, Azure Monitor managed service for Prometheus, or Google Cloud Managed Service for Prometheus. If your queue depth lives in another system, KEDA can read it with an external trigger, which calls a gRPC service that you run. The arithmetic stays the same: report the number of queued and running tasks, and set the threshold to syncSlots. For the list of triggers, see the KEDA scalers reference.

Metric target types

Kubernetes computes the replica count differently for each target type. Use AverageValue for queue depth.

Target type	Replica count	Use it for
`AverageValue`	`metric / target`, rounded up	A metric that counts work for the whole Deployment, such as queue depth
`Value`	`metric / target × current replicas`, rounded up	A metric that already describes a single replica

A Value target multiplies the ratio by the current replica count. With a queue-depth metric, the replica count therefore multiplies again in each scaling cycle for as long as the queue stays above the target, until the HPA reaches maxReplicaCount. For the full algorithm, see the Kubernetes HPA documentation.

Combine queue depth with resource metrics

Queue depth measures outstanding work, not the load on a Pod. Tasks that use a lot of memory can still exhaust a worker before its slots are full. Add resource metrics with hpa.extraMetrics, which takes Kubernetes HPA metric objects:

values.yaml

    hpa:
      enabled: true
      minReplicaCount: 2
      maxReplicaCount: 10
      metric:
        enabled: true
        name: astro_agent_client_queued_or_running_tasks
        target:
          type: AverageValue
          averageValue: "20"
      extraMetrics:
        - type: Resource
          resource:
            name: memory
            target:
              type: Utilization
              averageUtilization: 80

The HPA evaluates every metric and uses the highest replica count that any of them recommends. Resource metrics need resources.requests on the worker, because the HPA calculates utilization against the request. With KEDA, add a second trigger of type memory or cpu to the same ScaledObject.

Scale down without task failures

A worker that stops during a task fails that task. Two settings protect running work:

terminationGracePeriodSeconds on the worker gives a Pod time to finish its tasks before Kubernetes stops it. The default is 600. Set it higher than your longest task if your tasks run for more than 10 minutes.
behavior.scaleDown.stabilizationWindowSeconds makes the autoscaler wait before it removes replicas. The examples on this page use 300.

A queue of short tasks therefore makes the value dip and recover between scaling cycles. scaleDown.stabilizationWindowSeconds makes the HPA use the highest replica count it recommended over the trailing window, so a brief dip does not remove a worker.

Overview

Get started

Develop DAGs

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Autoscale Remote Execution Agent workers on queue depth

Slot capacity and queue depth

Prerequisites

Choose a scaling method

Scale with prometheus-adapter

Scale with KEDA

Other KEDA scalers

Metric target types

Combine queue depth with resource metrics

Scale down without task failures

​Slot capacity and queue depth

​Prerequisites

​Choose a scaling method

​Scale with prometheus-adapter

​Scale with KEDA

​Other KEDA scalers

​Metric target types

​Combine queue depth with resource metrics

​Scale down without task failures

​Related documentation

Slot capacity and queue depth

Prerequisites

Choose a scaling method

Scale with prometheus-adapter

Scale with KEDA

Other KEDA scalers

Metric target types

Combine queue depth with resource metrics

Scale down without task failures

Related documentation