Helm chart configuration reference

This reference describes configuration values for the Remote Execution Agent Helm chart. For complete configuration options, see the values.yaml file downloaded from the Astro UI.

Required configuration values

The following values must be configured before installing the Helm chart:

Agent authentication

agentToken / agentTokenSecretName / agentTokenFile You must specify exactly one of these to provide the agent token generated in the Astro UI.

agentToken: Token value as plain text in values.yaml (not recommended for production)
agentTokenSecretName: Name of existing Kubernetes secret containing the token
agentTokenFile: Path to file containing the token (agent reads at runtime)

See Agent token configuration for detailed instructions.

Image registry access

imagePullSecretName / imagePullSecretData You must specify exactly one of these to allow agents to pull images from the registry.

imagePullSecretName: Name of existing Kubernetes secret with Docker credentials
imagePullSecretData: Docker config JSON as string (Helm creates secret named image-pull-secret)

See Image pull secret configuration for detailed instructions.

Kubernetes namespace

namespace Kubernetes namespace where the agent will be deployed.

If createNamespace: true, Helm creates the namespace
If createNamespace: false, namespace must exist before installation

If using agentTokenSecretName and imagePullSecretName, set createNamespace: false and create the namespace manually with secrets already present.

See Install in restricted Kubernetes namespace for restricted namespace configuration.

Resource name prefix

resourceNamePrefix Name prefix for all Kubernetes resources (Deployments, ConfigMaps, Secrets) created by the Helm chart.

Secrets backend

secretBackend Airflow secrets backend class for accessing connections and variables. Required for agent operation. Supported backends:

airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
airflow.providers.hashicorp.secrets.vault.VaultBackend
airflow.secrets.local_filesystem.LocalFilesystemBackend (not recommended for production)

See Configure secrets backend for detailed configuration instructions.

XCom backend

xcomBackend Airflow XCom backend class for passing data between tasks. Required for agent operation. Typically set to: airflow.providers.common.io.xcom.backend.XComObjectStorageBackend See Configure XCom backend for detailed configuration instructions.

State store backend

stateStoreBackend Airflow worker-side state store backend class for persisting task and asset state on Astro Runtime 3.3 and later. Available in Helm chart 2.3.0 and later. Defaults to airflow.providers.common.io.state_store.backend.StateStoreObjectStorageBackend with a local file path. For production, point the path at shared object storage. See Configure state store backend for detailed configuration instructions.

DAG bundles

dagBundleConfigList JSON string defining how agents access dag code. Required for running dags. See Configure DAG sources for detailed configuration instructions.

Common environment variables

commonEnv Environment variables applied to all agent components (worker, DAG processor, triggerer). Used to configure secrets backend parameters, XCom paths, logging settings, and other Airflow configuration. Example:

values.yaml

commonEnv:
  - name: AIRFLOW__SECRETS__BACKEND_KWARGS
    value: '{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}'
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
    value: "s3://bucket/xcom"

Worker resource configuration

workers Workers are configured as a list in values.yaml. Each entry defines a worker Deployment with its own name, resource allocation, replica count, and optional queue assignment.

Parameter	Description	Default
`name`	Unique identifier for the worker Deployment. Used in Kubernetes resource names.	`default-worker`
`replicas`	Number of worker Pod replicas. Ignored when `hpa.enabled` is `true`.	`1`
`queues`	Comma-separated list of Airflow queues this worker listens on.	`default`
`resources.requests.cpu`	Minimum CPU allocated to the worker Pod.	—
`resources.requests.memory`	Minimum memory allocated to the worker Pod.	—
`resources.limits.cpu`	Maximum CPU the worker Pod can use.	—
`resources.limits.memory`	Maximum memory the worker Pod can use.	—
`env`	List of environment variables specific to this worker.	`[]`
`volumes`	Additional volumes to mount on the worker Pod.	`[]`
`volumeMounts`	Mount paths for the additional volumes.	`[]`
`nodeSelector`	Kubernetes node selector for scheduling worker Pods.	`{}`
`tolerations`	Kubernetes tolerations for scheduling worker Pods.	`[]`
`serviceAccount.name`	Custom service account name. Overrides the default `{{ resourceNamePrefix }}-worker-{{ worker.name }}`.	—
`serviceAccount.create`	Whether the Helm chart creates the service account. Set to `false` when using a pre-existing service account.	`true`
`terminationGracePeriodSeconds`	The grace period for the worker Pod to finish existing tasks before terminating.	`600`

Example with two workers:

values.yaml

workers:
  - name: default-worker
    replicas: 2
    queues: "default"
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
  - name: high-memory-worker
    replicas: 1
    queues: "high-memory"
    resources:
      requests:
        cpu: "1"
        memory: "4Gi"
      limits:
        cpu: "4"
        memory: "16Gi"

When you configure multiple workers, each worker creates a separate Kubernetes Deployment. The service account name for each worker defaults to {{ resourceNamePrefix }}-worker-{{ worker.name }}. If you use IRSA (AWS), Workload Identity (GCP), or managed identity (Azure), annotate each worker’s service account.

Horizontal Pod Autoscaler

workers[].hpa Each worker supports a Horizontal Pod Autoscaler (HPA) configuration to automatically scale the number of worker Pod replicas based on resource utilization or custom metrics. When hpa.enabled is true, the Helm chart creates a HorizontalPodAutoscaler resource for the worker Deployment. The replicas value is ignored because the HPA controls replica count.

Parameter	Description	Default
`hpa.enabled`	Enable the Horizontal Pod Autoscaler for this worker.	`false`
`hpa.minReplicas`	Minimum number of worker Pod replicas.	`1`
`hpa.maxReplicas`	Maximum number of worker Pod replicas.	`10`
`hpa.metrics`	List of metric targets that the HPA uses to make scaling decisions. Follows the Kubernetes HPA metrics spec.	—

Example with CPU-based autoscaling:

values.yaml

workers:
  - name: default-worker
    queues: "default"
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
    hpa:
      enabled: true
      minReplicas: 1
      maxReplicas: 5
      metrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 80

You must set resources.requests for the metrics you use in HPA targets. For example, CPU-based autoscaling requires resources.requests.cpu to be set. Without resource requests, the HPA cannot calculate utilization percentages.

You can combine multiple metrics to define more sophisticated scaling behavior. The HPA evaluates all specified metrics and scales to the highest recommended replica count. Example with CPU and memory metrics:

    hpa:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
      metrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 75
        - type: Resource
          resource:
            name: memory
            target:
              type: Utilization
              averageUtilization: 80

Triggerer resource configuration

triggerer The triggerer runs deferred tasks asynchronously. Configure the triggerer to control replica count, async capacity, resource allocation, and Pod-level settings.

Parameter	Description	Default
`replicas`	Number of triggerer Pod replicas.	`1`
`asyncSlots`	Number of concurrent triggers the triggerer Pod can compute.	`1000`
`image`	Docker image for the triggerer. Defaults to the top-level `image` if not set.	—
`imagePullPolicy`	Image pull policy for the triggerer. Defaults to the top-level `imagePullPolicy` if not set.	—
`resources.limits.cpu`	Maximum CPU the triggerer Pod can use.	`1`
`resources.limits.ephemeral-storage`	Maximum ephemeral storage the triggerer Pod can use.	`1Gi`
`resources.limits.memory`	Maximum memory the triggerer Pod can use.	`2Gi`
`resources.requests.cpu`	Minimum CPU allocated to the triggerer Pod.	`1`
`resources.requests.ephemeral-storage`	Minimum ephemeral storage allocated to the triggerer Pod.	`1Gi`
`resources.requests.memory`	Minimum memory allocated to the triggerer Pod.	`2Gi`
`env`	List of environment variables specific to the triggerer.	`[]`
`livenessProbe`	Liveness probe configuration for the triggerer Pod.	See following example
`readinessProbe`	Readiness probe configuration for the triggerer Pod.	See following example
`podSecurityContext`	Pod security context for the triggerer Pod. By default, the agent runs as a non-root user with UID 50000 and group ID 50000.	`~`
`containerSecurityContext`	Security context for the triggerer container.	`{}`
`initContainers`	Init containers to add to the triggerer Pod.	`[]`
`extraContainers`	Sidecar containers to add to the triggerer Pod.	`[]`
`volumes`	Additional volumes to mount on the triggerer Pod.	`[]`
`volumeMounts`	Mount paths for the additional volumes.	`[]`
`nodeSelector`	Kubernetes node selector for scheduling triggerer Pods.	`~`
`affinity`	Affinity rules for the triggerer Pod.	`{}`
`tolerations`	Kubernetes tolerations for scheduling triggerer Pods.	`[]`

Example:

values.yaml

triggerer:
  replicas: 2
  asyncSlots: 500
  resources:
    limits:
      cpu: "1"
      ephemeral-storage: "1Gi"
      memory: "2Gi"
    requests:
      cpu: "1"
      ephemeral-storage: "1Gi"
      memory: "2Gi"
  livenessProbe:
    httpGet:
      path: healthz
      port: 39091
    initialDelaySeconds: 30
    periodSeconds: 5
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 5
  readinessProbe:
    httpGet:
      path: healthz
      port: 39091
    initialDelaySeconds: 30
    periodSeconds: 5
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 5
  podSecurityContext:
    runAsUser: 50000
    fsGroup: 50000
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containerSecurityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
  initContainers:
    - name: vault-agent
      image: vault:1.13.0
      command: ["vault", "agent", "-config=/etc/vault/config.hcl"]
      volumeMounts:
        - name: vault-config
          mountPath: /etc/vault
      env:
        - name: VAULT_ADDR
          value: "https://vault.example.com"
        - name: VAULT_TOKEN
          valueFrom:
            secretKeyRef:
              key: token
              name: vault-token
  extraContainers:
    - name: logging-sidecar
      image: timberio/vector:0.45.0-debian
      env:
        - name: VECTOR_CONFIG
          value: /etc/vector/vector.yaml
      volumeMounts:
        - name: vector-config
          mountPath: /etc/vector
      resources:
        limits:
          cpu: "0.5"
          memory: "256Mi"
        requests:
          cpu: "0.5"
          memory: "256Mi"
  volumes:
    - name: task-logs
      emptyDir: {}
    - name: vector-config
      configMap:
        name: vector-config
  volumeMounts:
    - name: task-logs
      mountPath: /var/log/airflow
      readOnly: true
  nodeSelector:
    node.kubernetes.io/instance-type: c5.large

To run multiple triggerers, increase replicas. The configured number of replicas runs continuously. For restricted namespaces with Pod security standards set to restricted, configure podSecurityContext and containerSecurityContext to meet your cluster’s requirements.

Optional configuration

Logging sidecar

loggingSidecar Optional sidecar for exporting task logs to external platforms or viewing logs in the Airflow UI before task completion. See Configure logging sidecar for configuration instructions.

OpenLineage

openLineage Optional configuration for data lineage collection.

You must configure OpenLineage to use Astro Observe with Remote Execution Deployments.

See Configure OpenLineage for configuration instructions.

Sentinel monitoring

sentinel Monitoring service for agent health reporting (agent version 1.2.0+). Astronomer recommends enabling Sentinel for all deployments. See Sentinel for Remote Execution Agents for configuration instructions.

Cloud provider annotations

annotations and labels Kubernetes annotations and labels to configure Pods to run using a specific IAM role (AWS), workload identity (GCP) or managed identity (Azure).

Helm commands

After the Remote Execution Agent is installed, any updates to the agent use the helm upgrade command.

Install agent

helm repo add astronomer https://helm.astronomer.io
helm repo update
helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Update agent

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

View current configuration

helm get values astro-agent

Overview

Get started

Develop DAGs

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Helm chart configuration reference

Required configuration values

Agent authentication

Image registry access

Kubernetes namespace

Resource name prefix

Secrets backend

XCom backend

State store backend

DAG bundles

Common environment variables

Worker resource configuration

Horizontal Pod Autoscaler

Triggerer resource configuration

Optional configuration

Logging sidecar

OpenLineage

Sentinel monitoring

Cloud provider annotations

Helm commands

Install agent

Update agent

View current configuration

​Required configuration values

​Agent authentication

​Image registry access

​Kubernetes namespace

​Resource name prefix

​Secrets backend

​XCom backend

​State store backend

​DAG bundles

​Common environment variables

​Worker resource configuration

​Horizontal Pod Autoscaler

​Triggerer resource configuration

​Optional configuration

​Logging sidecar

​OpenLineage

​Sentinel monitoring

​Cloud provider annotations

​Helm commands

​Install agent

​Update agent

​View current configuration

Required configuration values

Agent authentication

Image registry access

Kubernetes namespace

Resource name prefix

Secrets backend

XCom backend

State store backend

DAG bundles

Common environment variables

Worker resource configuration

Horizontal Pod Autoscaler

Triggerer resource configuration

Optional configuration

Logging sidecar

OpenLineage

Sentinel monitoring

Cloud provider annotations

Helm commands

Install agent

Update agent

View current configuration