Install Remote Execution Agents in a restricted kubernetes namespace

Airflow 3This feature is only available for Airflow 3.x Deployments.

You can install the Remote Execution Agent in a Kubernetes namespace with restricted pod security standards. Your organization might have different security standards for infrastructure supporting internal-only sandboxes compared to production environments. Kubernetes Pod Security Standards define different security levels for Pods:

Privileged: No restrictions (least secure)
Baseline: Prevents known privilege escalations
Restricted: Highly-constrained settings following security best practices (most secure)

The Restricted profile enforces the following limitations:

Runs containers as non-root users
Prevents privilege escalation
Drops all Linux capabilities
Uses read-only root filesystems when possible
Requires a runtime default seccomp profile

However, because of these limitations, you need to complete the following additional Remote Execution Agent configuration set up.

Step 1: Create a restricted namespace

Create a Namespace in your Kubernetes manifest with the following restricted Pod security standards:

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: astro-agent-restricted
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Step 2: Configure Global Security Settings

Modify your Agent’s values.yaml file to set global security context settings that apply to all Agent components’ Pods and containers:

values.yaml

# Global pod security context for all components
podSecurityContext:
  seccompProfile:
    type: RuntimeDefault
  runAsUser: 50000
  fsGroup: 50000
  runAsNonRoot: true

# Global container security context for all components
containerSecurityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

Step 3: Configure component-specific settings

When using the Agent in a restricted namespace, you must configure volume mounts because:

The container security context sets readOnlyRootFilesystem: true
These directories need write access during runtime
Using emptyDir volumes provides isolated, writable storage that meets security requirements

Worker configuration

values.yaml

workers:
  - name: default-worker
    # Other worker settings...

    # Required volumes for filesystem access
    volumes:
      - name: tmp
        emptyDir: {}
      - name: task-logs
        emptyDir: {}

    volumeMounts:
      - name: tmp
        mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file
      - name: task-logs
        mountPath: /usr/local/airflow/logs # Or the configured folder that holds the task logs

Dag processor configuration

values.yaml

dagProcessor:
  # Other Dag processor settings...

  # Required volumes for filesystem access
  volumes:
    - name: tmp
      emptyDir: {}

  volumeMounts:
    - name: tmp
      mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file

Triggerer configuration

values.yaml

triggerer:
  # Other triggerer settings...

  # Required volumes for filesystem access
  volumes:
    - name: tmp
      emptyDir: {}
    - name: task-logs
      emptyDir: {}

  volumeMounts:
    - name: tmp
      mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file
    - name: task-logs
      mountPath: /usr/local/airflow/logs # Or the configured folder that holds the task logs

Step 4: (Optional) Add logging sidecar configuration

values.yaml

loggingSidecar:
  # Other logging sidecar settings...

  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
  volumeMounts:
    - name: task-logs
      mountPath: /etc/airflow/logs # Or the configured folder that holds the task logs

⌘I

Overview

Get started

Develop DAGs

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Install Remote Execution Agents in a restricted kubernetes namespace

Step 1: Create a restricted namespace

Step 2: Configure Global Security Settings

Step 3: Configure component-specific settings

Worker configuration

Dag processor configuration

Triggerer configuration

Step 4: (Optional) Add logging sidecar configuration

​Step 1: Create a restricted namespace

​Step 2: Configure Global Security Settings

​Step 3: Configure component-specific settings

​Worker configuration

​Dag processor configuration

​Triggerer configuration

​Step 4: (Optional) Add logging sidecar configuration

Step 1: Create a restricted namespace

Step 2: Configure Global Security Settings

Step 3: Configure component-specific settings

Worker configuration

Dag processor configuration

Triggerer configuration

Step 4: (Optional) Add logging sidecar configuration