Configure OpenLineage for a Remote Execution Agent

Airflow 3This feature is only available for Airflow 3.x Deployments.

OpenLineage enables you to access data lineage and provenance across your Airflow workflows for your Remote Execution Agent. Features like Observe and Astro Alerts require that you enable OpenLineage for your data pipelines. When you create your Remote Execution Agent, Astro automatically generates a Helm values.yaml file with OpenLineage configurations pre-filled. To set up OpenLineage, you need to configure an access credential for OpenLineage. This can be an Astro Deployment API token used as your OpenLineage API key. There are three methods you can use to add your API key to your Helm values:

Method 1 - Configure the API key as plain text. This stores your API key in your values.yaml file as plaintext, which is the simplest but least secure option. It’s appropriate for development or testing environments.
Method 2 - Use a pre-created Kubernetes secret. This procedure stores your API key separately from your values.yaml file, which provides more security than storing as plaintext. This option provides security with standard Kubernetes features.
Method 3 - Inject your API key with a secrets manager. This approach uses an init container to inject the Agent token into the Remote Execution Agent component Pods. This example uses the HashiCorp Vault Agent, but you can use your own secrets manager. This option provides enhanced security with the potential for secret rotation.

Prerequisites

A Kubernetes cluster
Helm
The values.yaml file downloaded from the Remote Execution Agent registration modal in the Astro UI
A Deployment API Token that uses a Custom Deployment role with Observe Ingest permissions

Setup

Step 1: Retrieve your OpenLineage variables

Open the Remote Agent registration

In the Astro UI, go to the Deployment page and choose Agent. Then click Register Remote Agent.

Download the values file

Click Download values.yaml file.

The downloaded Helm values file includes most OpenLineage variables pre-filled, so you only need to configure the OpenLineage API key.

Step 2: Configure the OpenLineage API key

Configure key as plaintext
Use pre-created Kubernetes secret
Use secrets manager

This method stores an Astro Deployment API token as your OpenLineage API key in plain text in your values.yaml file, so that the Remote Execution Agent Helm chart can use it to create a Kubernetes secret named openlineage-api-key-secret. This API key is base64-encoded in the Kubernetes secret.All Remote Execution Agent components — the worker, Dag processor, and triggerer — use this API key to authenticate with the OpenLineage endpoint.

Update your values file

Add the following OpenLineage configuration to your values.yaml file:

values.yaml

openLineage:
  # Enable OpenLineage integration
  enabled: true

  # Set your OpenLineage API key directly in the values file
  apiKey: "<OPENLINEAGE_API_KEY>"

  # Do NOT set apiKeySecret when using apiKey
  # apiKeySecret: ~

  # The following fields are prefilled in the values.yaml downloaded from the Astro UI
  url: "<OPENLINEAGE_URL>"
  namespace: "<ASTRO_DEPLOYMENT_NAMESPACE>"
  endpoint: "<OPENLINEAGE_ENDPOINT>"
  facetsEnvironmentVariables: '<OPENLINEAGE_FACETS_ENV_VARS>'

Install the Helm chart

Apply the chart using the values.yaml file with the following command:

helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml

When you use this method, the Remote Execution Agent Helm chart doesn’t create a new secret for the OpenLineage API key. Instead, it configures all Agent components — the worker, Dag processor, and triggerer — to use the existing secret to authenticate with the OpenLineage endpoint.The secret must have a key named api-key containing the OpenLineage API key. You can use an Astro Deployment API token as your OpenLineage API key.

Create the Kubernetes secret

Create a Kubernetes secret containing your OpenLineage API key:

kubectl create secret generic openlineage-api-key-secret \
  --from-literal=api-key=<OPENLINEAGE_API_KEY> \
  --namespace <YOUR_NAMESPACE>

Reference the secret from your values file

Configure your values.yaml file so that OpenLineage uses your pre-created secret:

values.yaml

openLineage:
  # Enable OpenLineage integration
  enabled: true

  # Do NOT set apiKey when using apiKeySecret
  # apiKey: ~

  # Reference the pre-created secret you created in the previous step
  apiKeySecret: "openlineage-api-key-secret"

  # The following fields are prefilled in the values.yaml downloaded from the Astro UI
  url: "<OPENLINEAGE_URL>"
  namespace: "<ASTRO_DEPLOYMENT_NAMESPACE>"
  endpoint: "<OPENLINEAGE_ENDPOINT>"
  facetsEnvironmentVariables: '<OPENLINEAGE_FACETS_ENV_VARS>'

Install the Helm chart with the referenced secret

Apply the chart using the values.yaml file with the following command:

helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml

You can also use a secrets manager to securely store your API keys. The following procedure specifically uses the HashiCorp Vault Agent.The Vault Agent init container runs before the main Remote Execution Agent container. The Vault Agent authenticates with Vault to retrieve the OpenLineage API key and then writes the API key to a file in the shared volume. The Remote Execution Agent container can then read the OpenLineage API key from the file using the OPENLINEAGE_API_KEY environment variable. Use your Astro Deployment API token as your OpenLineage API key.

Configure OpenLineage and init containers

Configure OpenLineage and add init containers for the dagProcessor, workers, and triggerer components:

values.yaml

openLineage:
  # Enable OpenLineage integration
  enabled: true

  # Don't set apiKey or apiKeySecret
  # apiKey: ~
  # apiKeySecret: ~

  # The following fields are prefilled in the values.yaml downloaded from the Astro UI
  url: "<OPENLINEAGE_URL>"
  namespace: "<ASTRO_DEPLOYMENT_NAMESPACE>"
  endpoint: "<OPENLINEAGE_ENDPOINT>"
  facetsEnvironmentVariables: '<OPENLINEAGE_FACETS_ENV_VARS>'

# Configure each component to use the Vault init container
dagProcessor:
  initContainers:
    - name: vault-openlineage
      image: hashicorp/vault:1.13.1
      command: ["/bin/sh", "-c"]
      args:
        - |
          export VAULT_ADDR=https://vault.example.com
          vault agent -config=/vault/config/agent.hcl
      volumeMounts:
        - name: vault-config
          mountPath: /vault/config
        - name: openlineage-volume
          mountPath: /vault/secrets
  volumes:
    - name: vault-config
      configMap:
        name: vault-agent-config
    - name: openlineage-volume
      emptyDir:
        medium: Memory
  volumeMounts:
    - name: openlineage-volume
      mountPath: /vault/secrets
  env:
    - name: OPENLINEAGE_API_KEY
      valueFrom:
        fileRef:
          path: /vault/secrets/openlineage-api-key

triggerer:
  initContainers:
    - name: vault-openlineage
      image: hashicorp/vault:1.13.1
      command: ["/bin/sh", "-c"]
      args:
        - |
          export VAULT_ADDR=https://vault.example.com
          vault agent -config=/vault/config/agent.hcl
      volumeMounts:
        - name: vault-config
          mountPath: /vault/config
        - name: openlineage-volume
          mountPath: /vault/secrets
  volumes:
    - name: vault-config
      configMap:
        name: vault-agent-config
    - name: openlineage-volume
      emptyDir:
        medium: Memory
  volumeMounts:
    - name: openlineage-volume
      mountPath: /vault/secrets
  env:
    - name: OPENLINEAGE_API_KEY
      valueFrom:
        fileRef:
          path: /vault/secrets/openlineage-api-key

workers:
  initContainers:
    - name: vault-openlineage
      image: hashicorp/vault:1.13.1
      command: ["/bin/sh", "-c"]
      args:
        - |
          export VAULT_ADDR=https://vault.example.com
          vault agent -config=/vault/config/agent.hcl
      volumeMounts:
        - name: vault-config
          mountPath: /vault/config
        - name: openlineage-volume
          mountPath: /vault/secrets
  volumes:
    - name: vault-config
      configMap:
        name: vault-agent-config
    - name: openlineage-volume
      emptyDir:
        medium: Memory
  volumeMounts:
    - name: openlineage-volume
      mountPath: /vault/secrets
  env:
    - name: OPENLINEAGE_API_KEY
      valueFrom:
        fileRef:
          path: /vault/secrets/openlineage-api-key

Create a ConfigMap for the Vault Agent

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-agent-config
  namespace: <YOUR_NAMESPACE>
data:
  agent.hcl: |
    auto_auth {
      method "kubernetes" {
        mount_path = "auth/kubernetes"
        config = {
          role = "astro-agent"
        }
      }
    }

    template {
      destination = "/vault/secrets/openlineage-api-key"
      contents = "{{ with secret \"secret/data/openlineage/api-key\" }}{{ .Data.data.key }}{{ end }}"
    }
EOF

Install the Helm chart with the Vault Agent

Apply the chart with the values.yaml file:

helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Read more about secrets backends on Astro.

Step 3: Set OpenLineage environment variables on the orchestration plane

Open the Deployment

In the Astro UI, open your Deployment.

Open the Environment tab

Click the Environment tab.

Add the OpenLineage variable

Click Edit Deployment Variables (or + New Environment Variable if you have no variables configured yet), then click Add Variable and add the following environment variable:

OPENLINEAGE_DISABLED=False

Apply changes

Click Update Environment Variables to save your changes.

Setting this variable ensures that all required OpenLineage events, including task and Dag run events, are collected from the scheduler, workers, Dag processor, and triggerer components. This provides complete lineage in Observe and Astro Alerts.

Overview

Get started

Develop DAGs

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Configure OpenLineage for a Remote Execution Agent

Prerequisites

Setup

Step 1: Retrieve your OpenLineage variables

Download the values file

Step 2: Configure the OpenLineage API key

Install the Helm chart

Reference the secret from your values file

Install the Helm chart with the referenced secret

Create a ConfigMap for the Vault Agent

Install the Helm chart with the Vault Agent

Step 3: Set OpenLineage environment variables on the orchestration plane

Open the Environment tab

Add the OpenLineage variable

Apply changes

​Prerequisites

​Setup

​Step 1: Retrieve your OpenLineage variables

​Download the values file

​Step 2: Configure the OpenLineage API key

​Install the Helm chart

​Reference the secret from your values file

​Install the Helm chart with the referenced secret

​Create a ConfigMap for the Vault Agent

​Install the Helm chart with the Vault Agent

​Step 3: Set OpenLineage environment variables on the orchestration plane

​Open the Environment tab

​Add the OpenLineage variable

​Apply changes

Prerequisites

Setup

Step 1: Retrieve your OpenLineage variables

Download the values file

Step 2: Configure the OpenLineage API key

Install the Helm chart

Reference the secret from your values file

Install the Helm chart with the referenced secret

Create a ConfigMap for the Vault Agent

Install the Helm chart with the Vault Agent

Step 3: Set OpenLineage environment variables on the orchestration plane

Open the Environment tab

Add the OpenLineage variable

Apply changes