Configure DAG sources

Remote Execution Agents require configuration to access your DAG code. This guide covers configuring DAG bundles, which are collections of DAG files and supporting code introduced in Airflow 3.

This feature requires Airflow 3.x Deployments. Configuring multiple DAG bundles in a single Deployment is only supported in Remote Execution mode.

DAG bundle types

Choose between two types of DAG bundles:

GitDagBundle: Dags stored in a Git repository (recommended for production)
LocalDagBundle: Dags stored in the container image or persistent volume (default)

When to use each bundle type

Use GitDagBundle when:

Running production deployments
Tracking DAG versions with full rerun capabilities
Storing dags in version control systems
Managing multiple teams or DAG repositories

Use LocalDagBundle when:

Running development or testing environments
Building dags into container images
Using existing PVC-based DAG management
Preferring simpler configuration

See GitDagBundle compared to LocalDagBundle for functional differences.

Dag hashing

The Dag processor computes a hash of each Dag and caches it. On each processing cycle, the Dag processor compares the current hash of a Dag against the cached value:

If the hash matches, the Dag is unchanged, so the Dag processor skips it instead of re-sending it to the Astro orchestration plane.
If the Dag is new, changed, or not yet in the cache, the Dag processor processes it and updates the cache.

Because the Dag processor skips unchanged Dags, Dag hashing lowers memory utilization and network bandwidth, and speeds up Dag updates in Deployments with many Dags. To monitor cache behavior, use the dag_processor_cache_hits_total, dag_processor_cache_misses_total, and dag_processor_cache_size metrics. See Dag processor metrics.

Dag hashing functionality is enabled by default, starting on Astro Agent client release 1.8.0 and later. You can disable that feature at any time with the following environment variable: ASTRO_AGENT_CLIENT_DAG_PROCESSOR__ENABLE_DAG_CACHING=False.

Configure GitDagBundle

GitDagBundle fetches dags from Git repositories and provides automatic versioning capabilities.

GitDagBundle is recommended for production Remote Execution deployments.

Supported authentication methods

GitDagBundle supports the following authentication methods:

Access tokens (personal access tokens, OAuth tokens)
SSH keys
SSH agent

Choose the method that aligns with your security requirements and infrastructure.

Required token permissions by provider

When you create an access token, grant the minimum permissions required to read repository contents:

Provider	Token type	Required permissions
GitHub	Fine-grained PAT	`Contents: Read-only` on the target repository. `Metadata: Read-only` is included automatically
GitHub	Classic PAT	`repo` scope for private repositories. Public repositories require no scope
GitLab	PAT	`read_repository`
Bitbucket	App password	`Repositories: Read`

Configure public repository

For public repositories, no authentication configuration is required. Configure only the repository URL and tracking reference:

values.yaml

dagBundleConfigList: '[{"name": "public-dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"repo_url": "https://github.com/your-org/public-dags", "tracking_ref": "main", "subdir": "dags"}}]'

Configure private repository

For private repositories, configure both the DAG bundle and an Airflow connection for authentication.

Create Git connection

Add an Airflow connection environment variable in values.yaml. The connection name suffix must match the git_conn_id value in your DAG bundle configuration.

Access token (HTTPS)
SSH key (deploy key)

Use this method with a Personal Access Token (PAT) or OAuth token. Set login to your Git username and password to the token value.

values.yaml

commonEnv:
  - name: AIRFLOW_CONN_GIT_REPO
    value: >-
      {
        "conn_type": "git",
        "login": "<git-username>",
        "password": "<personal-access-token>",
        "host": "github.com",
        "schema": "https",
        "extra": {
          "repo": "<your-org>/<private-repo>",
          "branch": "main"
        }
      }

See Required token permissions by provider for the minimum permissions each provider requires.

Use this method with an SSH deploy key. Set login to git and provide the private key in the extra field.

values.yaml

commonEnv:
  - name: AIRFLOW_CONN_GIT_REPO
    value: >-
      {
        "conn_type": "git",
        "login": "git",
        "host": "<your-git-host>",
        "schema": "ssh",
        "extra": {
          "private_key": "<private-ssh-key>"
        }
      }

The connection name AIRFLOW_CONN_GIT_REPO creates a connection with ID git_repo. This ID must match the git_conn_id value in your DAG bundle configuration.

For production environments, store connection credentials in a secrets backend instead of values.yaml. See Use a secrets backend for Git credentials for an example using Azure Key Vault.

Configure DAG bundle

Configure the DAG bundle with matching git_conn_id:

values.yaml

dagBundleConfigList: '[{"name": "private-dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"tracking_ref": "main", "subdir": "dags", "repo_url": "https://github.com/<your-org>/<private-repo>.git", "git_conn_id": "git_repo"}}]'

Note that git_conn_id: "git_repo" matches the connection ID from the AIRFLOW_CONN_GIT_REPO environment variable.

Update Helm release

Apply the configuration:

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Configure refresh interval

Control how frequently agents check for repository updates using the refresh_interval parameter:

values.yaml

dagBundleConfigList: '[{"name": "private-dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"repo_url": "https://github.com/your-org/private-dags", "tracking_ref": "main", "subdir": "dags", "git_conn_id": "git_repo", "refresh_interval": 300}}]'

The default refresh interval is 300 seconds. Reducing this value across many bundles may increase the risk of hitting Git provider rate limits.

Use a secrets backend for Git credentials

For production environments, use a secrets backend to store Git connection credentials instead of hardcoding them in values.yaml. The following example shows how to configure Azure Key Vault with workload identity authentication on Azure AKS.

Configure Azure Key Vault as secrets backend

Add the secrets backend configuration to your values.yaml:

values.yaml

secretBackend: airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend

commonEnv:
  - name: AIRFLOW__SECRETS__BACKEND_KWARGS
    value: '{"connections_prefix": "airflow-connection", "variables_prefix": "airflow-variable", "vault_url": "<your-vault-url>", "workload_identity_tenant_id": "<your-tenant-id>", "managed_identity_client_id": "<your-managed-identity-client-id>"}'

This configuration uses Azure workload identity for authentication, which is the recommended approach for Azure AKS environments. For other authentication methods, see Azure Key Vault secrets backend.

Store Git connections in Azure Key Vault

Create secrets in Azure Key Vault for each Git connection. The secret name must follow the pattern <connections_prefix>-<connection-id>. For example, to create a connection with ID git-repo1-conn:

In Azure Key Vault, create a secret named airflow-connection-git-repo1-conn.
Set the secret value to a JSON connection string:

{
  "conn_type": "git",
  "login": "<git-username>",
  "password": "<personal-access-token>",
  "host": "github.com",
  "schema": "https",
  "extra": {
    "repo": "<your-org>/<your-repo>",
    "branch": "main"
  }
}

Repeat this process for each Git repository connection you need.

Configure DAG bundles with connection references

Configure your DAG bundles to reference the connections stored in Azure Key Vault:

values.yaml

dagBundleConfigList: '[
  {
    "name": "dags-folder",
    "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
    "kwargs": {
      "repo_url": "https://github.com/<your-org>/<repo-1>.git",
      "tracking_ref": "main",
      "subdir": "dags",
      "refresh_interval": 120,
      "git_conn_id": "git-repo1-conn"
    }
  },
  {
    "name": "dags-folder2",
    "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
    "kwargs": {
      "repo_url": "https://github.com/<your-org>/<repo-2>.git",
      "tracking_ref": "main",
      "subdir": "dags",
      "refresh_interval": 120,
      "git_conn_id": "git-repo2-conn"
    }
  }
]'

The git_conn_id values must match the connection IDs you created in Azure Key Vault (without the airflow-connection- prefix).

Update Helm release

Apply the configuration:

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Configure LocalDagBundle

LocalDagBundle reads dags from the local filesystem. This is the default dag bundle type.

DAG storage options

Choose one of two methods to provide dags to agents: Option 1: Include dags in container image Build a custom agent image that includes your DAG files. Copy dags into the /dags folder during image build. Option 2: Mount Persistent Volume Claim Create a PVC containing your dags and mount it into all agent components (Dag Processor, Worker, and Triggerer) at the same path.

Configure DAG path

LocalDagBundle looks for dags in /dags by default. Specify a different path using the path parameter:

values.yaml

dagBundleConfigList: '[{"name": "local-dags", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/opt/airflow/dags"}}]'

Configure with container image

Build custom image

Create a Dockerfile extending the base agent image:

Dockerfile

FROM images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0

# Copy dags into the image
COPY dags/ /dags/

# Install additional dependencies if needed
COPY requirements.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt

Update values file

Reference your custom image in values.yaml:

values.yaml

workers:
  - name: default-worker
    image: your-registry.example.com/custom-agent:1.0.0

dagProcessor:
  image: your-registry.example.com/custom-agent:1.0.0

triggerer:
  image: your-registry.example.com/custom-agent:1.0.0

dagBundleConfigList: '[{"name": "local-dags", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/dags"}}]'

Update Helm release

Apply the configuration:

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Configure with Persistent Volume Claim

Create PVC

Create a PersistentVolumeClaim in your Kubernetes namespace:

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dags-pvc
  namespace: <your-namespace>
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
  storageClassName: <your-storage-class>

Apply the PVC:

kubectl apply -f pvc.yaml

Configure volume mounts

Update values.yaml to mount the PVC into all components:

values.yaml

workers:
  - name: default-worker
    volumes:
      - name: dags-volume
        persistentVolumeClaim:
          claimName: dags-pvc
    volumeMounts:
      - name: dags-volume
        mountPath: /opt/airflow/dags
        readOnly: true

dagProcessor:
  volumes:
    - name: dags-volume
      persistentVolumeClaim:
        claimName: dags-pvc
  volumeMounts:
    - name: dags-volume
      mountPath: /opt/airflow/dags
      readOnly: true

triggerer:
  volumes:
    - name: dags-volume
      persistentVolumeClaim:
        claimName: dags-pvc
  volumeMounts:
    - name: dags-volume
      mountPath: /opt/airflow/dags
      readOnly: true

dagBundleConfigList: '[{"name": "local-dags", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/opt/airflow/dags"}}]'

Update Helm release

Apply the configuration:

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

GitDagBundle compared to LocalDagBundle

Both bundle types support DAG versioning in the Airflow UI, but GitDagBundle provides additional capabilities:

Scenario	LocalDagBundle	GitDagBundle
Viewing previous DAG runs	Displays DAG as it existed at run time	Displays DAG as it existed at run time
Creating new DAG runs	Uses current DAG code	Uses current DAG code
Rerunning whole previous DAG run	Uses current DAG code	Uses DAG version from original run time
Rerunning individual tasks	Uses latest version for rerun tasks	Uses task code from original run time
Code changes during DAG run	Uses current DAG code at task start time	Completes using bundle version from run start
Running backfills	Uses current DAG code	Uses latest bundle version
Version creation	Every structural DAG change creates new version	Every committed structural change creates new version

DAG versioning

Airflow 3 automatically tracks DAG versions when you use DAG bundles. Each DAG run associates with a specific DAG version visible in the Airflow UI. Key behaviors:

New versions are created for structural changes (tasks, dependencies, schedules)
The scheduler uses the latest DAG version to create new runs
You can view code for any previous DAG version in the UI
GitDagBundle allows rerunning tasks with their original code version

See Airflow DAG versioning for detailed information about versioning behavior.

Next steps

After configuring DAG sources:

Deploy Remote Execution project - Build and deploy your Airflow project
Configure logging - Set up task log collection
Configure OpenLineage - Enable data lineage tracking

Overview

Get started

Develop DAGs

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Configure DAG sources

DAG bundle types

When to use each bundle type

Dag hashing

Configure GitDagBundle

Supported authentication methods

Required token permissions by provider

Configure public repository

Configure private repository

Configure refresh interval

Use a secrets backend for Git credentials

Configure LocalDagBundle

DAG storage options

Configure DAG path

Configure with container image

Configure with Persistent Volume Claim

GitDagBundle compared to LocalDagBundle

DAG versioning

Next steps

​DAG bundle types

​When to use each bundle type

​Dag hashing

​Configure GitDagBundle

​Supported authentication methods

​Required token permissions by provider

​Configure public repository

​Configure private repository

​Configure refresh interval

​Use a secrets backend for Git credentials

​Configure LocalDagBundle

​DAG storage options

​Configure DAG path

​Configure with container image

​Configure with Persistent Volume Claim

​GitDagBundle compared to LocalDagBundle

​DAG versioning

​Next steps

​Related documentation

DAG bundle types

When to use each bundle type

Dag hashing

Configure GitDagBundle

Supported authentication methods

Required token permissions by provider

Configure public repository

Configure private repository

Configure refresh interval

Use a secrets backend for Git credentials

Configure LocalDagBundle

DAG storage options

Configure DAG path

Configure with container image

Configure with Persistent Volume Claim

GitDagBundle compared to LocalDagBundle

DAG versioning

Next steps

Related documentation