Skip to main content
You can deploy Dags to an Astro Private Cloud Deployment using git-sync. After setting up this feature, you can deploy Dags from a Git repository without any additional CI/CD. Dags deployed with git-sync automatically appear in the Airflow UI without requiring additional action or causing downtime. You can also roll back images with the Astro Private Cloud UI and APC API.
git-sync-relay RWX volume doesn’t work with azurefile-csi.
This guide provides details about setup options and the steps for configuring git-sync as a Dag deploy option.

​
Choose a git-sync strategy

When you configure git-sync, you must choose both a repo fetch mode and a repo share mode:

​
Repo fetch mode

Repo fetch mode determines how the git-sync relay retrieves changes from your Git repository. Choose one of the following options:

​
Poll mode

The git-sync relay checks the remote Git repository for changes at regular intervals. Use poll mode for repositories with frequent changes across branches. Tradeoff: Frequent polling generates unnecessary network traffic between your Deployment and the repository when changes are infrequent.

​
Webhook

The git-sync relay fetches changes only when a push event fires from the Git repository. Use Webhook mode for repositories that don’t change frequently, to avoid unnecessary network traffic between your Deployment and the repository. Tradeoff: If configured for a specific branch, the git-sync relay only downloads changes for that branch regardless of fetch mode. However, in Webhook mode, the webhook fires for every push event in the repository — not just pushes to the configured branch. This means a busy repository can still generate frequent webhook calls even when branch filtering is in place.

​
Repo share mode

Repo share mode determines how the git-sync relay distributes the synced repository to Airflow Pods in the Deployment. Choose one of the following options:

​
git-daemon

A git-daemon container serves the repository within the namespace using the Git protocol on port 9418. The Airflow Deployment contains a git-sync relay Pod with both a git-sync container that stores the Git repository and a git-daemon container that serves the repository to the namespace. Tradeoff: All Airflow containers must clone the repository at startup, which can cause significant network use with large repositories and increase startup time.

​
Shared volume

The git repository contents are stored on a ReadWriteMany (RWX) storage volume mounted into each Airflow Pod, which eliminates git clone activity between Pods. The git-sync relay Pod pulls from the external Git repository and writes to the RWX volume. Requirement: An RWX-compatible StorageClass volume. RWX-compatible StorageClasses aren’t included in standard Kubernetes. You must provision additional cloud infrastructure to support RWX volumes, and the configuration steps differ between cloud providers. See your cloud provider’s documentation for details.

​
Prerequisites

To enable the git-sync deploy feature, you need:
  • An Astro Private Cloud installation that runs Deployments with the Astronomer Runtime Helm chart. This is the default for most installations.
  • Permission to push new configuration changes to your Astro Private Cloud installation.
  • (Shared volume mode) A ReadWriteMany (RWX) compatible StorageClass volume. RWX-compatible StorageClasses require additional cloud infrastructure that varies between providers. See your cloud provider’s documentation for configuration steps.
To configure a git-sync deploy mechanism for a Deployment on APC, you need Workspace Editor permissions. To deploy Dags to a Deployment using a git-sync deploy mechanism, you need permission to push code to a Git repository configured for git-sync deploys.

​
Enable git-sync

Git-sync deploys must be explicitly selected in the UI for each Airflow Deployment, for both git-daemon and shared-volume repo share modes. However, for the shared-volume mode, an APC Admin must configure the RWX shared volume storage class name, storageClassName, in the cluster configuration. To do so:
  1. In the APC UI, go to your Clusters page and select your cluster.
  2. In the cluster details, click Edit and add the following override to the Cluster Deployments Configuration field, including the path to your RWX compatible storage:
    For details on using the UI for configuration, see Override base configuration.
  3. Save and apply your changes in the UI.
For shared-volume repo share mode, an APC Admin must also set the RWX shared-volume storage class name, storageClassName, under deployMechanisms.gitSyncRelay. This is a global default; the repo share mode itself is still chosen per Deployment in the UI.

​
Configure your APC Deployment

Workspace editors can configure a new or existing Airflow Deployment to use a git-sync mechanism for Dag deploys. From there, any member of your organization with write permissions to the Git repository can deploy Dags to the Deployment. To configure a Deployment for git-sync deploys:
  1. In the Astro Private Cloud UI, create a new Airflow Deployment or open an existing one.
  2. Go to the Dag Deployment section of the Deployment’s Settings page.
  3. For your Mechanism, select Git Sync.
  4. Configure the following values:
    • Repository URL: The URL for the Git repository that hosts your Astro project. Use an SSH URL (for example, git@github.com:org/repo.git).
    • Branch Name: The name of the Git branch that you want to sync with your Deployment
    • Ssh Key: The SSH private key for your Git repository
    • Known Hosts: The public key for your Git provider, which can be retrieved using ssh-keyscan -t rsa <provider-domain>. For an example of how to retrieve GitHub’s public key, refer to Apache Airflow documentation.
    • Authentication Method: (APC 2.1 and later) How git-sync authenticates to the repository. Choose one of:
      • SSH: Clone over SSH using a private key — the same Ssh Key and Known Hosts fields described previously.
      • HTTPS + Personal Access Token: Clone a private repository over HTTPS using a personal access token. Requires Personal Access Token and, optionally, HTTPS Username. Set Repository URL to an https:// URL instead of an SSH URL. See Authenticate to a private repository over HTTPS.
      • HTTPS (public repository): Clone a public repository over HTTPS with no credentials. Set Repository URL to an https:// URL.
    • HTTPS Username: (APC 2.1 and later; HTTPS + Personal Access Token only) Optional. The username paired with your personal access token. Most providers (GitHub, GitLab) accept any non-empty username when a token is supplied; some (for example, Azure DevOps) require a specific username.
    • Personal Access Token: (APC 2.1 and later; HTTPS + Personal Access Token only) The token used to clone your repository. The token is stored write-only and is never displayed after saving.
    • Sync Interval: The time interval between checks for updates in your Git repository, in seconds. A sync is only performed when an update is detected. Astronomer recommends a minimum interval of 60 seconds.
    • Dags Directory: The directory in your Git repository that hosts your Dags. Specify the directory’s path as relative to the repository’s root directory. To use your root directory as your Dags directory, specify this value as ./. Other changes outside the Dags directory in your Git repository must be deployed using astro deploy.
    • Rev: The commit reference of the branch that you want to sync with your Deployment
    • Repo Fetch Mode: Choose Poll or WebHook. If you select WebHook, you need the Webhook URL and Webhook Secret Key for your GitHub Configuration.
    • Webhook URL: (Webhook mode only)
    • Webhook Secret Key: (Webhook mode only)
    • Ephemeral Storage Overwrite Gigabytes: The storage limit for your Git repository. If your Git repository is larger than 2 GB, Astronomer recommends setting this slider to your repository size + 1 Gi
    • Sync Timeout: The maximum amount of seconds allowed for a sync. Astronomer recommends increasing this value if your repository is larger than 1 GB
  5. (Webhook Only) You can now open your GitHub repository and set up a Repository Webhook, or you can return to your Deployment details page to configure this later. Be sure to set the following configurations:
    • Payload URL: Paste the Webhook URL from the Astro Private Cloud UI
    • Content Type: Select JSON.
    • Secret: Paste the Webhook Secret Key from the Astro Private Cloud UI
    • Enable SSL verification
    • Choose Just the push event for the event trigger
  6. Save your changes.
Repo Share Mode - First deploy errorIf you complete your Deployment configuration for git-sync and encounter an error during the first Deployment, you might need to force restart the Airflow Deployment at least once, several minutes after you initially create it. For example, you can add any new environment variable to your Deployment, like FOO=foo, to force the Deployment containers to restart.After you see your Dags update in the Airflow UI, you can remove the environment variable.
After you configure your Deployment, any code pushes to your Dag directory of the specified Git repository and branch will appear in your Deployment with zero downtime.
Newly created Dag files can take up to five minutes (default configuration) from syncing to appear in the Airflow UI. To shorten this delay, we recommend tuning AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL in your Airflow deployment.
Starting in APC 2.1, the Deployment’s status indicator in the Astro Private Cloud UI reflects git-sync health alongside overall Deployment health, so you can tell at a glance whether git-sync is syncing successfully without opening the Logs tab.

​
Configure a Git repository for git-sync deploys

The Git repository you want to sync should contain a directory of Dags that you want to deploy to APC. You can include additional files in the repository, such as your other Astro project files, but note that this might affect performance when deploying new changes to Dags. To deploy Dags from a private Git repository, add a deploy key to your Git repository and provide the matching private key as the Ssh Key so your APC Deployment can access the repository. This process varies slightly between Git repository management tools. For an example, read GitLab’s SSH Key documentation. Starting in APC 2.1, you can authenticate to a private repository over HTTPS with a personal access token instead of SSH. See Authenticate to a private repository over HTTPS.

​
Authenticate to a private repository over HTTPS

Astro Private Cloud 2.1This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
Use the HTTPS + Personal Access Token authentication method to clone a private repository over HTTPS — useful for enterprise Git hosts such as GitHub Enterprise, self-managed GitLab, or Bitbucket, or where SSH (port 22) isn’t available.
  1. In your Git provider, create a personal access token (PAT) with read access to the repository (for example, GitHub’s repo / read:repository scope).
  2. In the Deployment’s Dag Deployment settings, set Authentication Method to HTTPS + Personal Access Token.
  3. Set Repository URL to the repository’s https:// clone URL.
  4. Enter the Personal Access Token. Optionally set HTTPS Username if your provider requires a specific username; otherwise leave it blank.
  5. Save your changes.
The token is stored securely and is never returned by the APC API or shown in the UI after saving. To rotate the token, open the Deployment settings, replace it, and save again.
You can validate the entered credentials before saving with the Validate credentials button in the Deployment settings. The check runs from the target data plane, so it reflects the network path git-sync actually uses.

​
Trust a private certificate authority (CA)

Astro Private Cloud 2.1This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
If your Git host presents a TLS certificate signed by a private or internal certificate authority — common for self-managed GitHub Enterprise, GitLab, or Bitbucket — git-sync must trust that CA, or HTTPS clones fail TLS verification. Astro Private Cloud never disables certificate verification. To make git-sync trust your private CA:
  1. Create a Kubernetes Secret in the platform namespace (for example, astronomer) containing the CA certificate under the key cert.pem, and annotate it so commander replicates it into your Deployment namespaces:
  2. Reference the Secret in your APC API configuration. Houston emits deployments.privateCaCertSecretNames as global.privateCaCerts in the Airflow chart values, and the git-sync container trusts the listed CAs via update-ca-certificates. Setting the platform-level global.privateCaCerts to the same Secret lets the control-plane credential-validation check trust the host as well:
The Secret name must be added to privateCaCertSecretNames for each cluster that runs git-sync Deployments. Add a git-sync CA to the existing global.privateCaCerts list rather than replacing it — Helm replaces list values across -f files, so dropping an existing platform CA can break other platform TLS.

​
Deploy a project image to a git-sync Deployment

git-sync updates only the Dags in your Deployment. Python packages, OS-level packages, and the other files in your Astro project stay in the Deployment image. To update them, deploy a new image. The Deployment keeps git-sync as its Dag deploy mechanism. Use one of the following methods:
  • Build a new image and push it with the Astro CLI:
    The --image flag deploys only the image. The CLI doesn’t deploy the Dags from your Astro project, because git-sync provides them.
  • Point the Deployment at an image that is already in a registry:
    This command calls the updateDeploymentImage mutation of the APC API. Deploys to a custom registry and direct API calls use the same mutation. See Update a Deployment image with the APC API.

​
Enable the update deployment image endpoint

APC disables the updateDeploymentImage endpoint by default. When the endpoint is disabled, an image update returns the following error:
To enable the endpoint, add the following values to your values.yaml file:
Then push the configuration change. See Apply a config change. This setting doesn’t change the Dag deploy mechanism of a Deployment. The Deployment continues to sync Dags from your Git repository.

​
Add Kubernetes scheduling configurations for git-sync relay

You can add Kubernetes scheduling configurations — tolerations, nodeSelector, and affinity — to your global git-sync relay configuration. These configurations allow you to:
  • Specify node selection criteria with nodeSelector
  • Configure Pod affinity and anti-affinity rules
  • Set tolerations for tainted nodes
These settings can help you comply with security or compliance requirements for workload isolation, optimize resource utilization by co-locating related components, and handle tainted nodes in mixed-use Kubernetes clusters. They aren’t required for git-sync relay functionality, so add nodeSelector, affinity, or tolerations only if you need specific node placement for your git-sync-relay components.