Automate preview Deployments with any CI/CD tool

A preview Deployment is an Astro Deployment that a CI/CD workflow automatically creates and deletes based on feature branches in your Git repository. The workflow creates the preview Deployment when you create a temporary feature branch and deletes it when you delete the branch. Astronomer recommends using preview Deployments if you regularly need to test a small set of dags on Astro before promoting those dags to a base, production Deployment. This lowers the infrastructure cost of Deployments that are dedicated to development and testing. To implement this feature, you need a CI/CD workflow that:

Creates the preview Deployment when you create a new branch.
Deploys code changes to Astro when you make updates in the branch.
Deletes the preview Deployment when you delete the branch.
Deploys your changes to your base Deployment after you merge your changes into your main branch.

If you use GitHub Actions as your CI/CD tool, you can find preview Deployment templates as part of the Astronomer GitHub action in the GitHub Marketplace. This GitHub action includes sub-actions for each of these four steps. To learn more, see GitHub Actions templates for preview Deployments. To configure your own automated workflow for preview Deployments with another CI/CD tool, use the following scripts. Each of the following four shell scripts is equivalent to the steps required to implement this feature with GitHub Actions.

Create a preview Deployment

In a Deployment preview CI/CD pipeline, you run this script when you create a feature branch off of the main branch of your Astro project.

# Install the Astro CLI
curl -sSL https://install.astronomer.io | sudo bash -s

# Get preview Deployment name
DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --key configuration.name)"
BRANCH_DEPLOYMENT_NAME=$BRANCH_NAME_$DEPLOYMENT_NAME
BRANCH_DEPLOYMENT_NAME="$BRANCH_DEPLOYMENT_NAME// /_"

# Create template of Deployment to be copied
astro deployment inspect $DEPLOYMENT_ID --template > deployment-preview-template.yaml # automatically creates deployment-preview-template.yaml file

# Add name to Deployment template file
sed -i "s|  name:.*|  name: $BRANCH_DEPLOYMENT_NAME}|g"  deployment-preview-template.yaml

# Create new preview Deployment based on the Deployment template file
astro deployment create --deployment-file deployment-preview-template.yaml

# Get the ID of the new preview Deployment
PREVIEW_DEPLOYMENT_ID="$(astro deployment inspect -n $BRANCH_DEPLOYMENT_NAME --key metadata.deployment_id)"

# Copy connections, Airflow variables, and pools from the source Deployment
astro deployment connection copy --source-id $DEPLOYMENT_ID --target-id $PREVIEW_DEPLOYMENT_ID
astro deployment airflow-variable copy --source-id $DEPLOYMENT_ID --target-id $PREVIEW_DEPLOYMENT_ID
astro deployment pool copy --source-id $DEPLOYMENT_ID --target-id $PREVIEW_DEPLOYMENT_ID

# Deploy new code to the deployment preview
astro deploy -n $BRANCH_DEPLOYMENT_NAME

Creating a Deployment from a template does not copy the connections, Airflow variables, or pools from the source Deployment, so the script copies them separately with astro deployment connection copy, astro deployment airflow-variable copy, and astro deployment pool copy.

The copy commands fail if the source Deployment is hibernating. Resume the source Deployment before you run them.

Update a preview Deployment

In a Deployment preview CI/CD pipeline, you run this script whenever you make changes in your feature branch.

# Install the Astro CLI
curl -sSL https://install.astronomer.io | sudo bash -s

# Get preview Deployment name
DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --key configuration.name)"
BRANCH_DEPLOYMENT_NAME=$BRANCH_NAME_$DEPLOYMENT_NAME
BRANCH_DEPLOYMENT_NAME="$BRANCH_DEPLOYMENT_NAME// /_"

# Deploy new code to the preview Deployment
astro deploy -n $BRANCH_DEPLOYMENT_NAME

Delete a preview Deployment

In a Deployment preview CI/CD pipeline, you run this script when you delete your feature branch.

# Install the Astro CLI
curl -sSL https://install.astronomer.io | sudo bash -s

DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --key configuration.name)"
BRANCH_DEPLOYMENT_NAME=$BRANCH_NAME_$DEPLOYMENT_NAME
BRANCH_DEPLOYMENT_NAME="$BRANCH_DEPLOYMENT_NAME// /_"

# Delete preview Deployment
astro deployment delete -n $BRANCH_DEPLOYMENT_NAME -f

Deploy changes from a preview Deployment to a base Deployment

In a Deployment preview CI/CD pipeline, you run this script when you merge your feature branch into your main branch.

# Install the Astro CLI
curl -sSL https://install.astronomer.io | sudo bash -s

# Deploy new code to base Deployment
astro deploy $DEPLOYMENT_ID

Overview

Get started

Develop DAGs

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Automate preview Deployments with any CI/CD tool

Create a preview Deployment

Update a preview Deployment

Delete a preview Deployment

Deploy changes from a preview Deployment to a base Deployment

​Create a preview Deployment

​Update a preview Deployment

​Delete a preview Deployment

​Deploy changes from a preview Deployment to a base Deployment

Create a preview Deployment

Update a preview Deployment

Delete a preview Deployment

Deploy changes from a preview Deployment to a base Deployment