Astro CI/CD templates for CircleCI

CircleCI is a continuous integration and continuous delivery platform that can be used to implement DevOps practices. This document provides sample CI/CD templates to automate deploying Apache Airflow dags from a GitHub repository to Astro using CircleCI. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If your team builds custom Docker images, use the custom image implementation. Refer to Template overview to see generic templates expressed as simple shell scripts or configure your own. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites

An Astro project hosted in a Git repository that CircleCI can access.
An Astro Deployment.
A Deployment API token, Workspace API token, or Organization API token.
A CircleCI account.

Image deploy templates

Image deploy templates build a Docker image and push it to Astro whenever you update any file in your Astro project.

Single branch
Multiple branch
Custom Image

To automate code deploys to a Deployment using CircleCI for a single branch implementation, complete the following setup in a Git-based repository that hosts an Astro project:

Configuration requirements

You have a main branch of an Astro project hosted in a single GitHub repository.
You have a production Deployment on Astro where you want to deploy your main GitHub branch.
You have a production CircleCI context that stores environment variables for your CI/CD workflows.

Implementation

Set the following environment variables in a CircleCI context:

ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
ASTRO_DEPLOYMENT_ID: The ID for your Deployment.

Create a new YAML file in .circleci/config.yml that includes the following configuration:

.circleci/config.yml

# Use the latest CircleCI pipeline process engine version.
# See: https://circleci.com/docs/2.0/configuration-reference
version: 2.1

orbs:
  docker: circleci/docker@2.0.1
  github-cli: circleci/github-cli@2.0.0

# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:

  build_image_and_deploy:
    docker:
      - image: cimg/base:stable
    # Add steps to the job
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
    steps:
      - setup_remote_docker:
          version: 20.10.11
      - checkout
      - run:
          name: "Deploy to Astro"
          command: |
            curl -sSL install.astronomer.io | sudo bash -s
            astro deploy ${ASTRO_DEPLOYMENT_ID} -f

# Invoke jobs with workflows
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
workflows:
  version: 2.1
  wf-build-and-deploy:
    jobs:
      - build_image_and_deploy:
          context:
            - <YOUR-CIRCLE-CI-CONTEXT>
          filters:
            branches:
              only:
                - <YOUR-BRANCH-NAME>

The following template can be used to create a multiple branch CI/CD pipeline using CircleCI. A multiple branch pipeline can be used to test dags in a development Deployment and promote them to a production Deployment.

Configuration requirements

You have both a dev and main branch of an Astro project hosted in a single GitHub repository.
You have respective development and production Deployments on Astro where you deploy your GitHub branches to.
You have respective development and production CircleCI contexts that store environment variables to use in your CI/CD workflows.

Implementation

Set the following environment variables in both your production and development CircleCI contexts:

ASTRO_API_TOKEN = <your-workspace-or-organization-api-token-with-access-to-prod-deployment>
ASTRO_DEPLOYMENT_ID = <your-prod-deployment-id>

Create a new YAML file in .circleci/config.yml that includes the following configuration:

.circleci/config.yml

# Use the latest CircleCI pipeline process engine version.
# See: https://circleci.com/docs/2.0/configuration-reference
version: 2.1

orbs:
  docker: circleci/docker@2.0.1
  github-cli: circleci/github-cli@2.0.0

# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:

  build_image_and_deploy:
    docker:
      - image: cimg/base:stable
    # Add steps to the job
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
    steps:
      - setup_remote_docker:
          version: 20.10.11
      - checkout
      - run:
          name: "Deploy to Astro"
          command: |
            curl -sSL install.astronomer.io | sudo bash -s
            astro deploy ${ASTRO_DEPLOYMENT_ID} -f

# Invoke jobs with workflows
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
workflows:
  version: 2.1
  wf_build-and-deploy:
    jobs:
      - build_image_and_deploy:
          context:
            - <YOUR-PROD-CIRCLE-CI-CONTEXT>
          filters:
            branches:
              only:
                - <YOUR-PRODUCTION-BRANCH-NAME>
    jobs:
      - build_image_and_deploy:
          context:
            - <YOUR-DEV-CIRCLE-CI-CONTEXT>
          filters:
            branches:
              only:
                - <YOUR-DEVELOPMENT-BRANCH-NAME>

Read more about multiple workflows in the CircleCI documentation.

If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments in docker build command and then use the image tag to deploy to Astro. See docker build for reference.

Configuration requirements

You have a main branch of an Astro project hosted in a single GitHub repository.
You have a production Deployment on Astro where you want to deploy your main GitHub branch.
You have a production CircleCI context where you store environment variables to use in your CI/CD workflows.

Implementation

Set the following environment variables in a CircleCI context:

ASTRO_API_TOKEN = <your-workspace-or-organization-api-token-with-access-to-prod-deployment>
ASTRO_DEPLOYMENT_ID = <your-deployment-id>

Create a new YAML file in .circleci/config.yml that includes the following configuration:

.circleci/config.yml

# Use the latest CircleCI pipeline process engine version.
# See: https://circleci.com/docs/2.0/configuration-reference
version: 2.1

orbs:
  docker: circleci/docker@2.0.1
  github-cli: circleci/github-cli@2.0.0

# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:

  build_image_and_deploy:
    docker:
      - image: cimg/base:stable
    # Add steps to the job
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
    steps:
      - setup_remote_docker:
          version: 20.10.11
      - checkout
      - run:
          name: "Build image and deploy"
          command: |
            set -e
            echo "export image_tag=astro-$(date +%Y%m%d%H%M%S)" >> $BASH_ENV
            source "$BASH_ENV"
            docker build -t ${image_tag} --build-arg="<your-build-arg>=<your-build-arg-value>" .
            curl -sSL install.astronomer.io | sudo bash -s
            astro deploy --image-name ${image_tag} ${ASTRO_DEPLOYMENT_ID} -f

  # Invoke jobs with workflows
  # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
  workflows:
    version: 2.1
    build-and-deploy-prod:
      jobs:
        - build_image_and_deploy_prod:
            context:
              - <YOUR-CIRCLE-CI-CONTEXT>
            filters:
              branches:
                only:
                  - <YOUR-BRANCH-NAME>

If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to Astronomer support.

Dag deploy templates

A dag deploy template uses the --dags flag in the astro deploy command in the Astro CLI to push only dags to your Deployment. This CI/CD pipeline deploys your dags to Astro when one or more files in your dags folder are modified. It deploys the rest of your Astro project as a Docker image when other files or directories are also modified. For more information about the benefits of this workflow, see Deploy dags only.

Configuration requirements

For each Deployment that you use with dag deploy templates, you must enable dag deploys.

If you stage multiple commits to dag files and push them all at once to your remote branch, the template only deploys dag code changes from the most recent commit. It will miss any code changes made in previous commits.To avoid this, either push commits individually or configure your repository to Squash commits for pull requests that merge multiple commits simultaneously.

Single branch implementation

To automate code deploys to a Deployment using CircleCI, complete the following setup in a Git-based repository that hosts an Astro project:

Set the following environment variables in a CircleCI context:

ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
ASTRO_DEPLOYMENT_ID: The ID for your Deployment.

In your project repository, create a new YAML file in .circleci/config.yml that includes the following configuration:

.circleci/config.yml

# Use the latest CircleCI pipeline process engine version.
# See: https://circleci.com/docs/2.0/configuration-reference
version: 2.1

orbs:
  docker: circleci/docker@2.0.1
  github-cli: circleci/github-cli@2.0.0

# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:

  build_image_and_deploy:
    docker:
      - image: cimg/base:stable
    # Add steps to the job
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
    steps:
      - setup_remote_docker:
          version: 20.10.11
      - checkout
      - run:
          name: "Build image and deploy"
          command: |
            curl -sSL install.astronomer.io | sudo bash -s
            files=($(git diff-tree HEAD --name-only --no-commit-id))
            echo ${files}
            find="dags"
            if [[ ${files[*]} =~ (^|[[:space:]])"$find"($|[[:space:]]) && ${#files[@]} -eq 1 ]]; then
              echo "only deploying dags"
              astro deploy ${ASTRO_DEPLOYMENT_ID} --dags -f;
            else
              echo "image deploy"
              astro deploy ${ASTRO_DEPLOYMENT_ID} -f;
            fi

# Invoke jobs with workflows
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
workflows:
  version: 2.1
  wf-build-and-deploy:
    jobs:
      - build_image_and_deploy:
          context:
            - <YOUR-CIRCLE-CI-CONTEXT>
          filters:
            branches:
              only:
                - <YOUR-BRANCH-NAME>

This script checks the diff between your current commit and the HEAD of your branch to which you are pushing the changes to. If the changes are only in dags then it executes a dag-only deploy. Otherwise, it executes an image-based deploy. Make sure to customize the script to use your specific branch and context. You can customize this script to work for multiple branches as shown in the image-based multi-branch deploy template by creating separate job and workflow for each branch.

Overview

Get started

Develop DAGs

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Astro CI/CD templates for CircleCI

Prerequisites

Image deploy templates

Configuration requirements

Implementation

Configuration requirements

Implementation

Configuration requirements

Implementation

Dag deploy templates

Configuration requirements

Single branch implementation

​Prerequisites

​Image deploy templates

​Configuration requirements

​Implementation

​Configuration requirements

​Implementation

​Configuration requirements

​Implementation

​Dag deploy templates

​Configuration requirements

​Single branch implementation

Prerequisites

Image deploy templates

Configuration requirements

Implementation

Configuration requirements

Implementation

Configuration requirements

Implementation

Dag deploy templates

Configuration requirements

Single branch implementation