Astro CI/CD templates for Jenkins

Use the following CI/CD templates to automate deploying Apache Airflow dags from a Git repository to Astro with Jenkins.

The following templates for Jenkins are available:

• Image deploy templates
• dag deploy templates

Each template type supports multiple implementations. If you have one Deployment and one environment on Astro, use the single branch implementation. If you have multiple Deployments that support development and production environments, use the multiple branch implementation. If your team builds custom Docker images, use the custom image implementation.

For more information on each template or to configure your own, see Template overview. To learn more about CI/CD on Astro, see Choose a CI/CD strategy.

Prerequisites

• An Astro project hosted in a Git repository that Jenkins can access.
• An Astro Deployment.
• A Deployment API token, Workspace API token, or Organization API token.
• Access to Jenkins.

Each CI/CD template implementation might have additional requirements.

Image deploy templates

pipeline {
    agent any
    stages {
        stage('Deploy to Astronomer') {
            when {
                expression {
                    return env.GIT_BRANCH == "origin/main"
                }
            }
            steps {
                checkout scm
                sh '''
                curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.38.0/astro_1.38.0_linux_amd64.tar.gz
                tar -zxvf astro_1.38.0_linux_amd64.tar.gz astro && rm astro_1.38.0_linux_amd64.tar.gz
                ./astro deploy env.ASTRONOMER_DEPLOYMENT_ID
                '''
            }
        }
    }
    post {
        always {
            cleanWs()
        }
    }
}

Dag deploy templates

The dag deploy template uses the --dags flag in the Astro CLI to push dag changes to Astro. These CI/CD pipelines deploy your dags only when files in your dags folder are modified, and they deploy the rest of your Astro project as a Docker image when other files or directories are modified. For more information about the benefits of this workflow, see Deploy dags only.

Single branch implementation

Use the following template to implement dag-only deploys to a single Deployment using Jenkins.

1. In your Jenkins pipeline configuration, add the following parameters:

   • ASTRO_API_TOKEN: The value for your Workspace or Organization API token.
   • ASTRONOMER_DEPLOYMENT_ID: The Deployment ID of your production deployment

   Be sure to set the values for your API token as secret.

2. At the root of your Git repository, add a Jenkinsfile that includes the following script:

pipeline {
    agent any
    stages {
        stage('Dag Only Deploy to Astronomer') {
            when {
                expression {
                    return env.GIT_BRANCH == "origin/main"
                }
            }
            steps {
                checkout scm
                sh '''
                curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.38.0/astro_1.38.0_linux_amd64.tar.gz
                tar -zxvf astro_1.38.0_linux_amd64.tar.gz astro && rm astro_1.38.0_linux_amd64.tar.gz
                files=($(git diff-tree HEAD --name-only --no-commit-id))
                find="dags"
                if [[ ${files[*]} =~ (^|[[:space:]])"$find"($|[[:space:]]) && ${#files[@]} -eq 1 ]]; then
                ./astro deploy env.ASTRONOMER_DEPLOYMENT_ID --dags;
                else
                ./astro deploy env.ASTRONOMER_DEPLOYMENT_ID;
                fi
                '''
            }
        }
    }
    post {
        always {
            cleanWs()
        }
    }
}