Roll back an image deploy on Astronomer Software

Deploy rollbacks are an emergency option if a Deployment unexpectedly stops working after a recent deploy. For example, if one of your DAGs worked in development but suddenly fails in a mission-critical production Deployment, you can roll back to your previous deploy to quickly get your pipeline running again. This allows you to revert your deployment to a known good state while you investigate the cause of the failure. You can roll back to any deploy in the last three months regardless of your DAG code or Deployment settings. However, rollbacks are only supported for Runtime 5.0.0 and above or its equivalent Airflow version, 2.3.0 and above.

Prerequisites​

• You must be at least a Deployment Editor or otherwise have the permission deployment.images.push or deployment.dags.push or both to roll back images and/or DAGs, respectively, for a Deployment.
  • If you want to limit users to either rollback images or DAGs, you can configure granular rollback permissions
• Your Deployment must be on Astro Runtime 5 (Airflow 2.3) or later. Rolling back to any version before Astro Runtime 5 is not supported.
• Your Deployment must be configured to use image or DAG-only deploys. Rollbacks are not supported for NFS deploys and Git sync deploys.
• Your Software installation must have global.dagOnlyDeployment.enabled=True

Configure deploy rollbacks​

To use deploy rollbacks, you must have them enabled for your entire installation. This configuration allows individual Deployment users with sufficient permissions to optionally enable them in specific Deployments.

Configure the following values in your values.yaml file:

# Enable APIs to modify deploys
astronomer:
  houston:
    config:
      deployments:
        enableUpdateDeploymentImageEndpoint: true
# Enable rollback feature flag
global:
    deployRollbackEnabled: true
# Enable cleanup feature flag
houston:
    cleanupDeployRevisions:
        enabled:  true
# Configure which data to clean up
houston:
    cleanupDeployRevisions:
        olderThan:  90

After you set this configuration, you can enable deploy rollbacks for individual Deployments. This configuration also enables cleanup of any deploy information that are older than 90 days. See Apply a Config Change for more information on how to modify these values.

Enable deploy rollbacks​

If deploy rollbacks are configured for your installation, you can enable them in individual Deployments through the Deployment Settings page.

1. In the Software UI, open your Deployment.
2. On the Settings page, toggle Rollback Deploy to Enable.

Roll back to a deploy​

1. In the Software UI, open your Deployment, then go to Deploy History.
2. Find the deploy you want to roll back to in the Deploys table, then click Deploy.
3. Write a description for why you're triggering the rollback, then confirm the rollback.

What happens during a deploy rollback​

A deploy rollback is a new deploy of a previous version of your code. This means that the rollback deploy appears as a new deploy in Deploy History, and the records for any deploys between your current version and rollback version are still preserved. In Git terms, this is equivalent to git revert.

When you trigger a rollback, the following information is rolled back:

• All project code, including DAGs.
• Your Astro Runtime version. Note that if you roll back to a restricted Runtime version they can include major bugs and performance issues.
• Your Deployment's DAG deploy setting.

The following information isn't rolled back:

• Your Deployment's resource configurations, such as executor and scheduler configurations.
• Your Deployment's environment variable values.
• Any other Deployment settings that you configure through the Astro UI, such as your Deployment name and description.
• For Runtime version downgrades, any data related to features that are not available in the rollback version are erased from the metadata database and not recoverable.

Logs related to the rollback are exported to Elasticsearch.

Granular rollback permissions​

You might want to enable your users to roll back DAG-only or image-only deploys, without granting them the ability to roll back complete project deploys. This allows you to maintain higher security measures, by limiting who has permission to roll back both image and DAG deploys, or only one of the two.

• deployment.images.push: Grants permission to roll back just the image of a revision in an imaged-based, DAG-only, and gitSync deployment.
• deployment.dags.push: Grants permission to roll back only DAGs for DAG-only deploys.

Users with both permissions can roll back both images and DAGs. If users attempt to roll back a deploy and have limited permissions, they can only choose the type of deploy to roll back that they have permission for.

Follow the procedure in Customize role permissions to configure these user permissions.