Run a failover upgrade

Deployments created before you enabled data plane failover on a cluster don’t have the inactive database connection that failover needs — only Deployments created after you enable it get one automatically. Running a failover upgrade retrofits existing Deployments with that missing connection so they become eligible for failover, without you needing to recreate them.

For a conceptual overview of failover, see Data plane failover.

The examples on this page show common ways to use the APC API — they’re not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to https://houston.<your-base-domain>/v1, and click on the Docs tab. See Develop and test APC API queries for more on using the built-in GraphQL explorer.

Prerequisites

You’ve enabled data plane failover on the cluster. See Enable data plane failover.
The Deployments you want to upgrade use image-based deploys. Deployments that use git-sync or Dag-only deploy mechanisms aren’t eligible.
The Deployments aren’t cordoned, and the cluster doesn’t have a failover in progress.

If you hard-delete a Deployment that used a manually specified (custom) release name on a failover-enabled cluster, the secrets pushed to your external secrets store aren’t removed immediately — they enter that provider’s soft-delete window first (for AWS Secrets Manager, 30 days by default, configurable down to a minimum of 7 days). If you recreate a Deployment with the same release name during that window, every other component comes up normally, but the PushSecret for that Deployment fails, because the secret name is still reserved by the pending deletion. Either wait out the provider’s recovery window, force-delete the secrets from the store to skip it, or use a different release name when recreating the Deployment.

Upgrade one Deployment

From the APC UI

Open the Deployment you want to upgrade.
Click Failover Upgrade.

Using the APC API

Use the upgradeDeploymentsForFailover mutation with deploymentIds set to the Deployment’s ID:

mutation upgradeDeploymentsForFailover(
  $deploymentIds: [Uuid]
) {
  upgradeDeploymentsForFailover(
    deploymentIds: $deploymentIds
  ) {
    missions {
      deploymentId
      missionId
      __typename
    }
    skipped {
      deploymentId
      reason
      __typename
    }
    __typename
  }
}
{
  "deploymentIds": ["deployment-id-1"]
}

Upgrade multiple Deployments

From the APC UI

Go to the cluster’s Deployments list.
Select the Deployments you want to upgrade, or select all of them to upgrade every eligible Deployment on the cluster at once.
Click Failover Upgrade.

Using the APC API

Provide exactly one of clusterId or deploymentIds. Providing both, or neither, returns an error.

To upgrade every eligible Deployment on a cluster at once, pass clusterId:

mutation upgradeDeploymentsForFailover(
  $clusterId: Uuid
) {
  upgradeDeploymentsForFailover(
    clusterId: $clusterId
  ) {
    missions {
      deploymentId
      missionId
      __typename
    }
    skipped {
      deploymentId
      reason
      __typename
    }
    __typename
  }
}
{
  "clusterId": "cluster-id"
}

To upgrade a specific set of Deployments instead, pass deploymentIds and omit clusterId:

mutation upgradeDeploymentsForFailover(
  $deploymentIds: [Uuid]
) {
  upgradeDeploymentsForFailover(
    deploymentIds: $deploymentIds
  ) {
    missions {
      deploymentId
      missionId
      __typename
    }
    skipped {
      deploymentId
      reason
      __typename
    }
    __typename
  }
}
{
  "deploymentIds": ["deployment-id-1", "deployment-id-2"]
}

What happens

APC upgrades each selected Deployment individually. A Deployment that’s already upgraded, cordoned, or using an unsupported deploy type is skipped automatically — it doesn’t block the other Deployments in the same batch. If you used the API, the mutation response lists each Deployment APC started an upgrade for (missions, with a missionId you can use to track that upgrade) and each Deployment APC didn’t upgrade (skipped, with a reason):

Reason	Meaning
`ALREADY_UPGRADED`	The Deployment already has the inactive database connection it needs.
`UNSUPPORTED_DEPLOY_TYPE`	The Deployment doesn’t use an image-based deploy.
`CLUSTER_NOT_FAILOVER_CAPABLE`	The Deployment’s cluster doesn’t have data plane failover enabled.
`CLUSTER_FAILOVER_IN_PROGRESS`	A cluster-level failover is currently in progress.
`DEPLOYMENT_CORDONED`	The Deployment is cordoned.
`CLUSTER_CORDONED`	The Deployment’s cluster is cordoned.
`CURRENT_MISSION_CONFLICT`	The Deployment already has another migration in progress.

If you used the UI, the cluster shows an overall failover upgrade status so you can confirm when every eligible Deployment has finished:

Not started: You haven’t upgraded any eligible Deployments yet.
In progress: An upgrade is running for at least one Deployment.
Complete: Every eligible Deployment on the cluster has the inactive database connection it needs for failover.

Check failover readiness

Each Deployment on a failover-enabled cluster shows a failover-readiness shield in the Deployments list. The shield tells you, at a glance, whether that Deployment can fail over right now, still needs a failover upgrade, or needs your attention first:

No shield: Failover isn’t enabled on the Deployment’s cluster, so readiness doesn’t apply.
Gray — Needs upgrade: The Deployment is on a failover-enabled cluster but hasn’t been upgraded yet. Run a failover upgrade to make it eligible.
Yellow — Upgrading: A failover upgrade is currently running for the Deployment.
Green — Ready: The Deployment has everything it needs for failover — the inactive database connection, an image-based deploy, and its secrets synced to the external secrets store.
Red — Needs attention: Either the Deployment’s last failover upgrade failed, it’s no longer eligible after being upgraded, or its secrets failed to sync to the external secrets store (a PushSecret failure). A PushSecret failure can also happen if you recently recreated the Deployment with the same manually specified release name as a hard-deleted one — see the note under Prerequisites. See Verify secret replication before a failover.

Debug a red shield

Start by checking the PushSecret resources in the Deployment’s namespace:

kubectl -n <deployment-namespace> get pushsecrets

A healthy Deployment shows every PushSecret as Synced:

NAME                                             AGE   STATUS   LAST SYNC
<deployment-namespace>-active-metadata           41s   Synced   40s
<deployment-namespace>-active-result-backend     41s   Synced   40s
<deployment-namespace>-elasticsearch             41s   Synced   41s
<deployment-namespace>-env                       41s   Synced   41s
<deployment-namespace>-fernet-key                41s   Synced   41s
<deployment-namespace>-inactive-metadata         41s   Synced   40s
<deployment-namespace>-inactive-result-backend   40s   Synced   39s

If one instead shows an error, or no status at all, describe it to see why:

kubectl -n <namespace> describe pushsecret <pushsecret-name>

Check the Events section at the bottom of the output for the sync failure reason — typically a permission error from the external secrets store rejecting the write, or (per the note under Prerequisites) a secret name still reserved by a pending soft-deletion.

Overview

Architecture

Upgrade to Astro Private Cloud 2.0

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

APC API

Platform Observability

Release notes

Reference

Run a failover upgrade

Prerequisites

Upgrade one Deployment

From the APC UI

Using the APC API

Upgrade multiple Deployments

From the APC UI

Using the APC API

What happens

Check failover readiness

Debug a red shield

​Prerequisites

​Upgrade one Deployment

​From the APC UI

​Using the APC API

​Upgrade multiple Deployments

​From the APC UI

​Using the APC API

​What happens

​Check failover readiness

​Debug a red shield

​Related documentation

Prerequisites

Upgrade one Deployment

From the APC UI

Using the APC API

Upgrade multiple Deployments

From the APC UI

Using the APC API

What happens

Check failover readiness

Debug a red shield

Related documentation