Manage and observe per-deployment migration

This guide covers day-2 operations for per-deployment migration: reading the new UI surfaces, triggering a migration, watching one run, understanding what gets blocked while it runs, and recovering when one fails. Per-deployment migration is available in Astro Private Cloud (APC) 2.1 and later. It assumes you have already completed the setup in Configure per-deployment migration: data plane failover enabled on both clusters, real regions created, and every participating cluster assigned to one. For the lookup tables this guide links to — mission and flight states, skip reasons, and the GraphQL surface — see Per-deployment migration reference. Selecting N Deployments and migrating them creates N independent missions, one per Deployment. There is no batch object on the server. Each mission moves through the same state machine as a cluster failover, and each succeeds or fails on its own. Partial success is normal and expected.

Read this before your first migration: there is no rollback. Per-deployment migration always runs in controlled mode, which means the Deployment is fully removed from the source cluster before it is created on the destination. There is no operation to put it back.If the destination then fails — insufficient capacity, quota, or an image-pull failure — the mission ends FAILED, the Deployment’s clusterId stays pointed at the source, and the Deployment is installed nowhere. Its database, secrets, and Dag storage are intact, so no data is lost, but the Deployment is down until you act.Recovery is retry-forward only: re-run the migration. Because the failed mission releases its per-Deployment lock, the Deployment is immediately eligible to migrate again. Fix whatever failed on the destination, then re-run the migration for the same Deployment to the same destination — or to another in-region cluster. Don’t try to hand-restore the Deployment on the source; its release there was already purged. See Recover a failed migration.Verify destination capacity and image availability before you migrate, and canary a single Deployment before you move a batch.

Migration moves the Deployments you pick, within a region, from a healthy source. Cluster failover moves everything on a cluster, can run in forced mode against a dead source, and is sequenced with your database promotion. If your source cluster is down, migration is the wrong tool — use data plane failover.

Read the migration UI

Regions page

The Regions admin page lists every region with its Name, Cloud Provider, Status, and Created date, with Create Region, Edit, and Delete actions. For per-deployment migration, only Name and Cloud Provider matter. The Status badge and the Activate action belong to control plane high availability and have no effect on data plane migration — the migration gate never reads them. Deleting a region is blocked while any cluster still references it.

Clusters list

A Failover column shows a cluster-level failover-readiness shield that mirrors the cluster’s Failover status on its detail page, separate from the existing health dot:

Grey Pending upgrade: failover-capable but not yet upgraded.
Amber Upgrading: a failover upgrade is in flight.
Green Ready: the cluster is Enabled.
Red Issues: failover-capable but not enabled, for example an External Secrets Operator (ESO) secret sync failing on one of its Deployments.

The column is empty only when the cluster isn’t failover-capable. For the same status in words, open the cluster detail page’s Failover field.

Cluster detail

Two elements matter here:

Failover is a status field that replaces the old binary “Failover Enabled”: Not Capable, Pending Upgrade, Upgrading, Enabled, or Issues Detected. Only a cluster reading Enabled can be a migration destination. For the full table, see Cluster failover states.
Trigger Failover is enabled only when the status is Enabled. In any other state it is disabled, with a message explaining why. This triggers a cluster-level failover, not a migration.

For a cluster failover, this page also shows progress across the request’s missions and the skipped-Deployments list at reconcile finalization — which is where you find Deployments that a cluster failover passed over because they were already in a migration.

The cluster detail page, where the Failover status field appears

Deployments list

Four elements are new or changed:

Element	What it does
Checkbox column	A persistent leading column, one checkbox per Deployment. Selection is the entry point for every Deployment-scoped bulk action. A checkbox is disabled when the Deployment has an active mission of any kind, or when you lack update permission on it.
Bulk-actions menu	Appears once one or more Deployments are selected. Contains Migrate to cluster and Upgrade for failover.
Failover-readiness shield	A small shield icon next to the Deployment name, shown only on failover-capable clusters. See the following shield states.
Mission Status column	A per-Deployment phase indicator reflecting any in-flight mission — migration, failover, or upgrade — rendered as, for example, “Migration - In progress.” This is where you watch a migration from the list.

The Deployments list with the checkbox column, bulk-actions menu, failover-readiness shield, and Mission Status column

The shield uses one outline icon, distinguished by color, plus an exclamation badge for the Attention state:

No shield: failover-incapable cluster.
Grey Pending upgrade: failover-capable but not yet upgraded for failover.
Amber In progress: an upgrade mission is in flight.
Green Ready: upgraded and currently failover-eligible.
Red Attention: the upgrade failed, or the Deployment is upgraded but now ineligible.

The API field behind the shield is Deployment.failoverReadiness. See Per-deployment migration reference.

The shield doesn’t track migrations. It tracks the one-time failover-upgrade lifecycle only. A Deployment that is mid-migration keeps a green Ready shield the entire time — the shield means “this Deployment is failover-capable,” not “this Deployment is idle.” Use the Mission Status column, or Deployment.activeMission, to see an in-flight migration.

Deployment detail

Migrate to cluster is a per-Deployment action in the Deployment header’s Actions menu, alongside Upgrade for failover. It calls the same migrateDeployments mutation with a single ID. Use it for canary moves, retries, and one-offs.
There is no dedicated mission or flight view in the APC UI in 2.1. The durable, per-mission detail is available through the missionProgress GraphQL query. The Mission Status column on the Deployments list is the at-a-glance UI surface.

There is deliberately no batch progress page. If you reload after starting a 20-Deployment migration, you haven’t lost anything: you can query each Deployment’s status individually through its Mission Status column or missionProgress.

Trigger a migration

From the Deployments list

Select the Deployments with the checkbox column. Deployments with an active mission aren’t selectable.
Open the bulk-actions menu and select Migrate to cluster.
Select the destination cluster from the picker.
Confirm.

Results come back immediately: successfully claimed Deployments start their missions, and the outcome appears inline as a result banner, for example “47 claimed, 3 skipped (2 current mission conflict, 1 cross region migration not allowed).” The banner is orange when anything was skipped and green when everything was claimed. Read the skip banner. Skipping is normal and non-fatal, but it means those Deployments didn’t move. Every skip reason and its remedy is in Per-deployment migration reference.

From the Deployment detail page

Open the Deployment and select Migrate to cluster from the header Actions menu, then select a destination. It’s the same mutation, the same rules, and one ID. Unlike the list, the result here is a toast: a “Migration started” toast on success, or a “Migration not started” error toast whose text is the raw skip reason when the Deployment is skipped.

The Actions menu on the Deployment detail page, showing Migrate to cluster

The Migrate to cluster destination picker opened from the Deployment detail page

With the API

mutation {
  migrateDeployments(
    deploymentIds: ["<deployment-id-1>", "<deployment-id-2>"]
    destinationClusterId: "<destination-cluster-id>"
  ) {
    missions { deploymentId missionId }
    skipped  { deploymentId reason }
  }
}

deploymentIds takes one or more IDs. Duplicates are de-duplicated. A single-element list is the trivial case; there is no separate single-Deployment mutation.
destinationClusterId is the one cluster all listed Deployments move to. They may come from different source clusters, as long as each source is in the same region as the destination.
Mode is always CONTROLLED. It isn’t a parameter. Migration drains a healthy source before bringing the Deployment up elsewhere; if the source is unavailable, you need a cluster failover, not a migration.

Save the returned missionId values — they are the handle for missionProgress.

The response is synchronous in acceptance, asynchronous in execution. A returned missionId means the mission was durably created, not that the Deployment has moved. Watch progress in the APC UI or with missionProgress.

What migration isn’t for

Draining a whole cluster. Use full-cluster failover to decommission a cluster.
Cross-region or cross-provider moves. The region gate blocks them.
Moving a Deployment off a dead cluster. Migration requires a healthy source. Use cluster failover.
Automatic destination selection. You always specify the destination.

What happens during a migration

Each mission runs the controlled-move sequence:

Mission (kind = MOVE, mode = CONTROLLED)
  │
  ├─▶ Scavenger flight  — on the SOURCE cluster
  │     DRAIN   scale down components
  │     DELETE  uninstall the Helm release and purge its resources. The
  │             namespace itself is deleted only when namespace pools
  │             are disabled
  │
  └─▶ Hyperjump flight  — on the DESTINATION cluster
        (dispatched only after the Scavenger flight SUCCEEDS)
        NAMESPACE   create the destination namespace
        SECRETS     ESO pulls the Deployment's secrets from your backend
        FENCE       disable the source database users and terminate their
                    connections, enable the destination database users
        DEPLOYMENT  install the Helm release on the destination
        WAIT        wait for the Airflow components to become healthy

The phase names — DRAIN, DELETE, NAMESPACE, SECRETS, FENCE, DEPLOYMENT, and WAIT — are the literal values recorded against each flight and emitted in the data plane logs, which is useful when correlating a stuck migration with data plane output. Controlled mode drains and deletes the source first, so that no Airflow instance is writing to the metadata database while another is starting. That is what makes a controlled move zero-data-loss — and also what makes it non-reversible, because by the time the Hyperjump flight runs, the source release is gone. The migration doesn’t touch the Airflow metadata database, your secrets backend, Dag storage, or the Deployment’s record in the control plane. Only the Kubernetes-side installation moves.

On success, the mission reaches COMPLETED, the Deployment’s clusterId flips from source to destination, and its active-mission reference clears.
On failure, the mission reaches FAILED, clusterId stays on the source, and the active-mission reference clears. See Recover a failed migration.

Observe a migration

In the APC UI

The Mission Status column on the Deployments list gives a one-glance phase per Deployment, for example “Migration - In progress.” There is no mission or flight detail view in the APC UI in 2.1; for the full per-mission detail, use the missionProgress API, which is the durable source of truth.

A Deployment with a newly claimed migration mission in the Mission Status column

A Deployment showing a migration in progress in the Mission Status column

With the API

Query the mission IDs the mutation returned:

query {
  missionProgress(missionIds: ["<mission-id-1>", "<mission-id-2>"]) {
    id
    kind                  # MOVE for a migration
    state
    originClusterId
    destinationClusterId
    failoverRequestId     # always null for a migration
    flights {
      id
      role                # SCAVENGER | HYPERJUMP
      state
    }
  }
}

Or ask a Deployment what it is currently doing:

query {
  deployment(where: { id: "<deployment-id>" }) {
    isBeingFailedOver     # true while any mission is active
    failoverReadiness     # the shield state (upgrade lifecycle, not migration)
    activeMission {
      id kind state
      flights { role state }
    }
  }
}

activeMission and isBeingFailedOver clear to null and false as soon as the Hyperjump flight succeeds — for a per-deployment migration the mission is then COMPLETED — and on any terminal state. An in-progress cutover has a non-null activeMission; a completed one is null plus a flipped clusterId. For the mission-state and flight-state enumerations, see Per-deployment migration reference.

Two API caveats. First, missionProgress silently drops missions you lack permission to see rather than returning an error, so a short result list may mean a permissions gap, not a missing mission. Second, missionProgress carries no timestamps, no error text, and no percentage — it tells you which flight is where in its state machine, not how long it has been there or why it failed. For failure detail, check the control plane logs for the mission ID.

missionProgress requires cluster-admin access — system.clusters.get or a cluster-admin role — the same authority needed to run a migration, so the operator who triggers migrations can observe them. A workspace member can’t call it. The deployment query is available to anyone who can view the Deployment.

What is blocked during a migration

While a Deployment has an active mission, mutations that would reach the data plane for that Deployment are rejected, because they would tear down or alter resources the Hyperjump flight is provisioning. The blocked mutations are upsertDeployment updates, deleteDeployment, updateDeploymentConfig, deleteDeploymentConfig, updateDeploymentVariables, updateDeploymentImage, updateDeploymentKedaConfig, createDeployRevision, and deployRollback. Creating a new Deployment isn’t blocked by an active mission, though it is still blocked by an active cluster failover or a cordoned target cluster. updateDeploymentsResources is gated too, but it silently skips the affected Deployments instead of returning an error — check its response rather than assuming it applied to everything you selected. The error is explicit:

deleteDeployment is disabled while deployment <deployment-id> is part of
active mission <mission-id>.

The block is per Deployment — other Deployments on either cluster stay fully mutable — and it lifts automatically when the mission reaches a terminal state. There is nothing to unlock manually. In practice, code deploys to a migrating Deployment fail while the move is in flight. Migrations are usually short, but tell the owning team before you start one, and avoid migrating during a team’s deploy window.

Recover a failed migration

A mission in FAILED needs your attention. What to do depends on where it failed. Query missionProgress, or open the Deployment detail page, and look at the flights:

Symptom	What it means	State of the Deployment
Scavenger `FAILED`, Hyperjump never dispatched	The source drain or uninstall failed.	Still on the source, possibly partially drained.
Scavenger `SUCCEEDED`, Hyperjump `FAILED`	The source was purged and the destination couldn’t bring the Deployment up.	Installed nowhere. Data intact, workload down.
Any flight `AWAITING_DP_HEALTH`	Not a failure — a wait.	Unchanged. It resumes automatically.

If the Hyperjump flight failed, the Deployment is installed nowhere and needs the following recovery:

Diagnose the destination. The usual causes are node capacity, resource quota, and image-pull failure. The destination cluster’s events and the control plane logs for that mission ID will say which.
Fix the underlying cause on the destination cluster.
Retry forward. Re-run migrateDeployments for the same Deployment and the same destination. The active-mission reference cleared when the mission reached its terminal state, so the Deployment is claimable again.
If the destination can’t be fixed quickly, retry forward to a different in-region destination rather than trying to restore the source.

Don’t try to hand-restore the Deployment on the source cluster. The Helm release, namespace, and Kubernetes secrets there were deliberately purged. Recreating them out of band bypasses the fencing model and risks two Airflow instances writing to the same metadata database. Always recover by retrying forward through migrateDeployments.

The Airflow metadata database, your secrets backend, and Dag storage are untouched by a failed migration. The workload is down, but nothing is lost. Dag runs that were active during the move may need attention once the Deployment is back up. To avoid this entirely:

Confirm destination headroom — nodes and quota — before migrating.
Confirm the Deployment’s image is present in a registry the destination can pull from. Replication lag between regional registries is a real cause of Hyperjump failure.
Canary one Deployment before moving a batch.
Migrate outside the owning team’s deploy window.

Overview

Architecture

Upgrade to Astro Private Cloud 2.0

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

APC API

Platform Observability

Release notes

Reference

Manage and observe per-deployment migration

Read the migration UI

Regions page

Clusters list

Cluster detail

Deployments list

Deployment detail

Trigger a migration

From the Deployments list

From the Deployment detail page

With the API

What migration isn’t for

What happens during a migration

Observe a migration

In the APC UI

With the API

What is blocked during a migration

Recover a failed migration

​Read the migration UI

​Regions page

​Clusters list

​Cluster detail

​Deployments list

​Deployment detail

​Trigger a migration

​From the Deployments list

​From the Deployment detail page

​With the API

​What migration isn’t for

​What happens during a migration

​Observe a migration

​In the APC UI

​With the API

​What is blocked during a migration

​Recover a failed migration

​Related documentation

Read the migration UI

Regions page

Clusters list

Cluster detail

Deployments list

Deployment detail

Trigger a migration

From the Deployments list

From the Deployment detail page

With the API

What migration isn’t for

What happens during a migration

Observe a migration

In the APC UI

With the API

What is blocked during a migration

Recover a failed migration

Related documentation