Manage a control plane reliability group

This guide covers day-2 operations for a control plane reliability group: how to read the admin UI, take a control plane in or out of service, roll a chart-version upgrade across your control planes, and perform a cross-region failover. It assumes you have already stood up a control plane reliability group by following Configure control plane reliability — two or more control planes sharing one database and one global domain, with weighted, health-checked DNS in front of them. For the terms used here (<global-domain-name>, the per-control-plane (per-CP) admin hostname <cpNN-domain>, /controlplane/status, and the shared JSON Web Token (JWT) keypair), see Control plane reliability.

Cross-region failover in this guide means moving the active region for the control planes. It’s unrelated to data plane failover, which moves Apache Airflow Deployments between data plane clusters and has its own components and runbook.

The admin UI

When you enable control plane reliability, two admin tabs appear in the left sidebar, along with a small indicator showing which control plane you’re connected to. All three are hidden on single-control-plane installations.

Control Planes tab: visible to system admins only. Lists every registered control plane. This is where you cordon, decommission, register, and deregister.
Regions tab: visible to any viewer, but its actions are admin-only. Lists regions. This is where you activate a region.
Connected-control-plane indicator: a sidebar chip, admin-only, that shows the region and status of the control plane your browser session is currently connected to.

APC admin UI showing the Control Planes tab, the Regions, Control Planes, and System entries in the left sidebar, and the connected-control-plane indicator at the bottom of the sidebar.

Control plane list

Each row on the Control Planes tab shows the following columns:

Column	Description
Health	A colored dot. See The health dot.
Name	The human-readable name you registered the control plane under, for example `cp01`.
Region	The region this control plane is attached to.
Status	The control plane’s effective status: `ACTIVE`, `STANDBY`, `CORDONED`, or `DECOMMISSIONED`. See Control plane status reference.
Chart Version	The platform chart version this control plane is running.
Ingress URL	The control plane’s own APC API ingress URL.
Registered	When the control plane was first registered.
Last Heartbeat	When the APC API last recorded a liveness tick from this control plane.

The available actions are Register Control Plane, and per row Edit (a dialog with Name, Region, and a Status dropdown) and Deregister (which removes the control plane from the registry after a typed confirmation).

There is no dedicated cordon or decommission action. You change a control plane’s state through Edit > Status. The dropdown offers only legal transitions. See Manage control plane status.

The health dot

The health dot is a three-state readiness indicator, separate from the Status badge. It’s computed in the browser from the control plane’s last heartbeat and its chart version relative to the group-wide maximum. It isn’t the /controlplane/status routing signal.

Dot	Meaning
Green	The heartbeat is fresh and the control plane is at the group-wide maximum chart version, so it’s ready to serve.
Orange	The heartbeat is fresh, but the chart version is behind the group-wide maximum, so the control plane is running but not eligible to serve at the current version.
Red	The heartbeat is stale, so the control plane’s APC API process is likely down or unreachable.

Staleness is checked first, because a control plane that isn’t running has a meaningless version. Hover the dot for the exact reason.

A red dot doesn’t, by itself, pull a control plane out of DNS rotation. Routing eligibility is driven entirely by /controlplane/status. The heartbeat and the dot are observability signals only. That said, if a control plane’s APC API is truly down, its /controlplane/status also fails at the connection level, so DNS drains it anyway.

Heartbeat

Every APC API replica writes a periodic liveness tick. The default cadence is 30 seconds, and a heartbeat is considered stale after about 90 seconds (three missed ticks). This threshold is server-authoritative. The Control Planes list refetches on navigation and after any control plane or region change. It doesn’t poll on a fixed timer, so the health dot’s freshness advances as the page re-renders rather than ticking live.

Region list

The Regions tab lists Name, Cloud Provider, Status (an Active or Inactive badge), and Created. The available actions are Create Region, and per row Edit, Delete, and Activate. Activate appears only on regions that aren’t currently active.

Regions tab in the APC UI, listing regions with Name, Cloud Provider, Status, and Created columns, and per-row Activate, Edit, and Delete actions.

There is deliberately no deactivate action. Activating one region atomically deactivates all others, so switching the active region is a single action. See Cross-region failover.

Manage control plane status

A control plane has one of three stored statuses, plus a derived fourth status that you only ever see and never set:

Status	Set by you	Serving	Mutations	Heartbeat	Meaning
`ACTIVE`	Yes	Yes, if the region is active	Accepted	Yes	Normal operation.
`STANDBY`	Derived	No, routed away	Rejected `REGION_INACTIVE`	Yes	An active control plane whose region is inactive — a hot standby waiting for its region to be activated. Computed from status and region state, not stored.
`CORDONED`	Yes	No, routed away	Rejected `CP_CORDONED`	Yes	Temporarily out of service and reversible. Used for maintenance, node work, or as a prelude to decommissioning. Still heartbeats, so it stays in the tracked fleet.
`DECOMMISSIONED`	Yes	No, routed away	Rejected `CP_DECOMMISSIONED`	No	Removed from the fleet. No heartbeat, and no participation in version or activation math. Recoverable to `ACTIVE` while its Pods are still running; otherwise treat it as gone.

Change a control plane’s status

From the APC UI, open the Control Planes tab, select Edit on the control plane, pick the new value from the Status dropdown, and save.

Edit Control Plane dialog in the APC UI with the Status dropdown open, showing the CORDONED option available from ACTIVE.

Through the API, use the updateControlPlane mutation, which requires system-admin access. Cordon and decommission aren’t separate mutations — they’re status values on this one call:

mutation {
  updateControlPlane(cpId: "<cp-uuid>", status: CORDONED) {
    id name status
  }
}

Legal transitions are enforced server-side, and the UI dropdown offers only legal ones:

ACTIVE          → CORDONED
CORDONED        → ACTIVE | DECOMMISSIONED
DECOMMISSIONED  → ACTIVE

The forward path is ACTIVE → CORDONED → DECOMMISSIONED, and both non-active states are reversible to ACTIVE. You can’t jump straight from ACTIVE to DECOMMISSIONED; you must cordon first. An illegal transition is rejected with INVALID_CP_STATUS_TRANSITION.

What happens when you cordon

Setting CORDONED writes one database field, and two independent consumers act on it:

The health endpoint fails. /controlplane/status on that control plane starts returning 503. Your global DNS load balancer, which health-checks that path, drains the control plane from rotation and sends customers to the remaining healthy control planes. Control plane reliability doesn’t touch DNS itself; it only flips the health signal.
Mutations are rejected. The cordoned control plane still serves read queries, but every state-changing mutation is rejected with CP_CORDONED, except the exempt admin mutations, so you can always un-cordon it. A cordoned control plane is also excluded from the chart-version-maximum computation, which is what makes it safe to use as a maintenance state during a rolling upgrade.

Control Planes list in the APC UI showing a control plane with a CORDONED status badge.

Decommissioning behaves the same way for serving and mutations, rejecting mutations with CP_DECOMMISSIONED, but it’s the terminal state: the control plane stops heartbeating and is excluded from all version and activation math, so a stale or dead control plane can never hold back the rest of the fleet.

Edit Control Plane dialog in the APC UI with the Status dropdown open, showing the DECOMMISSIONED option available from CORDONED.

Control Planes list in the APC UI showing a control plane with a DECOMMISSIONED status badge.

Recover or remove a control plane

Un-cordon or recover: Edit > Status > ACTIVE. This works from both CORDONED and DECOMMISSIONED as long as the control plane’s Pods are still running. These are exempt mutations that bypass the status gate.

Edit Control Plane dialog in the APC UI with the Status dropdown open, showing the ACTIVE option available to recover a decommissioned control plane.

Deregister: the Deregister action, or deregisterControlPlane(cpId), removes the control plane’s registry row entirely. Use this once a control plane is truly gone. After you deregister, that control plane’s /controlplane/status reports NOT_REGISTERED (503) until it’s registered again.

The typical workflow is to cordon, do the maintenance, then un-cordon. To retire a control plane permanently: cordon, confirm it has drained, decommission, tear down the infrastructure, then deregister.

Upgrade the chart version

Only control planes running the highest registered chart version within their own region are eligible to serve traffic. The APC API computes the maximum chart version across the active control planes in a region and marks any control plane below it as outdated (503, and mutations rejected with CP_VERSION_OUTDATED). There is no separately stored expected version — it’s computed dynamically as the peer maximum, using semantic-version comparison. Cordoned and decommissioned control planes are excluded from this maximum, so they can’t drag the bar up or down. A control plane’s reported version refreshes automatically right after helm upgrade and again on each heartbeat tick.

Upgrade procedure

Upgrade one control plane at a time, or several in parallel:

Run helm upgrade on the first control plane. Its chart version becomes the new maximum in its region.
Every not-yet-upgraded control plane in that region immediately becomes ineligible: its /controlplane/status returns 503 and it stops accepting mutations until it catches up.
During this window, customer traffic flows only to the already-upgraded control planes.
Upgrade the next control plane. It rejoins the eligible set. Repeat until every control plane in the region is upgraded.
Verify that each control plane’s health dot is green and /controlplane/status returns 200.

Availability during an upgrade

Within an active region, moving one control plane ahead makes the lagging control planes temporarily unhealthy, so the region runs at reduced capacity — potentially a single serving control plane — until the others catch up. To minimize that window, run the helm upgrade commands in parallel across the region’s control planes rather than strictly one after another. The version-maximum check is scoped to each control plane’s own region, so upgrading control planes in a standby (inactive) region has no effect on the active region’s serving. A common pattern is to fully upgrade the standby region first with zero customer impact, activate it, then upgrade the now-standby former-active region.

Roll back by running helm upgrade to the target older version, never helm rollback. helm rollback reverts the APC API Deployment but doesn’t re-render the astronomer-houston-config ConfigMap, which is a keep-policy pre-upgrade-hook resource. This leaves the APC API reporting a stale chart version, so the Control Planes UI shows the wrong version and the eligibility gate mis-ranks the control plane. Running helm upgrade to the older version re-fires the pre-upgrade hook and re-renders the config.Rollback is symmetric with upgrade: downgrading a single control plane makes it the lowest version and therefore ineligible, so to actually roll back you must downgrade all control planes in the region.

Cross-region failover

Cross-region failover moves the active region from one set of control planes to another. It’s always admin-driven; there is no automatic region failover.

Control plane reliability doesn’t fail over your database. Your managed-database tooling (for example, Amazon RDS or Google Cloud SQL) promotes a database replica in the destination region and repoints the connection, and you must complete this before you activate the destination region. Region activation is only the control plane half of the cutover — it flips which region serves; it doesn’t move data. Activating a region before its database is the writable primary is a known failure mode.

Control plane reliability needs only two outcomes from your database failover, and your database tooling is responsible for delivering both:

The database is failed over, so the destination region’s database is the writable primary.
APC always sees a single, stable endpoint — the hostname the APC API connects through (in astronomer-bootstrap) resolves to that primary at all times.

How you achieve those outcomes — replica promotion, DNS or CNAME switching, a single auto-switching endpoint, or another mechanism — is your choice. Any specific database steps in this section are suggestions from Astronomer’s internal testing, not requirements. The one ordering constraint is yours to meet: complete the database failover before you call activateRegion.

Planned region cutover

Use this graceful procedure for an intentional move — maintenance, migration, or cost — rather than an outage. Run every admin action against the per-CP admin hostnames (cpNN.<parent-domain>), not the global URL. During the cutover window the global URL is intentionally unavailable.

Cordon every control plane in the source region

Set Status to CORDONED on each source control plane, from the UI or with updateControlPlane(status: CORDONED). This blocks subsequent mutations on the source control planes, which quiesces writes so that database replication can catch up before the promotion in the next step.Within about 30 seconds — the health-check TTL — the global load balancer stops serving every cordoned control plane, so APC becomes unavailable through the global DNS name and is reachable only through the per-CP admin URLs, which is what admins should use to orchestrate the failover. This has no effect on the data plane or running Airflow Deployments; it only takes the APC UI and API offline on the global URL for the cutover window.

There is currently no in-flight-work drain. Any messages still queued in the source region’s NATS at deactivation stop being consumed.

Fail over the database to the destination region

Control plane reliability doesn’t do this; your managed-database tooling handles it, and the mechanism is your choice. Before you activate the region, both required outcomes must hold: the destination database is the writable primary, and APC sees a single, stable endpoint resolving to it. The following are suggestions from Astronomer’s internal testing, not a required procedure:

With the source control planes cordoned, writes are quiesced, so for a planned failover you can wait for replication lag to reach zero before promoting, to avoid data loss.
If you promote a read replica, wait until promotion is fully complete — for example, the instance reports available and is no longer attached to a replication source. Some engines briefly report available while still attached.
If you keep APC’s endpoint stable through a DNS or CNAME record, a low TTL (for example, 60 seconds) makes the switch propagate quickly, and the astronomer-bootstrap secret stays unchanged because only the record moves. If you use a single auto-switching endpoint, there’s nothing to repoint.
You may need to recycle the APC API’s database connections on the destination control planes (roll the APC API Kubernetes Deployment) if their Pods were pinned to the old endpoint, so that they reconnect to the new primary.
Plan failback as its own cutover. After a promotion, cross-region database replication is typically broken, and re-establishing it in the reverse direction is a separate rebuild.

Activate the destination region

Call activateRegion against a control plane in the destination region, using that control plane’s per-CP admin hostname. The global load balancer won’t route to a control plane whose region is still inactive. Within about 30 seconds every control plane’s /controlplane/status reflects the new region-active state, and the global load balancer begins serving the destination control planes. See Activate a region.

Restart the ingress controller on the source control planes

Scale the nginx ingress controller Deployment on the source (now-inactive) control planes to 0 and back up. New browser connections already route to the newly active region through the global load balancer, but existing keep-alive connections may still be held open against a source control plane and keep hitting the now-inactive region. Bouncing the ingress controller breaks those connections and forces browsers to reconnect, re-resolve the global DNS name, and land on the correct active control plane.

Outage cutover

If the source region’s control planes are down or unreachable, you can’t cordon them and there’s nothing to quiesce, so skip the cordon step. Fail over the database, accepting data loss up to the last replicated transaction (skip the zero-lag wait), then activate the destination region. Because the source control planes are already unreachable, existing browser connections to them are already broken, so the ingress-controller restart is unnecessary. Expect that some in-flight customer operations may need to be retried once traffic lands in the destination region.

Activate a region

From the APC UI, open the Regions tab, select Activate on the destination region, and type the region name to confirm. The confirmation dialog warns that activation shifts customer traffic to this region, deactivates the currently active one, and can’t be undone automatically.

Activate region dialog in the APC UI, warning that activation shifts customer traffic and deactivates the currently active region, with a field to type the region name to confirm.

After activation, the Control Planes tab reflects the new active region: the control plane in the newly active region shows ACTIVE, and the control plane in the now-inactive region shows STANDBY.

Control Planes list in the APC UI after a region activation, with the control plane in the newly active region showing ACTIVE and the control plane in the now-inactive region showing STANDBY.

Through the API, use the activateRegion mutation:

mutation {
  activateRegion(regionId: "<region-id>", force: false) {
    id name active
  }
}

It runs in a single transaction that deactivates every currently active region and activates the target, so the “exactly one region active” invariant holds with no window of zero or two active regions.
It’s an exempt mutation, so you can run it from a control plane whose region is currently inactive — otherwise failover would be impossible. Run it through the destination control plane’s per-CP admin hostname.
An inactive region’s control planes report REGION_INACTIVE (503, and mutations rejected), but they stay alive, heartbeating and upgradeable, serving no customer traffic until their region is activated. This is the STANDBY state.

The activation version gate

To prevent a silent downgrade, activateRegion refuses to activate a region unless at least one control plane in it is at the group-wide maximum chart version (the maximum across all active control planes in every region, with cordoned and decommissioned control planes excluded). Otherwise it fails with TARGET_REGION_NOT_UPGRADED, whose payload lists each target control plane’s version against the group-wide maximum, so you can see exactly what to upgrade. In the UI this surfaces as a Cannot activate region dialog listing the lagging control planes.

Cannot activate region dialog in the APC UI, explaining that no control plane in the target region is at the fleet-wide maximum chart version, with Understood and Force activate anyway actions.

force: true bypasses the version gate. Use it only for an emergency rollback where a knowing downgrade is acceptable and you’re sure the schema is backward-compatible. Forced activations are written to the audit log. In the UI this is gated behind a second confirmation.

Force activation downgrade-risk dialog in the APC UI, warning that activating a region whose control planes are behind the fleet-wide maximum downgrades serving, with Back and Force activate actions.

If you activate a region that has no control plane — for example, an empty standby region — every control plane in the group reports STANDBY and no control plane serves customer traffic until a control plane in the active region is available.

Control Planes list in the APC UI after activating a region with no control plane, showing every control plane with a STANDBY status so none is serving traffic.

Why the customer session survives a cutover

Mid-session users aren’t signed out across a cutover because the session cookie is scoped to .<global-domain-name>, the JWT signing key is shared across all control planes (so a token minted on the old region is trusted on the new one), and the database is the same instance, already failed over by your database tooling before the cutover.

Before any failover, re-check that the global DNS records have health checks against /controlplane/status. Without them, DNS keeps sending customers to control planes in the now-inactive region. Those control planes correctly reject mutations with REGION_INACTIVE, but customers see errors instead of an uninterrupted cutover.

Overview

Architecture

Upgrade to Astro Private Cloud 2.0

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

APC API

Platform Observability

Release notes

Reference

Manage a control plane reliability group

The admin UI

Control plane list

The health dot

Heartbeat

Region list

Manage control plane status

Change a control plane’s status

What happens when you cordon

Recover or remove a control plane

Upgrade the chart version

Upgrade procedure

Availability during an upgrade

Cross-region failover

Planned region cutover

Outage cutover

Activate a region

The activation version gate

Why the customer session survives a cutover

​The admin UI

​Control plane list

​The health dot

​Heartbeat

​Region list

​Manage control plane status

​Change a control plane’s status

​What happens when you cordon

​Recover or remove a control plane

​Upgrade the chart version

​Upgrade procedure

​Availability during an upgrade

​Cross-region failover

​Planned region cutover

​Outage cutover

​Activate a region

​The activation version gate

​Why the customer session survives a cutover

​Related documentation

The admin UI

Control plane list

The health dot

Heartbeat

Region list

Manage control plane status

Change a control plane’s status

What happens when you cordon

Recover or remove a control plane

Upgrade the chart version

Upgrade procedure

Availability during an upgrade

Cross-region failover

Planned region cutover

Outage cutover

Activate a region

The activation version gate

Why the customer session survives a cutover

Related documentation