- In-region high availability (HA): if you stand up more than one control plane in the active region, they serve traffic together behind weighted, health-checked DNS. When one becomes unhealthy, DNS drains it from rotation and routes users to the remaining healthy control planes in that region automatically. A region with a single control plane has no in-region redundancy.
- Manual cross-region disaster recovery: you place control planes in a second, standby region and, during a regional outage or a planned migration, an admin fails the platform over to that region. Cross-region failover is always an explicit admin action — there is no automatic region failover, so the control plane isn’t automatically highly available across regions.
Control plane reliability and data plane failover are independent features that solve different problems. Control plane reliability keeps the control plane (the APC UI and API) available: highly available within a region, and manually recoverable across regions. Data plane failover moves Apache Airflow Deployments between data plane clusters. You can run either feature on its own, or both together. This document covers only the control plane.
How it works
A control plane reliability group is a set of control planes that share three things and are fronted by one load-balanced domain:- A shared database: every control plane connects to the same APC API database, so they present the same users, Workspaces, and Astro Deployments. Both clusters must use an identical database connection, supplied through the same
astronomer-bootstrapsecret. - Shared JWT signing material: every control plane signs JSON Web Tokens (JWTs) with the same key and certificate, so a token minted on one control plane is trusted by every other control plane. The first control plane generates this material; every subsequent control plane reuses a copy of it.
- A shared global domain: every control plane serves the same customer-facing domain,
app.<global-domain-name>. Weighted, health-checked DNS records spread traffic across all control planes and drain an unhealthy one automatically.
cpNN.<parent-domain>) that bypasses the load balancer, so you can reach and administer a specific control plane directly even when it isn’t in DNS rotation.
Regions and the control plane registry
When you enable control plane reliability, the APC API tracks two kinds of records that you manage from the APC UI or through GraphQL mutations:- A region is a logical grouping that control planes attach to. Every control plane in a control plane reliability group belongs to a region, and only one region is active at a time. Activating a region atomically deactivates every other region.
- A control plane registry entry records each control plane, the region it belongs to, its ingress URL, and its chart version. You register each control plane once against a region.
The health endpoint
Each control plane exposes an HA health endpoint at/controlplane/status. Your global DNS load balancer polls this endpoint per control plane and serves traffic only to control planes that report healthy. A control plane reports unhealthy (HTTP 503) when it isn’t registered or when it is cordoned, decommissioned, running a chart version behind its region’s maximum, or attached to an inactive region. At least one control plane must be healthy and in the active region to serve traffic.
Chart-version eligibility
Only control planes running the highest registered chart version within their own region are eligible to serve traffic. The APC API computes this maximum dynamically across the active control planes in each region. A control plane that lags behind its region’s maximum is drained until it catches up. This gate governs both rolling upgrades and cross-region failover. For the upgrade procedure, see Manage a control plane reliability group.What control plane reliability does and doesn’t own
Control plane reliability owns the control plane side of availability: which control planes are eligible to serve, which region is active, and the health signal the DNS load balancer polls. Control plane reliability doesn’t fail over the APC API database, and it doesn’t rotate DNS. Both are external to the platform:- Database failover is handled by your managed-database tooling (for example, Amazon RDS or Google Cloud SQL). During a cross-region failover you must promote the destination-region database and repoint the connection before you activate the destination region.
- DNS load balancing and failover is handled by your DNS provider through the weighted, health-checked records you configure. Control plane reliability only flips each control plane’s health signal.
Terminology
Requirements
Before you configure control plane reliability, confirm the following:- A TLS certificate for each control plane whose Subject Alternative Names cover both the global names (
*.<global-domain-name>) and that control plane’s per-CP admin names (*.<cpNN-domain>). Use the DNS-01 Automatic Certificate Management Environment (ACME) challenge, not HTTP-01. For details, see Configure control plane disaster recovery. - A single shared database that every control plane connects to through an identical
astronomer-bootstrapsecret. - The ability to create weighted, health-checked DNS records for the global domain in your DNS provider.
- A plan for sizing the shared database’s connection limit as you add control planes. See Control plane reliability reference.
Limitations
- Admin-driven region failover only: there is no automatic cross-region failover. Moving the active region is always an explicit admin action.
- No in-flight work drain: cordoning a control plane doesn’t drain in-flight work. Any messages still queued in a source region’s NATS at deactivation stop being consumed.
- Database and DNS are external: control plane reliability doesn’t fail over the database or rotate DNS. You are responsible for both. See What control plane reliability does and doesn’t own.