Configure control plane reliability

This guide walks you through standing up a control plane reliability group: two or more control planes that share one database, one set of JSON Web Token (JWT) signing material, and one global domain behind weighted, health-checked DNS. For a conceptual overview of the feature, see Control plane reliability. After the group is running, see Manage a control plane reliability group for day-2 operations. Complete the steps in order. The second control plane depends on secrets and registry records created by the first.

Prerequisites

An Astro Private Cloud (APC) 2.1.0 or later installation.
The ability to install the Astronomer platform on two or more Kubernetes clusters, one per control plane. See Install the control plane.
A shared database server that every control plane can reach through an identical connection.
A TLS certificate per control plane that covers both the global and per-control-plane (per-CP) admin names. See the following section.
Permission to create DNS records for both the global domain and each per-CP admin hostname.
kubectl access to each control plane cluster.

Set up TLS and DNS prerequisites

Set these up before you install any control plane. Every control plane is reachable on two kinds of hostnames — the shared global hostname behind the load balancer, and its own per-CP admin hostname that bypasses the load balancer. Both must be covered by TLS and DNS.

TLS certificates

For each control plane, provision a certificate whose Subject Alternative Names (SANs) cover both:

The global names — *.<global-domain-name>, which cover app., houston., and the other global subdomains.
That control plane’s per-CP admin names — *.<cpNN-domain>, for example *.cp01.<parent-domain>.

A single certificate per control plane that covers both wildcards is the simplest arrangement.

Use the DNS-01 Automatic Certificate Management Environment (ACME) challenge, not HTTP-01, for issued certificates. Under control plane reliability the global hostname resolves only to active-region control planes, so an HTTP-01 challenge for a global name can’t be served by a standby control plane and won’t validate reliably. DNS-01 doesn’t require the hostname to be routable and succeeds from any control plane that has DNS-provider API credentials.

DNS records

Per-CP admin hostnames: add a static DNS record for each control plane’s admin hostname (cp01.<parent-domain>, cp02.<parent-domain>) that points directly at that control plane and bypasses the load balancer. You need these to reach and register a specific control plane before it’s in load-balancer rotation.
Global hostname: app.<global-domain-name> and the other global names must resolve to all of the control plane load balancers, with a per-CP health check at /controlplane/status. This is the load-balanced, failover record set, configured in the last step of this guide. See Configure the global DNS.

Install control plane 1 (the bootstrap cluster)

Install the Astronomer platform on the first cluster with control plane reliability enabled. CP 1 is the cluster that generates the shared JWT signing material every other control plane reuses. Add the following to your Astronomer Helm values:

global:
  baseDomain: <cp01-domain>          # this control plane's own per-CP admin domain
  controlPlaneHA:
    enabled: true
    bootstrapJwks: true
    globalBaseDomain: <global-domain-name>
  dataPlaneFailover:
    enabled: true

Each setting does the following:

baseDomain is this control plane’s per-CP admin domain. Each control plane has its own baseDomain (cp01.<parent-domain>, cp02.<parent-domain>), distinct from the shared globalBaseDomain.
controlPlaneHA.enabled: true turns on control plane reliability mode. The chart renders the global-domain ingresses and the APC API runs as part of a high availability (HA) group.
controlPlaneHA.bootstrapJwks: true tells this control plane to generate the shared JWT signing key and certificate. Because this is the first cluster, it bootstraps the material itself, creating two secrets: <release-name>-houston-jwt-signing-key and <release-name>-houston-jwt-signing-certificate. This certificate also signs Docker Registry tokens; there is no separate registry keypair.
controlPlaneHA.globalBaseDomain is the shared domain that all control planes serve.
dataPlaneFailover.enabled: true is optional. It enables data plane failover so that Astro Deployments can be moved between clusters. It’s independent of control plane reliability. See Data plane failover.

Install or upgrade the release and wait for all Pods to come up. After the APC API is running, confirm that both JWT secrets exist in the Astronomer namespace:

kubectl get secret <release-name>-houston-jwt-signing-key \
  <release-name>-houston-jwt-signing-certificate -n <astronomer-namespace>

CP 2 reuses these two secrets in the next step.

Copy the JWT signing material to control plane 2

Both control planes must sign JWTs with the same key and certificate, so that a token minted on one control plane is accepted on the other. CP 1 generated this material in the previous step. CP 2 must reuse it rather than generate its own. Copy these two secrets from CP 1’s Astronomer namespace into CP 2’s Astronomer namespace, keeping the same names and data keys:

<release-name>-houston-jwt-signing-key holds data key tls.key.
<release-name>-houston-jwt-signing-certificate holds data key tls.crt.

The signing-certificate secret must carry the following annotation so that the certificate is propagated into the Airflow Deployment namespaces:

astronomer.io/commander-sync: platform-release=<release-name>

Copy the secrets with whatever tooling you use for secrets — kubectl get -o yaml and kubectl apply, sealed secrets, GitOps, the External Secrets Operator, or a secrets manager. For example:

# On CP 1, export the two secrets:
kubectl get secret <release-name>-houston-jwt-signing-key \
  <release-name>-houston-jwt-signing-certificate \
  -n <astronomer-namespace> -o yaml > jwt-secrets.yaml

# Remove cluster-specific metadata (resourceVersion, uid, creationTimestamp),
# keep the commander-sync annotation on the certificate secret, then on CP 2:
kubectl apply -f jwt-secrets.yaml -n <astronomer-namespace>

These secrets must exist on CP 2 before you install the Astronomer platform there. CP 2 runs with bootstrapJwks: false and consumes the pre-copied secrets. Verify that both secrets exist on CP 2 before you continue.

Install control plane 2 (and any subsequent control planes)

Install the Astronomer platform on the second cluster. The values are almost identical to CP 1, with two differences: use this control plane’s own baseDomain, and set bootstrapJwks: false so that CP 2 consumes the signing material you copied instead of generating its own.

global:
  baseDomain: <cp02-domain>          # THIS control plane's own per-CP admin domain
  controlPlaneHA:
    enabled: true
    bootstrapJwks: false             # consume the pre-copied JWT secrets
    globalBaseDomain: <global-domain-name>
  dataPlaneFailover:
    enabled: true

If CP 2 bootstrapped its own JSON Web Key Set (JWKS), it would sign tokens with a different key than CP 1, and tokens would fail validation when a user is routed to the other cluster. Setting bootstrapJwks: false makes CP 2 use the pre-copied secrets. You don’t need to disable JWKS generation any other way — HA mode skips generation by default.

Both control planes must use the same database. In practice, the astronomer-bootstrap secret that holds the database connection and credentials must be identical on both clusters. If the two control planes point at different databases, they don’t share users, Astro Deployments, or the control plane registry, and control plane reliability doesn’t work.Before you install CP 2, copy the astronomer-bootstrap secret from CP 1, strip its cluster-specific metadata, and apply it on CP 2:

kubectl get secret astronomer-bootstrap -n <astronomer-namespace> -o yaml
# (strip cluster-specific metadata) then apply on CP 2

Install or upgrade the release on CP 2 and wait for all Pods to be healthy. Because it shares CP 1’s database, CP 2 immediately sees the same users, Workspaces, and Astro Deployments. Repeat this step and the previous JWT-copy step for every additional control plane you want in the group. Each one copies the same JWT material and points at the same database.

Create the first admin user

Before you can register regions or control planes, you need an admin user to authenticate as. Because all control planes share one database, you create the admin only once and it works across every control plane. Open app.<global-domain-name> in a browser and complete the first-admin sign-up flow, or use the createUser mutation. The createRegion and registerControlPlane mutations you run next require you to be authenticated as a system admin.

Register the region and control plane 1

Tell the APC API that CP 1 exists and which region it belongs to. You can do this from the APC UI or with GraphQL mutations, authenticated as the admin from the previous step.

Create a region

A region is the logical grouping that control planes attach to. All control planes in the same group share the same region. Run the createRegion mutation:

mutation {
  createRegion(name: "us-east", cloudProvider: "aws") {
    region { id name cloudProvider }
  }
}

name is a human-readable region name and must be unique.
cloudProvider is the cloud provider, for example aws, gcp, or azure.

Save the returned region.id. You need it in the next call and again when you register CP 2. Only one region can be active at a time. To create the region from the APC UI instead, open the Regions tab in the left sidebar (visible after you enable control plane reliability) and select Create Region. Enter a Name and choose a Cloud Provider, then save. The region appears in the list with an Inactive status. New regions start inactive, matching the mutation’s default. Use the per-row Activate action when you’re ready to make it the serving region.

Create Region dialog in the APC UI, with a Name entered and the Cloud Provider dropdown open showing AWS, GCP, Azure, and Local.

Regions list in the APC UI showing the newly created region with an Inactive status badge.

Register the control plane

mutation {
  registerControlPlane(cpId: "<cp1-uuid>", name: "cp01", regionId: "<region-id>") {
    id
    name
    ingressUrl
    chartVersion
    region { id name }
  }
}

cpId is the stable UUID for this control plane. It comes from the cp-identity secret on the cluster. Read it from the control plane itself rather than inventing one:
kubectl -n <astronomer-namespace> get secret cp-identity \ -o jsonpath='{.data.cp_id}' | base64 -d
name is a human-readable identifier for this control plane and must be unique across the registry.
regionId is the region ID from the previous step.

Don’t pass ingressUrl or chartVersion — the APC API derives them from the control plane’s own configuration. The mutation is idempotent: re-running it with the same cpId refreshes those fields, which is useful after a Helm upgrade. The region must already exist before you register a control plane against it. To register from the APC UI instead, use that control plane’s own admin URL (cpNN.<parent-domain>) so the identity pre-fills correctly. From CP 1’s admin URL, open the Control Planes tab and select Register Control Plane. The Control Plane ID field is pre-filled from this control plane’s cp-identity secret — leave it as-is. Enter a Name and select the Region you created, then save. The APC API populates Ingress URL and Chart Version automatically.

Register Control Plane dialog in the APC UI, with the Control Plane ID pre-filled, a Name entered, and the Region dropdown open.

Register control plane 2

CP 2 needs its own registry entry so that the APC API knows it’s part of the group. Because both clusters share the same database, the region you created already exists, so you don’t create a new one. You only register the new control plane against that existing region. Run the registerControlPlane mutation with CP 2’s own cpId and the same regionId from before:

mutation {
  registerControlPlane(cpId: "<cp2-uuid>", name: "cp02", regionId: "<region-id>") {
    id
    name
    ingressUrl
    region { id name }
  }
}

Get CP 2’s cpId the same way as before, but from CP 2’s cluster:

kubectl -n <astronomer-namespace> get secret cp-identity \
  -o jsonpath='{.data.cp_id}' | base64 -d

To register from the APC UI instead, use CP 2’s admin URL (cp02.<parent-domain>), open the Control Planes tab, and select Register Control Plane. The Control Plane ID pre-fills with CP 2’s identity. Enter a Name and select the same Region as CP 1 — don’t create a new one — then save. After this, both control planes are registered against the same region and are part of the same group. List the registered control planes to confirm that both entries share the same region.

Control Planes list in the APC UI showing two registered control planes, cp-ha-01 and cp-ha-02, attached to the same region.

Configure the global DNS

This step load-balances user traffic across the active region’s control planes and provides in-region high availability. Point the global names — app.<global-domain-name> and the rest of *.<global-domain-name> — at all of the control plane load balancers using weighted, health-checked DNS records, so that traffic is spread across the control planes and drained away from an unhealthy one automatically.

Create the record sets

For the shared global domain, create a record set that covers both:

<global-domain-name>: the apex.
*.<global-domain-name>: the wildcard. This one record set covers every customer-facing subdomain (app., houston., grafana., prometheus., alertmanager.).

For each control plane, add an entry to both record sets that points at that control plane’s ingress load balancer, with:

Weighted routing that gives every control plane an equal weight, so DNS load-balances evenly. Each control plane’s entry needs its own unique set identifier.
A health check attached to each control plane’s entry, so that an unhealthy control plane is removed from rotation automatically.

With two control planes, both the apex and the wildcard record sets contain two equal-weight entries, one aliasing each control plane’s load balancer, each guarded by its own health check.

Keep this separate from the per-CP admin records. The static per-CP admin records (cp01.<parent-domain>, cp02.<parent-domain>) point directly at a single control plane and aren’t part of this weighted set. They intentionally bypass the load balancer.

Configure the health check

Tie each control plane’s DNS entry to a health check that targets that control plane’s own APC API endpoint on its per-CP hostname, not the global one:

Protocol HTTPS, port 443.
Path /controlplane/status, the APC API HA health endpoint. It reports the control plane as unhealthy if the control plane isn’t registered, cordoned, decommissioned, running an outdated chart version, or attached to an inactive region.
A reasonable cadence, for example a 30-second interval with a failure threshold of three.

When a control plane’s /controlplane/status starts failing, its DNS entry is pulled from the record set and users are routed to the remaining healthy control planes. When it recovers, it’s added back automatically. At least one control plane must be healthy and in the active region to serve traffic. If every control plane is unhealthy, the global name serves nothing.

Use weighted, health-checked records for the global domain, not plain records. Plain records send users to a single control plane with no automatic failover, and multiple plain records for the same name conflict with each other. If you run DNS automation that would otherwise create plain records, make sure it doesn’t manage <global-domain-name> or its subdomains, or it will fight with these weighted, health-checked records.

Verify

After the records are in place:

# The global names resolve, load-balancing across the control plane load balancers:
dig +short <global-domain-name>
dig +short app.<global-domain-name>

# Each control plane's health endpoint reports healthy:
curl -s https://houston.<cpNN-domain>/controlplane/status

In your DNS provider, the apex and * records for <global-domain-name> should show one weighted entry per control plane, each with its own health check. Take one control plane offline and confirm that its answer stops being served while the global domain stays reachable through the healthy control plane. After a control plane recovers, resolvers may keep serving a cached negative answer for the negative-cache TTL, so flush your local DNS cache and allow for upstream resolver TTLs when you verify recovery.

Add existing Astro Deployments to the global domain

Astro Deployments that existed before you enabled control plane reliability have their Apache Airflow ingress host rules and auth annotations (auth-url, auth-signin) pointing at the per-CP URL of whichever control plane last upserted them. For those Deployments to work under control plane reliability and be reachable on the global domain with cross-control-plane single sign-on (SSO), their ingresses must be re-stamped. Re-stamping:

Adds the global-domain alias hosts (<release>-airflow.….<globalBaseDomain>) alongside the existing per-CP hosts.
Repoints auth-signin to https://houston.<globalBaseDomain>/v1/auth/deployment-signin.
Regenerates auth-url (…/v1/authorization) against globalBaseDomain.

After re-stamping, a Deployment is reachable at both its per-CP URL and its new global URL, and the global session cookie (scoped to .<globalBaseDomain>) is sent to the Deployment’s global Airflow subdomain, so SSO works across control planes. For how the session cookie and Deployment URLs are scoped, see Cookie and URL strategy. Two ways to perform the re-stamp are available.

Automatic (default)

Because astronomer.houston.upgradeDeployments.enabled is true by default, the houston-upgrade-deployments hook runs on the same helm upgrade that enables control plane reliability, and on every control plane upgrade after that. It’s a post-upgrade hook, so it runs after the new configuration (including globalBaseDomain) is applied. It therefore re-stamps against the live global domain and doesn’t leave stale per-CP-only URLs. You don’t need to run any manual steps in the default configuration.

This hook re-templates every non-cordoned Astro Deployment on every control plane helm upgrade, so each upgrade triggers a fleet-wide re-upsert and can restart Airflow Pods. On a large production fleet this means a simultaneous restart of many Deployments, which can cause a load spike and brief disruption to running tasks, with no control over timing. The re-stamp is idempotent: it re-applies each Deployment’s desired state without changing its version.

Manual (when the hook is disabled)

To decouple the re-stamp from the upgrade so you can run it in a maintenance window and scope the rollout, set astronomer.houston.upgradeDeployments.enabled: false. Existing Deployments are then not re-stamped automatically, and you run the script yourself after globalBaseDomain is live:

# Run inside an APC API (houston) Pod, using the same command the hook uses:
yarn upgrade-deployments                              # all non-cordoned Deployments
yarn upgrade-deployments -- --clusterId=<uuid>        # scope to one cluster
yarn upgrade-deployments -- --deploymentUuid=<id>     # scope to one Deployment
yarn upgrade-deployments -- --canary                  # scope to canary Deployments only

The script is idempotent and re-runnable, and scoping to zero matches is a safe no-op. Run it only after globalBaseDomain and the URL-helper and annotation support are live, so that it stamps the correct global URLs rather than per-CP ones.

Set astronomer.houston.upgradeDeployments.enabled under the astronomer subchart, as shown. A top-level houston.upgradeDeployments.enabled doesn’t take effect and silently leaves the hook enabled.

Configure identity provider redirect URLs

If you integrate an external identity provider (IdP), the redirect and callback URLs you register with the IdP must point at the global domain, not a per-CP domain. Under control plane reliability the APC API templates all customer-facing URLs, including the OAuth redirect_uri, from globalBaseDomain. If you’re migrating an existing installation to control plane reliability and your IdP application was registered with a per-CP redirect URI, update it to the global-domain URI as part of enabling control plane reliability. For the exact values to register and the reason behind them, see Identity provider authentication and the OAuth redirect URL.

Overview

Architecture

Upgrade to Astro Private Cloud 2.0

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

APC API

Platform Observability

Release notes

Reference

Configure control plane reliability

Prerequisites

Set up TLS and DNS prerequisites

TLS certificates

DNS records

Install control plane 1 (the bootstrap cluster)

Copy the JWT signing material to control plane 2

Install control plane 2 (and any subsequent control planes)

Create the first admin user

Register the region and control plane 1

Create a region

Register the control plane

Register control plane 2

Configure the global DNS

Create the record sets

Configure the health check

Verify

Add existing Astro Deployments to the global domain

Automatic (default)

Manual (when the hook is disabled)

Configure identity provider redirect URLs

​Prerequisites

​Set up TLS and DNS prerequisites

​TLS certificates

​DNS records

​Install control plane 1 (the bootstrap cluster)

​Copy the JWT signing material to control plane 2

​Install control plane 2 (and any subsequent control planes)

​Create the first admin user

​Register the region and control plane 1

​Create a region

​Register the control plane

​Register control plane 2

​Configure the global DNS

​Create the record sets

​Configure the health check

​Verify

​Add existing Astro Deployments to the global domain

​Automatic (default)

​Manual (when the hook is disabled)

​Configure identity provider redirect URLs

​Related documentation

Prerequisites

Set up TLS and DNS prerequisites

TLS certificates

DNS records

Install control plane 1 (the bootstrap cluster)

Copy the JWT signing material to control plane 2

Install control plane 2 (and any subsequent control planes)

Create the first admin user

Register the region and control plane 1

Create a region

Register the control plane

Register control plane 2

Configure the global DNS

Create the record sets

Configure the health check

Verify

Add existing Astro Deployments to the global domain

Automatic (default)

Manual (when the hook is disabled)

Configure identity provider redirect URLs

Related documentation