Configure Hashicorp Vault for data plane failover

​

Prerequisites

• A Vault server reachable from every data plane cluster that will use it.
• Permission to create policies, auth mounts, and roles in Vault.
• The Vault CLI, or equivalent API access, authenticated with an admin or operator token.
• kubectl access to each data plane cluster. Depending on your ESO mode, you need permission to create either a cluster-scoped ClusterSecretStore or a namespaced SecretStore. See Which secret store to create.

​

Which secret store to create

​

Grant Vault access for data plane failover

• ESO creates ExternalSecret resources on every failover-enabled cluster, which read the replicated secrets from Vault.
• The deployment orchestrator on the source cluster also creates PushSecret resources, which write the fernet key, environment variables, and database credentials to Vault so the destination cluster can pull them.

​

Step 1: Create a Vault policy

path "secret/data/*" {
  capabilities = ["create", "read", "update", "list"]
}

path "secret/metadata/*" {
  capabilities = ["create", "read", "update", "list"]
}

vault policy write <policy-name> <path-to-policy-file>.hcl

​

Step 2: Enable a Kubernetes auth mount for each data plane cluster

1. Enable a dedicated mount for the cluster:
   vault auth enable -path=<data-plane-name>-kubernetes kubernetes
   
2. Configure the mount with that cluster’s API server details:
   vault write auth/<data-plane-name>-kubernetes/config \
     kubernetes_host="<cluster-api-server-url>" \
     kubernetes_ca_cert=@<path-to-ca-cert> \
     token_reviewer_jwt=@<path-to-reviewer-jwt>
   
   Retrieve kubernetes_host from the cluster’s API server endpoint, and retrieve kubernetes_ca_cert and token_reviewer_jwt from a service account token that Vault uses to validate other tokens. See Vault’s Kubernetes auth method documentation for how to generate the reviewer JWT for your Kubernetes version.

​

Step 3: Create a Vault role

vault write auth/<data-plane-name>-kubernetes/role/<role-name> \
  bound_service_account_names=<eso-service-account> \
  bound_service_account_namespaces=astronomer \
  policies=<policy-name> \
  ttl=1h

​

Step 4: Add the Vault provider to your secret store

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: astronomer-secret-store
spec:
  provider:
    vault:
      server: <vault-server-url>
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: <data-plane-name>-kubernetes
          role: <role-name>
          serviceAccountRef:
            name: <eso-service-account>
            namespace: astronomer

​

Network requirements

• Expose Vault through a network load balancer reachable from every data plane cluster that needs it.
• Open firewall or security group rules that allow traffic from each data plane cluster’s CIDR range to the Vault endpoint.
• Confirm connectivity from each cluster before you rely on the configuration — see the verification steps in the following section.

​

Verify the configuration

1. Write a test secret to Vault:
   vault kv put secret/variables/test-variable value=test-value
   
2. Create a test ExternalSecret in the data plane cluster’s astronomer namespace that references the store and reads the test secret. This example uses a ClusterSecretStore; on a namespace-pools cluster, set secretStoreRef.kind to SecretStore instead.
   apiVersion: external-secrets.io/v1
   kind: ExternalSecret
   metadata:
     name: <test-secret-name>
     namespace: astronomer
   spec:
     refreshInterval: 10s
     secretStoreRef:
       name: astronomer-secret-store
       kind: ClusterSecretStore
     target:
       name: <test-secret-name>
       creationPolicy: Owner
     data:
       - secretKey: value
         remoteRef:
           key: variables/test-variable
           property: value
   
3. Confirm it syncs:
   kubectl get externalsecret <test-secret-name> -n astronomer
   
   The STATUS column should show SecretSynced. If it doesn’t, check the ESO Pod logs for authentication or permission errors before continuing.

​

Troubleshooting

• PushSecret reports a failure, or Airflow secrets don’t appear on the destination cluster after a failover: The Vault policy likely grants read-only access on a cluster where dataPlaneFailover.enabled is true. Update the policy to include create and update on both secret/data/* and secret/metadata/*, then retry.
• ExternalSecret or PushSecret fails to authenticate from one data plane cluster but succeeds from another: Confirm the failing cluster has its own Kubernetes auth mount and that the mount’s kubernetes_host and reviewer JWT match that cluster, not another cluster’s.
• A cluster in a different network can’t reach Vault: Confirm Vault is exposed beyond in-cluster DNS for that network, and that firewall or security group rules allow traffic from the cluster’s CIDR range. See Network requirements.

​

Related documentation

• Data plane failover
• Enable data plane failover
• External Secrets Operator security
• Run a failover upgrade
• Trigger a data plane failover
• Hashicorp Vault as an Airflow secrets backend
• External Secrets Operator Vault provider documentation