Skip to main content
Astro Private Cloud 2.2This feature was introduced in Astro Private Cloud 2.2. To access this feature, upgrade your Astro Private Cloud installation to 2.2 or later.
The Astro Private Cloud (APC) control plane keeps a copy of each data plane cluster’s metadata. This metadata includes the cluster’s Kubernetes version, the data plane chart version, the Airflow chart version that new Deployments use, the Elasticsearch host for task logs, and the namespace pool settings. The APC API uses this copy when it creates and updates Deployments on the cluster. Starting in APC 2.2, each data plane sends its metadata to the control plane over DP-Link, the long-lived connection that the control plane already holds to each data plane. A change on the data plane reaches the control plane within about 30 seconds, instead of up to an hour. The hourly sync stays in place as a fallback. There is nothing to turn on. Metadata sync works after you upgrade, as long as the data plane runs APC 2.2, in installations with a separate control plane and data plane and in unified installations, whether or not data plane failover is enabled.

​
How metadata reaches the control plane

DP-Link is a worker on the control plane. It holds a gRPC stream to the data plane internal API on every registered cluster that isn’t cordoned or deleted.
  • When the data plane sends metadata. The data plane internal API sends a full metadata snapshot when the stream connects or reconnects, and then every 30 seconds, whether or not anything changed.
  • When the control plane writes it. Each snapshot carries a revision, a hash of its content. The APC API compares the revision with the one it stored and writes only when they differ, so an unchanged data plane causes no writes.
  • Heartbeats. The data plane internal API also sends a heartbeat every 10 seconds. DP-Link uses the heartbeats to track the health of each cluster’s link. See Link health.
  • The network path. DP-Link uses the same data plane internal API endpoint that the control plane already uses to provision Deployments. In an installation with a separate control plane and data plane, this is the commander.<baseDomain> ingress on port 443.

​
What the control plane updates

From each snapshot, the APC API updates the following for the cluster:
  • The Kubernetes version, the data plane chart version, the data plane internal API version, the region, and whether data plane failover is enabled.
  • The six Deployment configuration values that the data plane supplies. See Values the data plane supplies.
A snapshot never changes cluster overrides (Cluster.configOverride) or any other configuration. A snapshot replaces only the state that the data plane owns, so a value removed on the data plane stops being used. The control plane rejects a metadata document that fails validation, whether it comes from the stream, a pull, or registration. The stored values stay as they were.

​
Values the data plane supplies

The data plane owns the following Deployment configuration values. The control plane takes them from the data plane’s metadata, and they always win over an override. To change one of these values, change the setting on the data plane and upgrade the data plane. The new value reaches the control plane on the next snapshot. You can’t override these values on the control plane:
  • APC API: updateCluster rejects a new value at one of these paths in deploymentsConfigOverride, and registerCluster rejects any value at a path that the data plane reports. The error names each path and where its value comes from, for example: Cluster overrides cannot set the following data-plane-managed paths: "chart.version" (from commander's Airflow chart version). The data plane supplies these values, so an override would never take effect. Remove them from deploymentsConfigOverride. updateCluster accepts a value that matches what the cluster override already stores, so you can send a stored override back unchanged. Copying the data plane’s value into an override that doesn’t already hold it is rejected.
  • Workspace and Deployment overrides: These paths are blocked at the Workspace and Deployment tiers. See Blocklisted keys.
  • APC UI: The cluster configuration editor still shows these paths as editable, but the confirmation dialog warns that it doesn’t save edits to them, because the data plane supplies them.
Cluster.readOnlyConfigPaths returns the list of these paths for a cluster. To read the values that a Deployment actually uses, query Cluster.effectiveConfig or Deployment.effectiveConfig. See Effective configuration. A data plane that runs an older data plane internal API might not report helm.openshift.enabled or helm.ingress.extraIngressAnnotations. When it doesn’t, registerCluster accepts an override at that path. updateCluster rejects a new value at every path in the list, whether or not the data plane reports it. See Values the data plane supplies for how these values interact with the other configuration tiers. DP-Link tracks the health of each cluster’s link from the age of the last heartbeat: Every cluster that DP-Link holds a stream to has a link health, not only clusters with data plane failover enabled. Data plane failover doesn’t choose a destination whose link isn’t healthy. Link health means that the control plane can reach the data plane over DP-Link, not that the data plane is up. It’s separate from the cluster status that the APC UI shows, such as ACTIVE. See Cluster status management.

​
Fallback paths

APC 2.2 keeps the pull paths from earlier releases alongside DP-Link:
  • The hourly sync. The <release-name>-houston-sync-dataplane-clusters CronJob still pulls every cluster’s metadata once an hour, configured with astronomer.houston.syncDataplaneClusters.enabled and astronomer.houston.syncDataplaneClusters.schedule. It’s the fallback for every cluster, and the only path that notices a cluster whose link is healthy but whose metadata isn’t arriving.
  • Pulls during some operations. The APC API still pulls a cluster’s metadata before some Deployment and cluster operations, and during a data plane failover.
  • An optional pull for unreachable clusters. While a cluster’s link is UNREACHABLE, DP-Link can pull that cluster’s metadata every 300 seconds. This pull is off by default. To turn it on, set astronomer.dpLink.fallbackPull.enabled to true in your control plane values.yaml, and set astronomer.dpLink.fallbackPull.intervalSeconds to change the interval. It doesn’t run for DEGRADED clusters, which are still receiving snapshots.
To refresh a cluster’s metadata immediately, run the reconcileClusterMetadataJob query, or start a Job from the sync CronJob:
See Force a metadata reconciliation. A pull that finds drift clears the stored revision, so the next snapshot writes the same values once to record it again. This is expected and isn’t a sign of drift. Metadata freshness depends on DP-Link:
  • It runs by default. DP-Link runs on every control plane and every unified installation (astronomer.dpLink.enabled: true), as a Deployment with 2 replicas. It also runs when data plane failover or Laminar is enabled, whatever astronomer.dpLink.enabled is set to. On a unified installation, the chart adds a NetworkPolicy rule that lets DP-Link reach the data plane internal API on ports 50051 and 8880.
  • It holds a stream to every cluster. DP-Link connects to every registered cluster that isn’t cordoned or deleted, not only clusters with failover enabled. One DP-Link Pod owns each cluster’s stream at a time, through a lease.
  • Size it for your clusters. Each DP-Link Pod requests 250m CPU and 512Mi of memory by default. Size DP-Link for the number of registered clusters.
  • Proxies must carry long-lived gRPC streams. A load balancer or ingress between the control plane and a data plane that limits the lifetime of a stream or a connection ends the stream. DP-Link reconnects with backoff, and the new stream sends a snapshot, but the cluster can show as DEGRADED or UNREACHABLE in the meantime. A customer-managed or non-NGINX ingress controller must enable gRPC and HTTP/2 for commander.<baseDomain>.
If you set astronomer.dpLink.enabled to false, and neither data plane failover nor Laminar is enabled, DP-Link doesn’t run. Metadata then reaches the control plane only through the hourly sync and the pulls during operations.

​
Settings reference

Don’t set DP-Link environment variables through astronomer.dpLink.env. The chart renders every entry in it twice, which can make helm upgrade fail or leave DP-Link running with an old value. Use the astronomer.dpLink values in the table instead. For a DP-Link environment variable that has no chart value, set it on the DP-Link Deployment, and set it again after each upgrade:

​
Limitations

  • Link health isn’t visible in the APC UI or the APC API. Read the DP-Link logs on the control plane, which record each change.
  • Up to 30 seconds of lag. A change reaches the control plane on the next snapshot. An APC API Pod that misses a cache update can serve the previous values for up to about two more minutes.
  • A healthy link doesn’t prove that metadata is arriving. If the data plane internal API can’t build a valid snapshot, it skips the snapshot without closing the stream. The cluster’s link stays HEALTHY, and only the pulls update its metadata. Check the data plane internal API logs for an error or a warning.
  • A data plane on an older release sends no metadata over DP-Link. Its metadata reaches the control plane only through the pulls, as before. DP-Link keeps retrying the connection, and snapshots start once you upgrade the data plane to APC 2.2.
  • One invalid field stops metadata sync for the cluster. The control plane rejects the whole document on every path until the field is fixed on the data plane.
  • Cordoned clusters aren’t streamed. DP-Link drops the stream when you cordon a cluster and reconnects when you uncordon it. In the meantime, only the pulls update its metadata.
  • No sync timestamp. The cluster record doesn’t store when metadata last arrived. For when metadata last changed, use the cluster audit trail.
  • A stream that goes silent isn’t reconnected. If the network path to a data plane disappears without closing the connection, for example when the data plane’s load balancer is deleted or replaced, DP-Link keeps waiting on the old stream. Restart DP-Link with kubectl -n <platform-namespace> rollout restart deployment/<release-name>-dp-link. The new Pods reconnect to each cluster once the old lease expires, after 60 seconds by default.