Data plane clusters in Astro Private Cloud

In Astro Private Cloud (APC), a cluster is a data plane Kubernetes cluster, that runs Airflow Deployments and their runtime components. You can attach multiple data plane clusters to a single control plane, which allows you to centralize management of users, workspaces, configuration, Astro UI/API, metrics, and orchestration services. APC can run in split mode, where the data plane and control plane use separate Kubernetes clusters. Or, APC can also run in unified mode, where a single Kubernetes cluster performs both roles. Using specific Kubernetes environments for control and data plane functions provides the following performance and administration benefits:

• Isolation and compliance: Keep teams, environments, or regions separate to meet security and governance requirements.
• Performance and scale: Allocate resources independently and scale data planes without affecting control-plane availability.
• Cost and ownership: Attribute spend to the teams or business units that own each data plane.
• Reliability boundaries: Limit the affected area of failures and define clear service level objectives (SLO) per cluster.
• Network boundaries and residency: Keep data within specific networks or geographies.

Example

A company operates a single APC control plane and two data plane clusters at the same organization:

• Finance data plane cluster: Runs Airflow for data reconciliation and reporting with restricted network access and stricter compliance controls.
• Engineering data plane cluster: Runs Airflow for product analytics and machine learning feature pipelines with more flexible networking.

Both data planes are registered to the same control plane. Platform administrators manage users, Workspaces, and Deployments centrally, while teams operate their Dags within their isolated data planes.

How the control plane keeps track of a cluster

The control plane keeps a copy of each data plane cluster’s metadata, such as its Kubernetes version and chart versions. Starting in APC 2.2, each data plane sends this metadata over DP-Link, so changes reach the control plane within about 30 seconds. See Data plane metadata sync.

Manage clusters

• Register a cluster: Register a data plane cluster.
• Data plane metadata sync: Learn how the control plane keeps a cluster’s metadata current, and which values the data plane owns.
• Override cluster configurations: Update data plane cluster details, like name, labels, connection settings, or credentials, and override cluster default Deployment configurations.
• Deregister a cluster: Deregister a data plane cluster you no longer need.
• Troubleshoot registration: Diagnose and resolve errors during data plane registration.

If you are starting fresh, set up your control plane first, then register one or more data plane clusters before creating Airflow Deployments.