Derive cluster configuration from platform defaults

Astro Private Cloud 2.2This feature was introduced in Astro Private Cloud 2.2. To access this feature, upgrade your Astro Private Cloud installation to 2.2 or later.

When you register a data plane cluster, Astro Private Cloud (APC) stores a copy of the platform deployments defaults in the cluster record, called the snapshot. In unified mode, APC stores the snapshot for the default cluster when you first install the platform. A cluster that resolves its configuration from the snapshot doesn’t see later changes to the keys that the snapshot holds, even after you change astronomer.houston.config.deployments in your control plane values.yaml. Only clusters that you register after the change get the new values. Starting in APC 2.2, a cluster can derive its configuration instead. A deriving cluster builds its configuration from the platform defaults that the APC API runs with now, then layers the cluster, Workspace, and Deployment overrides on top. A change to the platform defaults then reaches every deriving cluster, wherever no override sets the same key. Values that the data plane supplies always win, because no override tier can set them. See Values the data plane supplies. Derivation is off by default, and upgrading to APC 2.2 doesn’t turn it on. For the 2.2 changes that apply to every cluster, see Changes when you upgrade to 2.2.

How derivation is controlled

Two settings decide whether a cluster derives:

The platform mode, astronomer.houston.config.clusters.configDerive.mode, set in the control plane values.yaml.
The cluster’s own choice, set on the cluster page in the APC UI or with the updateClusterConfigDerive mutation.

Platform mode	Cluster choice not set	Cluster opted in	Cluster opted out
`off` (default)	Snapshot	Snapshot	Snapshot
`optIn`	Snapshot	Derives	Snapshot
`on`	Derives	Derives	Snapshot

off overrides every cluster’s choice, so you can use it to stop all clusters from deriving at once. An opted-out cluster never derives, whatever the platform mode. APC keeps the snapshot in the cluster record while a cluster derives, and keeps updating it: each cluster override save and each data plane metadata update is merged into it. A cluster that stops deriving resolves from that snapshot again. It keeps its cluster overrides but not the platform default changes made since the snapshot was taken, so turning derivation off for a cluster needs no data restore.

When running Deployments change

APC resolves a Deployment’s configuration when it generates the Deployment’s Helm values. A Deployment picks up a value that it now inherits on its next update, including the Deployment upgrades that run when you upgrade the platform. Opting a cluster in doesn’t redeploy anything by itself.

Clusters registered on APC 2.0.3 or later 2.0.x releases

A cluster registered on APC 2.0.3 or a later 2.0.x release stores an incomplete container security context in its snapshot: securityContext.container holds only readOnlyRootFilesystem: true. APC applies that value whole to the containers of every Helm-based Deployment on the cluster, and doesn’t fill in the missing fields from the current platform defaults. While the cluster resolves from its snapshot, which is the default after an upgrade to 2.2, the Deployments’ Pods therefore lack allowPrivilegeEscalation: false and capabilities.drop: [ALL]. A cluster that derives its configuration gets the complete security context from the current platform defaults. See Known issues. To fix an affected cluster without turning on derivation, set the complete security context as a cluster override. Do this before you upgrade, so that the Deployment upgrades that run during the platform upgrade use it. The mutation works on APC 2.0 and 2.2:

mutation {
  updateCluster(
    id: "<cluster-id>"
    deploymentsConfigOverride: {
      securityContext: {
        container: {
          allowPrivilegeEscalation: false
          capabilities: { drop: ["ALL"] }
          runAsNonRoot: true
          seccompProfile: { type: "RuntimeDefault" }
        }
      }
    }
  ) {
    id
  }
}

The override leaves out readOnlyRootFilesystem, so the cluster keeps its current setting. If you’ve already upgraded, set the override, then update each affected Deployment so that APC re-creates its Pods. Once the cluster derives its configuration, this override pins the security context. To let later platform default changes reach the cluster, set securityContext.container to "DELETE_KEY" in a cluster override. See Remove a setting. Setting the platform mode to on also fixes the security context, but it makes every cluster that hasn’t opted out derive at once, including clusters that aren’t affected. Run the dry run and review every cluster first. If you later set the mode back to off, an affected cluster without the cluster override returns to the incomplete security context on its next Deployment update.

Prerequisites

APC 2.2 or later on the control plane.
System Admin permissions, or another role with permission to update clusters, to change a cluster’s choice.
kubectl access to the control plane namespace, to run the dry run.

Derive a cluster’s configuration

Predict the effect with a dry run

On a cluster that was registered long ago, many platform defaults can differ from its snapshot, not only the one you meant to change. Run the dry run before you opt a cluster in. It reports, per cluster and per key, how derived configuration would differ from the snapshot. It only reads the database, works whatever the platform mode is, and changes nothing.

kubectl exec -n <platform-namespace> deploy/<platform-release-name>-houston -- yarn derive-config-dry-run

To report only some clusters, add -c <cluster-id> once for each cluster. To get JSON instead of text, add --json --output /tmp/derive-dry-run.json, then copy the file out of the Pod with kubectl cp. Without --output, the APC API’s log lines are mixed into the JSON on standard output.Review every key that the dry run reports for the cluster:

A key whose platform default changed after the cluster was registered takes the new default once the cluster derives. This includes keys that later APC releases added to the platform defaults.
A key that someone set directly in the snapshot, without a cluster override, loses that value once the cluster derives. For example, a value written into the database by hand. Record each value that you want to keep as a cluster override before you opt in. See Override data plane cluster configurations.
A key labeled system-managed value missing from the cluster record would take the control plane’s own value once the cluster derives, such as its chart version, Elasticsearch host, or namespace pool. Don’t opt the cluster in while the dry run reports this. Contact Astronomer support.
A key labeled set by cluster intent or removed or withdrawn by cluster intent comes from your cluster override and is expected. A key labeled unexplained fits none of these cases. Contact Astronomer support before you opt the cluster in.

Set the platform mode

To opt clusters in one at a time, set the mode to optIn in your control plane values.yaml:

astronomer:
  houston:
    config:
      clusters:
        configDerive:
          mode: "optIn"

Quote the value. Some YAML parsers read a bare on or off as a boolean.Apply the change with a Helm upgrade of the control plane. See Apply a config change. The upgrade restarts the APC API Pods.Astronomer recommends optIn until you opt in and check every cluster that you want to derive. With on, every cluster that hasn’t opted out derives as soon as the upgrade finishes.If you’re upgrading clusters that were registered on APC 2.0.3 or a later 2.0.x release, see Clusters registered on APC 2.0.3 or later 2.0.x releases before you choose a mode.

Opt a cluster in

A cluster’s choice takes effect immediately, without a Helm upgrade or a restart.

Astro UI
APC API

In the APC UI, go to Clusters and select the cluster.
In the Derived configuration section, select Opt in.
(Optional) Enter a Reason. APC records it in the audit log and doesn’t store it on the cluster.
Click Apply, then click Apply in the confirmation dialog.

To clear the cluster’s choice so that the platform mode alone decides, select Follow the platform setting instead. Without permission to update the cluster, the section shows only the cluster’s status.The section shows the cluster’s choice and whether the cluster derives:

Cluster choice: Opted in, Opted out, or Not set, so the platform setting decides.
In effect: Deriving from platform defaults or Using the stored snapshot. When the platform mode overrides the cluster’s choice, for example while the mode is off, the section says so.

Use the updateClusterConfigDerive mutation. override takes ENABLED to opt in, DISABLED to opt out, or INHERIT to clear the cluster’s choice so that the platform mode alone decides.

mutation {
  updateClusterConfigDerive(
    id: "<cluster-id>"
    override: ENABLED
    reason: "<reason-for-the-audit-log>"
  ) {
    id
    configDeriveEnabled
    configDeriveEffective
  }
}

configDeriveEnabled is the cluster’s choice: true, false, or null when no choice is set. configDeriveEffective is whether the cluster derives after the platform mode is applied.

Check the result

Query the cluster’s resolved configuration and compare it with what the dry run predicted:

query {
  cluster(id: "<cluster-id>") {
    id
    configDeriveEffective
    effectiveConfig
  }
}

Cluster.effectiveConfig stops before the Workspace and Deployment tiers, so a Deployment with its own or its Workspace’s override can resolve differently. To see what one Deployment resolves, query Deployment.effectiveConfig. See Effective configuration.The Deployments on the cluster pick up the new values on their next update.

Roll back

To stop one cluster from deriving, opt it out with Opt out in the APC UI or override: DISABLED in the API. The change takes effect immediately, and the cluster resolves from its snapshot again. To stop every cluster from deriving, set mode: "off" and run a Helm upgrade of the control plane. A cluster registered on APC 2.0.3 or a later 2.0.x release goes back to an incomplete container security context when it stops deriving, unless it has the cluster override described in Clusters registered on APC 2.0.3 or later 2.0.x releases. As with opting in, the Deployments on the cluster change on their next update.

What changes once a cluster derives

Every platform default that changed since the cluster was registered applies wherever no cluster, Workspace, or Deployment override sets the key.
Keys that later APC releases add to the platform defaults apply everywhere APC reads them. Without derivation, such a key reaches the cluster only where APC reads that key on its own.
Values set in the snapshot without a cluster override are dropped. Record them as cluster overrides before you opt in.
A cluster override that withdraws a key with DELETE_KEY resolves to the current platform default. On a cluster that resolves from its snapshot, the withdrawn key keeps the value in the snapshot. See Remove a setting.
Restoring a removed cluster setting brings back the platform default, whether or not the cluster derives.

Limitations

APC doesn’t store a cluster’s past resolved configuration. To reconstruct what a deriving cluster resolved at an earlier time, you need the APC API configuration as it was at that time.
An APC API Pod that misses a cache update can serve the previous configuration for up to about two minutes.
APC still writes the snapshot to the cluster record. On a deriving cluster, the snapshot in Cluster.config isn’t the configuration that Deployments use. Read Cluster.effectiveConfig instead.

​How derivation is controlled

​When running Deployments change

​Clusters registered on APC 2.0.3 or later 2.0.x releases

​Prerequisites

​Derive a cluster’s configuration

​Roll back

​What changes once a cluster derives

​Limitations

How derivation is controlled

When running Deployments change

Clusters registered on APC 2.0.3 or later 2.0.x releases

Prerequisites

Derive a cluster’s configuration

Roll back

What changes once a cluster derives

Limitations