Troubleshoot, upgrade, and roll back worker autoscaling

This page covers what happens to worker queues when KEDA or Laminar is missing or stops, and how to upgrade to or roll back from Astro Private Cloud 2.2 with worker autoscaling.

​

Hibernation and autoscaling

Hibernation also needs KEDA and Laminar on the data plane. Removing KEDA, or setting global.keda.enabled to false, stops both. When hibernation is on for a Deployment and KEDA is installed, every queue gets a KEDA scaled object, including fixed-size queues, because hibernation pauses workers through KEDA. A fixed-size queue’s scaled object keeps its minimum and maximum equal, so its worker count doesn’t change.

​

When KEDA or Laminar is missing or stops

​

KEDA or Laminar was never available

This applies when KEDA isn’t installed, global.keda.enabled is false, Laminar is off, or the data plane still runs Astro Private Cloud 2.1.

• Every queue runs a fixed worker count.
• With the autoscaling setting on, the warnings at the top of the Deployment settings page show Autoscaling is unavailable: This deployment's data plane does not have both KEDA and Laminar, which worker autoscaling needs. Each queue runs a fixed worker count until they are installed.
• The APC API refuses a range with WorkerQueueRangeUnsupportedError, and a minimum of 0 with WorkerQueueZeroMinimumUnsupportedError. Both messages name KEDA and Laminar together.

The message doesn’t say which piece is missing. Check KEDA, global.keda.enabled, global.laminar.enabled, and the data plane’s platform version in that order.

​

KEDA is removed or Laminar is turned off

The control plane notices within a few minutes and reports autoscaling as available until then. Afterward, the warnings at the top of the Deployment settings page show Autoscaling has stopped, naming the affected queue or giving a count of affected queues: This deployment's data plane no longer has both KEDA and Laminar, which worker autoscaling needs. The warning shows as an error when a queue would run zero workers. You can dismiss it, but it returns when you reload the page. Scaling history disappears from the queue cards at the same time. What happens to each queue:

1. Until the Deployment’s next update, each queue keeps the worker count KEDA last set. That can be zero for a queue with a minimum of 0.
2. At the Deployment’s next update, each queue runs the larger of its replicas and its minimum. For a queue saved from the Astro Private Cloud UI, that’s its minimum. For example, a queue set to 1 to 10 workers that was running 9 drops to 1, and the 8 extra workers stop after the termination grace period. Every Airflow component restarts, because parallelism changes.
3. The range stays stored. When KEDA and Laminar return, each queue scales within its range again at the Deployment’s next update.

To keep a queue at a specific size while autoscaling is unavailable, click Edit on the queue, enter the count in Num workers, click Save, and click Deploy Changes. Saving a queue from the Astro Private Cloud UI while autoscaling is unavailable replaces its range with that single count, so set the range again after autoscaling returns.

​

KEDA stops running but stays installed

If the KEDA operator stops (for example, it crashes or is scaled to zero), the platform doesn’t notice. It still reports autoscaling as available and accepts ranges.

• Each queue keeps the worker count it had when KEDA stopped. Tasks wait in the queue instead of starting new workers.
• KEDA’s fallback doesn’t apply, because the KEDA operator runs it.
• A queue at zero workers can’t start a worker.
• Scaling resumes on its own when KEDA runs again.

Monitor KEDA’s operator yourself. A minimum of 1 or more keeps a queue running tasks while KEDA is down.

​

Laminar stops responding

If KEDA can’t get a worker count from Laminar, it keeps the current count for the first two failed requests. After three failed requests in a row, it moves each queue to its minimum. A queue with a minimum of 0 stops all of its workers. Scaling resumes when Laminar responds again. Nothing in the platform alerts you when this happens.

​

An administrator turns worker autoscaling off

Turning deployments.workerQueues.autoscaling.enabled off behaves like losing KEDA, with two differences. The warning names the setting instead of the data plane: Worker autoscaling has been turned off for this deployment. And the change reaches each Deployment at its next update, not at a time the data plane decides.

​

Upgrade to Astro Private Cloud 2.2

Worker queues and worker autoscaling both ship turned off. Upgrading to 2.2 doesn’t change how any Deployment’s workers run until an administrator turns them on.

• Turn on autoscaling after every data plane runs 2.2. A data plane still on 2.1 reports KEDA and Laminar as missing, so a Deployment on it shows Autoscaling is unavailable and tells you to install KEDA and Laminar, when the fix is to upgrade the data plane.
• Each existing Deployment gets a default queue after you turn on worker queues. The one-time backfill creates it, or the Deployment’s first save does. Every save from the Astro Private Cloud UI counts. An APC API update counts only when it sends workers or workerQueues. See The default queue. The queue reproduces the worker size and count the Deployment already runs, with equal minimum and maximum, so it doesn’t autoscale. The first update that applies the queue restarts every Airflow component once, because parallelism moves from the platform’s fixed 32 to the value derived from the queue. For example, a Deployment that runs 3 workers with no concurrency set and one scheduler moves from 32 to 48 (3 × 16).
• On a cluster with an overprovisioning factor below 1, that first update also raises each worker’s resource requests to its limits, because queue workers always run with requests equal to limits. For example, with a CPU factor of 0.5, a worker with a 2000-millicore limit requests 1000 millicores before the update and 2000 after it. Larger requests can leave a worker Pending on a full cluster. See Overprovision Airflow components.
• The Airflow resource can look like it has a range when it doesn’t. A queue that doesn’t autoscale shows the operator’s defaults, minWorkers: 1 and maxWorkers: 10, in the Deployment’s Airflow resource. Those aren’t a range the platform applied or refused. Read the queue’s stored range from the APC API instead:
  query deployment($releaseName: String) {
    deployment(where: { releaseName: $releaseName }) {
      workerQueues {
        name
        replicas
        minWorkerCount
        maxWorkerCount
      }
    }
  }
  

For the full upgrade procedure, see Upgrade to a minor version.

​

Roll back to Astro Private Cloud 2.1

Astro Private Cloud 2.1 has no worker queues. After a control plane rollback, the next update to a Deployment rebuilds its workers as a single group and removes every other queue. The operator deletes the removed queues’ workers without a grace period, so their running tasks stop, and tasks still routed to those queues wait with no worker to run them. This happens at each Deployment’s next update, not at rollback. If you roll back with helm upgrade to the 2.1 chart, the platform updates every Deployment as part of the upgrade, so it happens to all of them at once, unless the upgrade command sets astronomer.houston.upgradeDeployments.enabled=false. If you roll back with helm rollback, it happens later, when each Deployment is next updated. A Deployment with only the default queue, with equal minimum and maximum, loses nothing in a rollback. Its workers restart once, because 2.1 sets parallelism back to 32. For example, a default queue of 3 workers with no concurrency set goes from 48 back to 32. Roll back in this order:

1. On 2.2, reduce every Deployment to a single fixed-size default queue. Turn worker autoscaling off, set each queue’s minimum equal to its maximum, move Dag code off any named queue, and then delete the named queues. Deleting a queue stops its workers without a grace period, so delete each one when it’s idle.
2. Roll back the control plane.
3. Roll back each data plane. This removes Laminar and the scaling identity from the KEDA namespace.

Before and during the rollback:

• Export every Deployment’s queue list through the APC API, with the query in Upgrade to Astro Private Cloud 2.2, so you can enter the queues again later. The Astro Private Cloud UI on 2.1 doesn’t show them.
• Don’t update Deployments between rolling back the control plane and finishing the rollback. An update is what applies the collapse.
• Don’t patch the operator’s resource definitions by hand. A helm upgrade reverts the patch.
• After the data plane rollback, check your KEDA namespace for objects labelled astronomer.io/platform-release and remove any that remain.

If you upgrade to 2.2 again, each Deployment’s stored queues and ranges return at its next update. Worker changes made while on 2.1 are replaced by the stored queues.

​

Known limitations

• The platform detects only whether KEDA is installed. It doesn’t check KEDA’s version or whether KEDA is running. A stopped KEDA freezes every queue at its current count, with no warning in the platform. See KEDA stops running but stays installed.
• Messages name KEDA and Laminar together and don’t say which is missing. With global.keda.enabled set to false, they say KEDA is missing even when it’s installed. On a data plane that still runs 2.1, they say to install KEDA and Laminar when you need to upgrade the data plane.
• A wrong global.keda.namespace has no warning. If it names a namespace KEDA doesn’t run in, queues stay at their minimum and the platform still reports autoscaling as available. Find the namespace KEDA runs in with kubectl get deployment -A --field-selector metadata.name=keda-operator.
• Changes reach a Deployment only at its next update. Installing or removing KEDA or toggling the autoscaling setting doesn’t change a Deployment’s workers until then.
• The Astro Private Cloud UI doesn’t check Max workers against the per-queue limit. A maximum above the limit is refused only when you save.
• The UI suggests a minimum of 0 even when the platform doesn’t allow it. With the default worker minimum of 1, the Min workers hint still offers 0, and the save fails with INTERNAL_SERVER_ERROR. See Scale a queue to zero.
• Nothing explains a queue held below its maximum by parallelism. A customer-set AIRFLOW__CORE__PARALLELISM can stop a queue short of its maximum, and Scaling history still says it stayed within its range. See Parallelism.
• The Current workers bar has no pending state. Workers waiting for room on the cluster appear only in Scaling history, only for queues with a range, and only after they wait long enough to count as a finding.
• Editing or deleting a queue gives no warning about what stops. Lowering a maximum stops running workers, changing a maximum or concurrency restarts every component, and deleting a queue stops its workers without a grace period. See Scale-down and restarts.
• One scaling identity serves every Deployment on the cluster. A token for metrics-api-worker-trigger can read the worker counts of every Deployment on that data plane.
• Nothing alerts when Laminar stops responding. Queues drop to their minimum with no warning. See Laminar stops responding.
• The APC API stores a range while autoscaling is turned off. Where KEDA and Laminar are present, a range saved with the setting off is accepted and does nothing until the setting is turned on.
• The platform doesn’t compare control plane, data plane, and operator versions. An operator whose resource definitions lack the trigger field drops it without an error. See Verify the data plane.
• Worker autoscaling doesn’t apply to adopted Deployments. Astro Private Cloud doesn’t write the worker settings of an adopted Deployment.
• The UI doesn’t say that an empty worker concurrency means 16. The Worker concurrency field says it inherits the Deployment default, and AIRFLOW__CELERY__WORKER_CONCURRENCY doesn’t change it. See If you set the worker concurrency environment variable.

​

Related documentation

• Configure worker autoscaling
• Configure worker queues
• Hibernate Deployments
• Upgrade to a minor version