Skip to main content
When you supply your own Airflow metadata database or result backend connection for a Deployment, some Astro Private Cloud (APC) versions can write those credentials in plaintext to APC API logs and audit events. Use this guide to check whether you’re affected, stop new log entries, and contain credentials that are already in your logs.

​
Check if you’re affected

You’re affected if both of the following are true:
  • Your installation runs, or ran, an affected APC version.
  • You supply the database connection for one or more Deployments yourself, either in the metadataConnection, metadataConnectionJson, resultBackendConnection, or resultBackendConnectionJson arguments of the upsertDeployment mutation, or through a pre-deployment webhook.
This guide covers the APC API logs and audit events only. The credentials appear in the following locations:
  • The logs of the APC API and APC Worker Pods in your platform namespace.
  • Any system that collects those Pod logs, such as a security information and event management (SIEM) system, a data lake, or a log archive in cold storage.
  • APC audit events, including events for failed requests, and the sink that receives them.
The APC Vector DaemonSet doesn’t collect APC API logs, so the Elasticsearch indices for Airflow logs don’t contain these credentials.

​
Step 1: Stop new log entries

Upgrade to APC 2.1.1 or later. APC 2.1.1 redacts customer-managed database credentials written to log lines. No 2.0.x release includes the fix. For details, see the APC 2.1.1 release notes. If you run 2.0.x or 2.1.0 and can’t upgrade yet, contact Astronomer support. If you run 1.0.x, 1.1.x, 1.2.x, or 0.37.x and can’t upgrade yet, set the APC API log level to warn. The affected log lines use the info level, so this setting stops new entries. It also removes all other info log lines from the APC API. Add the following to your values.yaml file, then apply the config change:

​
Step 2: Restrict access to the logs

Limit who can read the affected logs while you rotate the credentials and remove the old entries.
  • In Kubernetes, restrict the get permission on pods/log in your platform namespace to the smallest group of users that need it. To check a user’s access, run kubectl auth can-i get pods/log -n <platform-namespace> --as <user>.
  • In every system that collects APC API or APC Worker Pod logs, restrict read access to the indices, tables, or buckets that hold them.
  • Restrict read access to the sink that receives your audit events.

​
Step 3: Rotate the exposed credentials

Because these credentials were stored in plaintext, Astronomer recommends that you rotate them. An upgrade or log level change affects only new log entries. Credentials in existing entries remain valid until you change them. For each affected Deployment:
  1. In your database, set a new password for the database user, or create a new database user.
  2. Update the Deployment connection in the same location you set it originally: the upsertDeployment mutation or your pre-deployment webhook. See Use a MySQL or PostgreSQL database for metadata or storage.
  3. Confirm that the Deployment’s Airflow components restart and connect to the database.
  4. Remove the old password or the old database user.
If a Deployment uses data plane failover, rotate the credentials for its database users on both data plane clusters.
On an affected version, the APC API logs the new credentials when you update the Deployment. Complete Step 1 first if you can do it quickly. If you can’t, rotate the credentials now, and contain the new log entries in the same way as the old ones.

​
Step 4: Purge the affected log entries

After you rotate the credentials, the old values in your logs no longer grant access. Remove or redact those entries anyway, so that a credential you missed during rotation doesn’t stay exposed.
  1. In every system that keeps APC API or APC Worker Pod logs, search for entries that contain metadataConnection or resultBackendConnection. Start the search from the date you first supplied a connection for an affected Deployment.
  2. Delete the matching entries, or redact the credentials in them.
  3. Do the same in the sink that receives your audit events.
  4. Repeat the search in copies of these logs, such as SIEM archives, data lake tables, cold storage buckets, and backups.
Each system keeps these entries for as long as its retention policy allows. Include systems that you don’t search regularly, such as archives and backups.