Set up APC audit log shipping

This document explains how to enable the Astro Private Cloud (APC) audit log sidecar and ship events to one supported sink. For background on what the feature does and which configurations are supported, see APC audit logging overview.

Exactly one sink can be enabled per installation for this release. Enabling the sidecar with zero or more than one sink causes a Helm validation error.

Prerequisites

An Astro Private Cloud 2.x installation that you can upgrade with Helm.
Access to the Helm values file used by the installation.
kubectl configured against the target cluster.
Permissions to create or modify cloud resources for the sink you choose:
- CloudWatch on EKS: permissions to create IAM policies and roles, and a CloudWatch log group.
- GCP Cloud Logging on GKE: permissions to create Google service accounts and grant IAM bindings in the target project.
- Elasticsearch: a reachable endpoint and, if required, credentials and a CA certificate.

Choose a sink

Use the AWS CloudWatch Logs sink when the Astro Private Cloud control plane runs on Amazon EKS. Use the GCP Cloud Logging sink when it runs on Google Kubernetes Engine (GKE). Use the Elasticsearch sink on Amazon EKS, GKE, or Azure Kubernetes Service (AKS).

AWS CloudWatch Logs
GCP Cloud Logging
Elasticsearch

Use this sink when the Astro Private Cloud control plane runs on Amazon EKS. The recommended authentication method is IAM Roles for Service Accounts (IRSA). Static AWS credentials held in a Kubernetes secret are supported as a fallback when IRSA isn’t in use on the EKS cluster.

Prerequisites

The Astro Private Cloud control plane runs on Amazon EKS.
The AWS CLI is installed and authenticated against the target account.
For the IRSA path, the EKS cluster has, or can be associated with, an OIDC identity provider.
For the static-credentials path, an IAM principal with permission to write to the target CloudWatch log group.

Environment variables

The following variables are referenced throughout this section. Set them to match your installation before running the commands.

export EKS_CLUSTER_NAME="<cluster-name>"
export AWS_REGION="<region>"
export AWS_ACCOUNT_ID="<account-id>"
export K8S_NAMESPACE="astronomer"
export HELM_RELEASE="astronomer"
export K8S_SA="${HELM_RELEASE}-houston-bootstrapper"
export IRSA_ROLE_NAME="HoustonCloudWatchRole"
export CW_LOG_GROUP="/astronomer/houston/audit"

Configure IRSA (recommended)

Create the CloudWatch log group

aws logs create-log-group \
  --log-group-name "$CW_LOG_GROUP" \
  --region "$AWS_REGION"

Optionally set a retention policy on the log group:

aws logs put-retention-policy \
  --log-group-name "$CW_LOG_GROUP" \
  --region "$AWS_REGION" \
  --retention-in-days 30

Associate the EKS cluster OIDC provider

OIDC_URL=$(aws eks describe-cluster \
  --name "$EKS_CLUSTER_NAME" \
  --region "$AWS_REGION" \
  --query "cluster.identity.oidc.issuer" \
  --output text)
OIDC_ID=${OIDC_URL#https://}

aws iam list-open-id-connect-providers \
  | grep "$(echo "$OIDC_ID" | awk -F/ '{print $NF}')" \
  || eksctl utils associate-iam-oidc-provider \
       --cluster "$EKS_CLUSTER_NAME" \
       --region "$AWS_REGION" \
       --approve

Create the IAM policy

cat > /tmp/houston-cloudwatch-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudWatchDescribe",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:*"
    },
    {
      "Sid": "CloudWatchWriteAuditGroup",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}",
        "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}:*"
      ]
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name HoustonCloudWatchLogsPolicy \
  --policy-document file:///tmp/houston-cloudwatch-policy.json

Create the IRSA role and attach the policy

cat > /tmp/trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_ID}:sub": "system:serviceaccount:${K8S_NAMESPACE}:${K8S_SA}",
          "${OIDC_ID}:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

aws iam create-role \
  --role-name "$IRSA_ROLE_NAME" \
  --assume-role-policy-document file:///tmp/trust-policy.json

aws iam attach-role-policy \
  --role-name "$IRSA_ROLE_NAME" \
  --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/HoustonCloudWatchLogsPolicy"

If the IRSA role already exists and you only need to bind it to a new EKS cluster, use aws iam update-assume-role-policy against the existing role instead of aws iam create-role.

Apply the Helm values override

Add the following to the values file used by the installation and run helm upgrade:

astronomer:
  houston:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/HoustonCloudWatchRole"
    logging:
      loggingSidecar:
        enabled: true
        cloudwatch:
          enabled: true
          region: "<AWS_REGION>"
          logGroupName: "/astronomer/houston/audit"
          useIRSA: true

Configure static AWS credentials (fallback)

Use this configuration on EKS when IRSA isn’t in use.

Create the CloudWatch log group

aws logs create-log-group \
  --log-group-name "$CW_LOG_GROUP" \
  --region "$AWS_REGION"

Create a Kubernetes secret with AWS credentials

kubectl create secret generic houston-cloudwatch-creds \
  --from-literal=aws_access_key_id="<AWS_ACCESS_KEY_ID>" \
  --from-literal=aws_secret_access_key="<AWS_SECRET_ACCESS_KEY>" \
  -n "$K8S_NAMESPACE"

The IAM principal whose credentials you use must be allowed to write to the target log group. The policy shown in the IRSA section is a suitable template.

Apply the Helm values override

astronomer:
  houston:
    logging:
      loggingSidecar:
        enabled: true
        cloudwatch:
          enabled: true
          region: "<AWS_REGION>"
          logGroupName: "/astronomer/houston/audit"
          useIRSA: false
          secretName: "houston-cloudwatch-creds"

Verify

After the upgrade completes, confirm that the Vector sidecar is running and that audit events are reaching CloudWatch:

HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
  -l component=houston \
  -o jsonpath='{.items[0].metadata.name}')

kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20

aws logs tail "$CW_LOG_GROUP" --region "$AWS_REGION" --since 5m

Perform any action that the APC API audits, such as creating a Workspace, and confirm a matching event appears in the log group within a few seconds.

Use this sink when the Astro Private Cloud control plane runs on Google Kubernetes Engine (GKE). The recommended authentication method is Workload Identity. A GCP service account JSON key held in a Kubernetes secret is supported as a fallback when Workload Identity isn’t in use on the GKE cluster.

Prerequisites

The Astro Private Cloud control plane runs on GKE.
gcloud is installed and authenticated against the target Google Cloud project.
For the Workload Identity path, the GKE cluster has Workload Identity enabled.
For the service-account-key path, a Google service account with the roles/logging.logWriter role in the target Google Cloud project.

Environment variables

export GCP_PROJECT_ID="<project-id>"
export GKE_CLUSTER_NAME="<cluster-name>"
export GKE_LOCATION="<zone-or-region>"
export K8S_NAMESPACE="astronomer"
export HELM_RELEASE="astronomer"
export K8S_SA="${HELM_RELEASE}-houston-bootstrapper"
export GCP_SA_NAME="houston-logging"
export GCP_SA_EMAIL="${GCP_SA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com"

Configure Workload Identity (recommended)

Verify Workload Identity is enabled

gcloud container clusters describe "$GKE_CLUSTER_NAME" \
  --location "$GKE_LOCATION" \
  --format="value(workloadIdentityConfig.workloadPool)"

Expected output is ${GCP_PROJECT_ID}.svc.id.goog. If the output is empty, enable Workload Identity on the cluster before continuing.

Create the Google service account and grant log write access

gcloud iam service-accounts create "$GCP_SA_NAME" \
  --display-name="Houston Audit Log Writer" \
  --project="$GCP_PROJECT_ID"

gcloud projects add-iam-policy-binding "$GCP_PROJECT_ID" \
  --member="serviceAccount:${GCP_SA_EMAIL}" \
  --role="roles/logging.logWriter"

Bind the Google service account to the APC API Kubernetes service account

gcloud iam service-accounts add-iam-policy-binding "$GCP_SA_EMAIL" \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${K8S_SA}]" \
  --project="$GCP_PROJECT_ID"

Apply the Helm values override

astronomer:
  houston:
    serviceAccount:
      annotations:
        iam.gke.io/gcp-service-account: "<GCP_SA_EMAIL>"
    logging:
      loggingSidecar:
        enabled: true
        gcpCloudLogging:
          enabled: true
          projectId: "<GCP_PROJECT_ID>"
          logId: "houston-audit"
          resource:
            type: "k8s_container"
            location: "<GKE_LOCATION>"
            clusterName: "<GKE_CLUSTER_NAME>"
          severityKey: "level"
          useWorkloadIdentity: true

projectId, resource.location, and resource.clusterName are required when gcpCloudLogging.enabled is true. Leaving any of these empty causes a Helm validation error.

Configure a service account key (fallback)

Use this configuration on GKE when Workload Identity isn’t in use.

Create a key for the Google service account

gcloud iam service-accounts keys create /tmp/houston-logging-key.json \
  --iam-account="$GCP_SA_EMAIL"

Create a Kubernetes secret from the key

kubectl create secret generic houston-gcp-logging-creds \
  --from-file=key.json=/tmp/houston-logging-key.json \
  --namespace "$K8S_NAMESPACE"

rm /tmp/houston-logging-key.json

Apply the Helm values override

astronomer:
  houston:
    logging:
      loggingSidecar:
        enabled: true
        gcpCloudLogging:
          enabled: true
          projectId: "<GCP_PROJECT_ID>"
          logId: "houston-audit"
          resource:
            type: "k8s_container"
            location: "<GKE_LOCATION>"
            clusterName: "<GKE_CLUSTER_NAME>"
          severityKey: "level"
          useWorkloadIdentity: false
          credentialsSecretName: "houston-gcp-logging-creds"
          credentialsSecretKey: "key.json"

Verify

HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
  -l component=houston \
  -o jsonpath='{.items[0].metadata.name}')

kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20

gcloud logging read \
  "logName=\"projects/${GCP_PROJECT_ID}/logs/houston-audit\"" \
  --project="$GCP_PROJECT_ID" \
  --limit=5 \
  --freshness=5m

Perform any action that the APC API audits and confirm a matching entry appears in the log stream within a few seconds.

Use this sink to ship APC API audit events to an external Elasticsearch cluster. The Elasticsearch sink is supported when the Astro Private Cloud control plane runs on Amazon EKS, GKE, or Azure Kubernetes Service (AKS). This sink needs no cloud-provider IAM configuration.

Prerequisites

A reachable Elasticsearch endpoint on version 8.
For basic auth, a username and password.
For custom CA trust, a PEM-formatted CA certificate.

Environment variables

export K8S_NAMESPACE="astronomer"
export HELM_RELEASE="astronomer"
export ES_ENDPOINT="https://es.example.com:9200"
export ES_USERNAME="elastic"
export ES_PASSWORD="<password>"

Configure basic auth

Create a Kubernetes secret with the Elasticsearch credentials

kubectl create secret generic houston-elasticsearch-creds \
  --from-literal=username="$ES_USERNAME" \
  --from-literal=password="$ES_PASSWORD" \
  -n "$K8S_NAMESPACE"

Apply the Helm values override

astronomer:
  houston:
    logging:
      loggingSidecar:
        enabled: true
        elasticsearch:
          enabled: true
          endpoint: "https://es.example.com:9200"
          index: "houston-audit-%Y.%m.%d"
          apiVersion: "v8"
          auth:
            strategy: "basic"
            secretName: "houston-elasticsearch-creds"

Configure basic auth with a custom CA

Use this variant when the Elasticsearch endpoint presents a certificate signed by a private or internal CA that isn’t already trusted by the Vector sidecar.

Create a Kubernetes secret with the credentials

kubectl create secret generic houston-elasticsearch-creds \
  --from-literal=username="$ES_USERNAME" \
  --from-literal=password="$ES_PASSWORD" \
  -n "$K8S_NAMESPACE"

Create a Kubernetes secret with the CA certificate

The secret key must be named ca.pem. The Vector sidecar expects the CA at /etc/es-tls/ca.pem and the chart mounts the referenced secret to that path.

kubectl create secret generic houston-es-ca \
  --from-file=ca.pem=/path/to/ca-certificate.pem \
  -n "$K8S_NAMESPACE"

Apply the Helm values override

astronomer:
  houston:
    logging:
      loggingSidecar:
        enabled: true
        elasticsearch:
          enabled: true
          endpoint: "https://es.example.com:9200"
          index: "houston-audit-%Y.%m.%d"
          apiVersion: "v8"
          auth:
            strategy: "basic"
            secretName: "houston-elasticsearch-creds"
          tls:
            enabled: true
            caSecretName: "houston-es-ca"

Configure anonymous auth

Use this configuration when the Elasticsearch endpoint accepts unauthenticated requests. This configuration needs no credentials secret.

astronomer:
  houston:
    logging:
      loggingSidecar:
        enabled: true
        elasticsearch:
          enabled: true
          endpoint: "http://es.example.com:9200"
          index: "houston-audit-%Y.%m.%d"
          apiVersion: "v8"
          auth:
            strategy: "none"

Configure anonymous auth with a custom CA

Use this variant when the Elasticsearch endpoint accepts unauthenticated requests over TLS and presents a certificate signed by a private or internal CA.

Create a Kubernetes secret with the CA certificate

The secret key must be named ca.pem. The Vector sidecar expects the CA at /etc/es-tls/ca.pem and the chart mounts the referenced secret to that path.

kubectl create secret generic houston-es-ca \
  --from-file=ca.pem=/path/to/ca-certificate.pem \
  -n "$K8S_NAMESPACE"

Apply the Helm values override

astronomer:
  houston:
    logging:
      loggingSidecar:
        enabled: true
        elasticsearch:
          enabled: true
          endpoint: "https://es.example.com:9200"
          index: "houston-audit-%Y.%m.%d"
          apiVersion: "v8"
          auth:
            strategy: "none"
          tls:
            enabled: true
            caSecretName: "houston-es-ca"

Verify

Check the Vector sidecar logs for sink health:

HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
  -l component=houston \
  -o jsonpath='{.items[0].metadata.name}')

kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20

Query the Elasticsearch endpoint for the houston-audit-* indices. If the endpoint requires authentication:

curl --cacert /path/to/ca-certificate.pem -u "$ES_USERNAME:$ES_PASSWORD" \
  "$ES_ENDPOINT/_cat/indices/houston-audit-*?v"

If the endpoint accepts anonymous requests:

curl --cacert /path/to/ca-certificate.pem \
  "$ES_ENDPOINT/_cat/indices/houston-audit-*?v"

Omit the --cacert flag when the endpoint uses a publicly trusted certificate. Perform any action that the APC API audits and confirm that a new document appears in the matching houston-audit-* index.

Disable audit log shipping

To stop shipping audit events, set the sidecar to disabled and run helm upgrade:

astronomer:
  houston:
    logging:
      loggingSidecar:
        enabled: false

helm upgrade removes the Vector sidecar from the next Pod restart. The APC API continues to emit audit events to standard output, so you can still inspect recent events with kubectl logs on the APC API and APC Worker Pods.

Common issues

Helm reports 'supports exactly one sink at a time'

The sidecar is enabled with more than one sink. Edit the values file so that only one of cloudwatch.enabled, gcpCloudLogging.enabled, or elasticsearch.enabled is true, then run helm upgrade again.

Helm reports 'requires at least one supported sink'

The sidecar is enabled but no sink is selected. Set one of cloudwatch.enabled, gcpCloudLogging.enabled, or elasticsearch.enabled to true, or set loggingSidecar.enabled to false.

Helm reports a required GCP Cloud Logging value is missing

gcpCloudLogging.projectId, gcpCloudLogging.resource.location, and gcpCloudLogging.resource.clusterName are required when GCP Cloud Logging is enabled. Whitespace-only values are also rejected. Set all three to concrete values and run helm upgrade again.

Vector reports AccessDenied errors on CloudWatch

The IAM principal that Vector uses can’t write to the target log group.

For IRSA, check that the IRSA role trust policy names the correct OIDC provider, namespace, and service account, and that HoustonCloudWatchLogsPolicy is attached to the role. Also verify that the eks.amazonaws.com/role-arn annotation on houston.serviceAccount.annotations points to the same role ARN.
For static credentials, check that houston-cloudwatch-creds contains valid aws_access_key_id and aws_secret_access_key values and that the corresponding IAM user or role is allowed to write to the target log group.

GCP Cloud Logging rejects entries with HTTP 400

The monitored resource is invalid. Confirm that resource.type is k8s_container, and that resource.location and resource.clusterName match the GKE cluster as it appears in Cloud Logging.

Vector can't connect to Elasticsearch

Check that endpoint is reachable from within the cluster and that, if TLS is in use, the CA certificate in caSecretName signs the server certificate presented by the endpoint. If basic auth is enabled, verify that the username and password in houston-elasticsearch-creds are correct.

Next steps

For a field-by-field description of each value under houston.logging.loggingSidecar, see Audit logging configuration reference.
For the shape of each audit event and the list of audited operations, see Audit log schema and operations.

Overview

Architecture

Upgrade to Astro Private Cloud 2.0

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

APC API

Platform Observability

Release notes

Reference

Set up APC audit log shipping

Prerequisites

Choose a sink

Prerequisites

Environment variables

Configure IRSA (recommended)

Configure static AWS credentials (fallback)

Verify

Prerequisites

Environment variables

Configure Workload Identity (recommended)

Configure a service account key (fallback)

Verify

Prerequisites

Environment variables

Configure basic auth

Configure basic auth with a custom CA

Configure anonymous auth

Configure anonymous auth with a custom CA

Verify

Disable audit log shipping

Common issues

Next steps

​Prerequisites

​Choose a sink

​Prerequisites

Environment variables

Configure IRSA (recommended)

​Configure static AWS credentials (fallback)

​Verify

​Prerequisites

Environment variables

Configure Workload Identity (recommended)

​Configure a service account key (fallback)

​Verify

​Prerequisites

Environment variables

​Configure basic auth

​Configure basic auth with a custom CA

​Configure anonymous auth

Configure anonymous auth with a custom CA

​Verify

​Disable audit log shipping

​Common issues

​Next steps

Prerequisites

Choose a sink

Prerequisites

Configure static AWS credentials (fallback)

Verify

Prerequisites

Configure a service account key (fallback)

Verify

Prerequisites

Configure basic auth

Configure basic auth with a custom CA

Configure anonymous auth

Verify

Disable audit log shipping

Common issues

Next steps