Database architecture

Astro Private Cloud (APC) uses PostgreSQL as the primary database system for both platform management (the APC API) and Apache Airflow Deployments. The following sections describe the database architecture, provisioning, connection pooling, and high availability configuration.

Database components

APC database

The APC API uses a PostgreSQL database to store platform configuration and state. Key tables:

User - System users with authentication credentials
Workspace - Organizational units for grouping deployments
Deployment - Airflow deployment configurations
Cluster - Data plane cluster properties including cloud provider, region, and configuration
RoleBinding - User and service account role assignments
ServiceAccount - API keys for programmatic access
DeployRevision - Deployment version history
TaskUsage - Task execution metrics

Airflow metadata database

Each Airflow Deployment gets its own isolated PostgreSQL database containing two schemas:

Schema	Purpose
`airflow`	Airflow scheduler and webserver metadata
`celery`	Celery task results backend

The database name is derived from the Deployment release name:

deployment-name → deployment_name_airflow

Each schema has a separate database user with permissions scoped to that schema. The deployment orchestrator provisions the database and both schemas automatically when you create a Deployment. For more information about what Airflow stores in the metadata database, see Understanding the Airflow metadata database.

Database provisioning

Automatic provisioning

By default, the deployment orchestrator automatically provisions databases for new Deployments. No additional configuration is required.

External database

To use pre-existing or managed databases, set skipAirflowDatabaseProvisioning to true in the upsertDeployment mutation:

mutation {
  upsertDeployment(
    workspaceUuid: "<workspace-uuid>"
    label: "<my-deployment-label>"
    skipAirflowDatabaseProvisioning: true
  ) {
    id
  }
}

When using external databases, provide the connection string in your Deployment configuration. For complete setup steps with connection string examples, see Bring your own Airflow database. For supported PostgreSQL versions, see the Apache Airflow database backend documentation.

Connection pooling (PgBouncer)

APC uses PgBouncer for connection pooling to reduce database connection overhead. Airflow can open many database connections due to its distributed nature, and each PostgreSQL connection creates a dedicated OS process. PgBouncer reduces this overhead by maintaining a pool of reusable server connections. For more detail on why PgBouncer is recommended, see the Apache Airflow Helm chart production guide.

PgBouncer proxies connections to whichever database backend you configure, whether that’s the embedded PostgreSQL container or an external database you provide through a metadataConnectionString URI. Enabling PgBouncer provides connection pooling either way.

PgBouncer operates in transaction pool mode by default, which means server connections are returned to the pool after each transaction completes. This mode doesn’t support session-level features such as prepared statements or SET commands that persist across transactions.

Configuration

pgbouncer:
  enabled: true
  port: 6543
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m

Pool sizes

Pool sizes are configured per Deployment in the Airflow chart values:

pgbouncer:
  metadataPoolSize: 10       # Airflow metadata connections per pool
  resultBackendPoolSize: 5   # Celery result backend connections per pool
  maxClientConn: 100         # Maximum client connections to PgBouncer

These defaults match the Apache Airflow Helm chart defaults and work for most Deployments. To determine whether you need to adjust pool sizes, monitor PgBouncer’s cl_waiting metric. If clients consistently wait for connections, increase pool sizes or scale PgBouncer replicas.

Airflow also maintains its own SQLAlchemy connection pool (default pool_size: 5, max_overflow: 10) between each Airflow component and PgBouncer. With these defaults, each Airflow process can open up to 15 simultaneous database connections. The full connection chain is: Airflow component → SQLAlchemy pool → PgBouncer → PostgreSQL.

Kerberos support

PgBouncer in APC includes Kerberos support:

pgbouncer:
  image:
    repository: quay.io/astronomer/ap-pgbouncer-krb
    tag: 1.25.0-3

For complete Kerberos database setup instructions, see Configure Kerberos authentication for Airflow databases.

High availability

PostgreSQL replication

Astronomer doesn’t recommend using internal PostgreSQL instance. If you need production database resiliency, use an externally managed database service such as Amazon RDS, Google Cloud SQL, or Azure Database for PostgreSQL. Managed services provide built-in replication, automated failover, and backup capabilities that are more reliable than running replication within the Helm chart.

Replication is disabled by default. If you run the embedded PostgreSQL database despite the previous warning, the following example enables streaming replication with synchronous commit:

postgresql:
  replication:
    enabled: true
    slaveReplicas: 2
    synchronousCommit: "on"
    numSynchronousReplicas: 1
    user: repl_user

Persistence

postgresql:
  persistence:
    enabled: true
    size: 8Gi
    storageClass: ""  # Use default storage class

Default values and tuning starting points

The following tables compare the chart default values with tuning starting points based on Astronomer operational experience. The Default column reflects the values shipped in the Helm chart. Adjust these values based on the number of Deployments, workload scale, and observed resource utilization in your environment. For additional production guidance, see the Apache Airflow Helm chart production guide. Astronomer recommends using a managed database service such as Amazon RDS, Google Cloud SQL, or Azure Database for PostgreSQL rather than the embedded PostgreSQL container for production workloads.

PostgreSQL

Astronomer doesn’t recommend or support the embedded PostgreSQL container for production use. The following values are tuning starting points if you run the embedded database beyond its defaults for non-production use. For production workloads, use a managed database service instead.

Setting	Default	Tuning starting point
`postgresql.resources.requests.cpu`	250m	1000m. Adjust based on query load
`postgresql.resources.requests.memory`	256Mi	2Gi. Adjust based on active connection count
`postgresql.resources.limits.cpu`	1000m	2000m. Adjust based on query load
`postgresql.resources.limits.memory`	2Gi	4Gi. Adjust based on active connection count
`postgresql.persistence.enabled`	`true`	`true`
`postgresql.persistence.size`	8Gi	50Gi or more. Scale based on the number of Deployments and retention policy
`postgresql.replication.enabled`	`false`	`true`
`postgresql.replication.slaveReplicas`	1	2
`postgresql.replication.synchronousCommit`	`"off"`	`"on"`
`postgresql.replication.numSynchronousReplicas`	0	1

PgBouncer (platform level)

Setting	Default	Recommended for production
`pgbouncer.resources.requests.cpu`	250m	250m
`pgbouncer.resources.requests.memory`	256Mi	256Mi
`pgbouncer.resources.limits.cpu`	250m	500m. Adjust based on connection throughput
`pgbouncer.resources.limits.memory`	256Mi	512Mi. Adjust based on connection count

PgBouncer (per Deployment)

These values match the Apache Airflow Helm chart defaults. No official formula ties pool sizes to a specific workload metric such as Dag count. Instead, adjust pool sizes based on observed connection utilization.

Setting	Default	Recommended for production
`pgbouncer.metadataPoolSize`	10	10. Increase if `cl_waiting` is consistently non-zero
`pgbouncer.resultBackendPoolSize`	5	5. Increase for Deployments with high Celery task throughput
`pgbouncer.maxClientConn`	100	100. Increase if Deployments run many concurrent Airflow components

Backup and recovery

For database backup and restore procedures, including size estimation, pg_dump/mysqldump commands, and restore steps, see Access Airflow database.

Connection strings

APC database

The APC database connection depends on whether you use an external database or the in-cluster PostgreSQL. You configure this during control plane installation. For external databases, provide the connection through houston.backendConnection in your Helm values:

houston:
  backendConnection:
    user: <username>
    pass: <password>
    host: <external-database-host>
    port: 5432
    db: <database-name>

Alternatively, provide a pre-existing Kubernetes secret name through houston.backendSecretName. The secret must contain a connection key with the full connection URI.

Custom schemas aren’t supportedAPC stores its platform tables in the houston$default schema and doesn’t support a different schema. When APC provisions the database, it creates this schema for you. When you supply the connection URI yourself, for example through houston.backendSecretName, add the schema to the URI:

postgresql://<username>:<password>@<host>:5432/<database>?schema=houston%24default

The URI is URL-encoded, so the $ in the schema name becomes %24. Without this parameter, APC creates its tables in the public schema and the database migrations may fail with errors such as relation "houston$default.Email" does not exist.

To retrieve the active APC database connection string, read the astronomer-bootstrap secret:

kubectl get secret -n astronomer astronomer-bootstrap \
  -o jsonpath='{.data.connection}' | base64 -d

The in-cluster PostgreSQL option (global.postgresql.enabled: true) is only for development or proof-of-concept environments and isn’t supported in production.

Airflow database

Both the metadata and result backend connections use the same database with different schemas and credentials:

# Metadata connection (airflow schema)
postgresql://<metadata-user>:<password>@<host>:5432/<deployment>_airflow

# Result backend connection (celery schema)
postgresql://<celery-user>:<password>@<host>:5432/<deployment>_airflow

When PgBouncer is enabled, Airflow components connect through PgBouncer on port 6543 instead of directly to PostgreSQL on port 5432. To retrieve the connection string for a specific Deployment, see Access Airflow database.

Monitoring

Monitor the following metrics to track database and connection pool health:

Metric	Description
`pg_stat_activity`	Active database connections
`pg_database_size`	Database size on disk
`pgbouncer_pools_*`	PgBouncer connection pool statistics, including `cl_waiting`

Best practices

Enable replication for production Deployments.
Use PgBouncer to manage connection overhead.
Monitor connection pools for cl_waiting and database size growth.
Configure regular backups. See Access Airflow database.
Size pools based on observed connection utilization, not Dag count.
Use separate credentials for each Deployment.
Enable SSL for database connections in production.
Use a managed database service for production workloads instead of the in-cluster PostgreSQL container.

Overview

Architecture

Upgrade Astro Private Cloud

Install Astro Private Cloud

Develop Dags

Airflow administration

Airflow 3

Deploy Dags

Manage Deployments

Platform administration

Manage Platform resources

Security and compliance

APC API

Platform Observability

Release notes

Reference

Database architecture

Database components

APC database

Airflow metadata database

Database provisioning

Automatic provisioning

External database

Connection pooling (PgBouncer)

Configuration

Pool sizes

Kerberos support

High availability

PostgreSQL replication

Persistence

Default values and tuning starting points

PostgreSQL

PgBouncer (platform level)

PgBouncer (per Deployment)

Backup and recovery

Connection strings

APC database

Airflow database

Monitoring

Best practices

​Database components

​APC database

​Airflow metadata database

​Database provisioning

​Automatic provisioning

​External database

​Connection pooling (PgBouncer)

​Configuration

​Pool sizes

​Kerberos support

​High availability

​PostgreSQL replication

​Persistence

​Default values and tuning starting points

​PostgreSQL

​PgBouncer (platform level)

​PgBouncer (per Deployment)

​Backup and recovery

​Connection strings

​APC database

​Airflow database

​Monitoring

​Best practices

Database components

APC database

Airflow metadata database

Database provisioning

Automatic provisioning

External database

Connection pooling (PgBouncer)

Configuration

Pool sizes

Kerberos support

High availability

PostgreSQL replication

Persistence

Default values and tuning starting points

PostgreSQL

PgBouncer (platform level)

PgBouncer (per Deployment)

Backup and recovery

Connection strings

APC database

Airflow database

Monitoring

Best practices