Configure cleanup jobs

Configure automated cleanup jobs to maintain database health by removing old data. Astro Private Cloud (APC) includes several cleanup jobs that run as CronJobs on configurable schedules to manage storage growth and query performance.

Cleanup jobs summary

Job	Default Schedule	Default Retention	Purpose
`cleanupDeployments`	Daily @ 00:00	14 days	Removes soft-deleted deployments
`cleanupDeployRevisions`	Daily @ 23:11	90 days	Archive deploy history
`cleanupTaskUsageData`	Daily @ 23:40	90 days	Purge task metrics
`cleanupClusterAudits`	Daily @ 23:49	90 days	Remove cluster audit logs
`cleanupAirflowDb`	Daily @ 05:23	365 days	Clean Airflow metadata (disabled by default)

`cleanupDeployments`

Permanently removes deployments that have been soft-deleted after the retention period.

What gets cleaned

Deployment database records marked with deletedAt
Associated Docker registry images
Deployment metadata database

Configuration

houston:
  cleanupDeployments:
    enabled: true
    schedule: "0 0 * * *"      # Midnight daily
    olderThan: 14              # Days since deletion
    dryRun: false              # Set true to preview

Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-deployments --older-than=14 --dry-run=false

`cleanupDeployRevisions`

Removes old deployment revision records to reduce database size.

What gets cleaned

deployRevision records older than retention period
Historical deployment configuration snapshots

Configuration

houston:
  cleanupDeployRevisions:
    enabled: true
    schedule: "11 23 * * *"    # 23:11 daily
    olderThan: 90              # Days to retain

Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-deploy-revisions --older-than=90

Per-deployment cleanup

Run this command from a machine with access to the underlying Kubernetes cluster to clean revisions for a specific deployment:

kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-deploy-revisions --older-than=90 --deploymentUuid=<uuid>

`cleanupTaskUsageData`

Purges task usage metrics and audit logs.

What gets cleaned

TaskUsage records (daily aggregated metrics)
TaskUsageAuditLog records (raw task data)

Configuration

houston:
  cleanupTaskUsageData:
    enabled: true
    schedule: "40 23 * * *"    # 23:40 daily
    olderThan: 90              # Minimum 90 days
    dryRun: false

Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-task-usage-data --older-than=90 --dry-run=false

GraphQL trigger

Use the cleanupTaskUsageDataJob query to manually trigger a purge of task usage metrics and audit logs:

query {
  cleanupTaskUsageDataJob(olderThan: 90)
}

Minimum retention is 90 days and can’t be reduced.

`cleanupClusterAudits`

Removes cluster audit log entries.

What gets cleaned

ClusterAudit records tracking cluster configuration changes
Historical cluster state snapshots

Configuration

houston:
  cleanupClusterAudits:
    enabled: true
    schedule: "49 23 * * *"    # 23:49 daily
    olderThan: 90              # Days to retain

Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-cluster-audit --older-than=90

Filter by cluster

Run this command from a machine with access to the underlying Kubernetes cluster to clean audits for specific clusters:

kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-cluster-audit --older-than=90 --cluster-ids=<id1>,<id2>

`cleanupAirflowDb`

Cleans Airflow metadata from individual Deployment databases.

This job is disabled by default due to potential impact on running Deployments.

What gets cleaned

Default tables:

callback_request - Task callback requests
celery_taskmeta, celery_tasksetmeta - Celery metadata
dag - Dag definitions
dag_run - Dag execution history
dataset_event - Dataset events
import_error - Import errors
job - Job records
log - Task execution logs
session - Session data
sla_miss - SLA violations
task_fail - Task failures
task_instance - Task execution records
task_reschedule - Reschedule events
trigger - Trigger records
xcom - Cross-communication data

Configuration

houston:
  cleanupAirflowDb:
    enabled: false             # Must explicitly enable
    schedule: "23 5 * * *"     # 05:23 daily
    olderThan: 365             # Days to retain
    outputPath: "/tmp"         # Archive location
    dropArchives: true         # Delete after archiving
    dryRun: false
    provider: local            # Storage: local/aws/azure/gcp
    bucketName: "/tmp"         # Cloud bucket or local path
    tables: ""                 # Specific tables (empty = all)

Cloud storage export

Export archived data to cloud storage:

houston:
  cleanupAirflowDb:
    enabled: true
    provider: aws              # aws, azure, or gcp
    bucketName: "my-archive-bucket"
    providerEnvSecretName: "aws-credentials-secret"

Specific tables only

Clean only specific tables:

houston:
  cleanupAirflowDb:
    enabled: true
    tables: "log,task_instance,xcom"

Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-airflow-db-data \
  --older-than=365 \
  --provider=local \
  --bucket-name=/tmp \
  --tables="log,task_instance"

Schedule reference

Default schedules are staggered to avoid simultaneous execution:

Time	Job
00:00	`cleanupDeployments`
05:23	`cleanupAirflowDb`
23:11	`cleanupDeployRevisions`
23:40	`cleanupTaskUsageData`
23:49	`cleanupClusterAudits`

Common configuration options

All cleanup jobs share these options:

houston:
  cleanup<JobName>:
    enabled: true/false        # Enable/disable the job
    schedule: "cron-expression"  # When to run
    olderThan: <days>          # Retention period
    dryRun: false              # Preview without deleting
    readinessProbe: {}         # Optional health probes
    livenessProbe: {}

Kubernetes CronJob behavior

All cleanup CronJobs use:

Concurrency policy: Forbid (prevents overlapping runs)
Backoff limit: 1 retry on failure
Restart policy: Never

Monitor cleanup jobs

Check job status

# List all cleanup CronJobs
kubectl get cronjobs -n astronomer | grep cleanup

# View recent job runs
kubectl get jobs -n astronomer | grep cleanup

# Check job logs
kubectl logs job/<job-name> -n astronomer

# Trigger individual jobs manually
kubectl create job --from=cronjobs/jobname jobname-hash -n astronomer

Verify data cleanup

-- Check remaining records by date
SELECT DATE(created_at), COUNT(*)
FROM deploy_revision
GROUP BY DATE(created_at)
ORDER BY DATE(created_at) DESC;

Troubleshooting

Job not running

Check CronJob exists:

kubectl get cronjob houston-cleanup-deployments -n astronomer

Check job is enabled in Helm values.
Verify schedule syntax is valid cron expression.

Job failing

Check job logs:

kubectl logs job/houston-cleanup-deployments-<timestamp> -n astronomer

Database connectivity: Ensure the APC API can reach the database.
Permissions: Verify service account has required database permissions.

Data not being cleaned

Check retention period: Data younger than olderThan won’t be deleted.
Verify timestamps: Check createdAt/deletedAt values in database.
Run with dry-run: Preview what would be deleted.

Best practices

Monitor database size before and after cleanup jobs
Start with dry-run when adjusting retention periods
Stagger schedules if adding custom cleanup jobs
Archive before delete for cleanupAirflowDb in production
Set alerts for failed cleanup jobs

Apply platform configuration

​Cleanup jobs summary

​cleanupDeployments

​What gets cleaned

​Configuration

​Manual trigger

​cleanupDeployRevisions

​What gets cleaned

​Configuration

​Manual trigger

​Per-deployment cleanup

​cleanupTaskUsageData

​What gets cleaned

​Configuration

​Manual trigger

​GraphQL trigger

​cleanupClusterAudits

​What gets cleaned

​Configuration

​Manual trigger

​Filter by cluster

​cleanupAirflowDb

​What gets cleaned

​Configuration

​Cloud storage export

​Specific tables only

​Manual trigger

​Schedule reference

​Common configuration options

​Kubernetes CronJob behavior

​Monitor cleanup jobs

​Check job status

​Verify data cleanup

​Troubleshooting

​Job not running

​Job failing

​Data not being cleaned

​Best practices

​Related documentation

Cleanup jobs summary

`cleanupDeployments`

What gets cleaned

Configuration

Manual trigger

`cleanupDeployRevisions`

What gets cleaned

Configuration

Manual trigger

Per-deployment cleanup

`cleanupTaskUsageData`

What gets cleaned

Configuration

Manual trigger

GraphQL trigger

`cleanupClusterAudits`

What gets cleaned

Configuration

Manual trigger

Filter by cluster

`cleanupAirflowDb`

What gets cleaned

Configuration

Cloud storage export

Specific tables only

Manual trigger

Schedule reference

Common configuration options

Kubernetes CronJob behavior

Monitor cleanup jobs

Check job status

Verify data cleanup

Troubleshooting

Job not running

Job failing

Data not being cleaned

Best practices

Related documentation