System environment variables
On Astro, certain environment variables have special handling. There are three types of special handling:- Override variables have their value set by Astro, regardless of what you set in your Deployment.
- Default Change variables have values that are different from the values in open source Apache Airflow. But, if you specify a value, your value takes precedence.
- Unsafe to Change variables are environment variables used by Astro to function. Do not override variables listed as Unsafe to Change, because it can break Astro functionality.
The Astro UI does not currently prevent you from setting the environment variables listed as Unsafe to Change. Attempting to set them can result in unexpected behavior that can include access problems, missing task logs, and failed tasks.If you need to set one of these variables for a particular use case, contact Astronomer support.
Default Changed
| Environment Variable | Applicability | Value |
|---|---|---|
AIRFLOW__TRIGGERER__DEFAULT_CAPACITY | Both Airflow 2 and 3 | 1000 |
AIRFLOW__CORE__PARALLELISM | Both Airflow 2 and 3 | based on worker queue settings |
AIRFLOW__DAG_PROCESSOR__FILE_PARSING_SORT_MODE | Only applies if you use HA or have an extra large size Deployment. Both Airflow 2 and 3. | random_seeded_by_host |
AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL | Airflow 3 | 30 |
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL | Airflow 2 | 30 |
AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE | Both Airflow 2 and 3 | 16 |
AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE | Both Airflow 2 and 3 | dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number|default(ti.try_number) }}/{{ ti.id }}.log (for AstroExecutor) |
AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT | Both Airflow 2 and 3 | 600 (for <12.5.0,11.15.0), 300 otherwise |
OPENLINEAGE_DISABLED=True | Airflow 3 Remote Deployments | |
AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD | Both Airflow 2 and 3 | 60, unless on Astro Executor, then 120 |
AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT | Both Airflow 2 and 3 | 60, unless on Astro Executor, then 120 |
AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION | Airflow 2 | FALSE |
AIRFLOW__KUBERNETES_EXECUTOR__WORKER_POD_PENDING_FATAL_CONTAINER_STATE_REASONS | Only applies if you are on Runtime Version 11.15.0 or higher. Both Airflow 2 and 3. | CreateContainerConfigError,CreateContainerError,ImageInspectError,InvalidImageName |
Override
| Environment Variable | Applicability | Value |
|---|---|---|
AIRFLOW__ASTRONOMER__CASBIN_DEPLOYMENT | Both Airflow 2 and 3 | Your Deployment ID |
ASTRO_LOGGING_ROLE_ARN | Both Airflow 2 and 3 | |
AIRFLOW_CONN_ASTRO_GCS_LOGGING | Both Airflow 2 and 3 | URI |
AIRFLOW_CONN_ASTRO_AZURE_LOGS | Both Airflow 2 and 3 | |
AIRFLOW_CONN_ASTRO_S3_LOGGING | Both Airflow 2 and 3 | |
AIRFLOW__WEBSERVER__STATIC_CDN | Both Airflow 2 and 3 | https://cdn.astronomer.io/airflow/ |
AIRFLOW__API__BASE_URL | Airflow 3 only | https://{orgid}.astronomer.run/d{deployment_id_suffix} |
AIRFLOW__OPENSEARCH__HOST | Airflow 3 only | “” |
ASTRO_AGENT_CLIENT_DISALLOW_DEFAULT_SECRET_BACKEND | Airflow 3 Astro Hosted | FALSE |
ASTRO_AGENT_CLIENT_DISALLOW_DEFAULT_XCOM_BACKEND | Airflow 3 Astro Hosted | FALSE |
AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST | Airflow 3 hosted | |
AIRFLOW__ASTRO__TASK_LOG_URL_PATTERN | Airflow 3 Remote | URI |
AIRFLOW__METRICS__STATSD_ALLOW_LIST | Both Airflow 2 and 3 | Variable is removed |
AIRFLOW__METRICS__STATSD_STATSD_CUSTOM_CLIENT_PATH | Both Airflow 2 and 3 | Variable is removed |
AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE | Both Airflow 2 and 3 | Variable is removed |
AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST | Both Airflow 2 and 3 | Variable is removed |
AIRFLOW__API__PORT | Both Airflow 2 and 3 | 9091 |
AIRFLOW__API_AUTH__JWT_ALGORITHM | Both Airflow 2 and 3 | EdDSA |
AIRFLOW__API_AUTH__JWT_KID | Both Airflow 2 and 3 | |
AIRFLOW__API_AUTH__JWT_PRIVATE_KEY_PATH | Both Airflow 2 and 3 | /var/run/secrets/airflow/api-token-secret/api-token-signing-key |
AIRFLOW__API_AUTH__TRUSTED_JWKS_URL | Both Airflow 2 and 3 | /var/run/secrets/airflow/api-token-secret/api-token-jwks-key |
AIRFLOW__CELERY__BROKER_URL | Both Airflow 2 and 3 | |
AIRFLOW__CORE__FERNET_KEY | The secret key for saving connection passwords in the metadata database. Both Airflow 2 and 3. | fernetKeySecret |
AIRFLOW__CORE__LAZY_LOAD_PLUGINS | Both Airflow 2 and 3 | FALSE |
AIRFLOW__DATABASE__CHECK_MIGRATIONS | Both Airflow 2 and 3 | FALSE |
AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS | Both Airflow 2 and 3 | fab in Airflow 2, fab+astro in Airflow 3 |
AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE | Both Airflow 2 and 3 | |
AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE | Both Airflow 2 and 3 | reference to mounted /usr/local/airflow/pod_template_file.yaml |
AIRFLOW__LOGGING__COLORED_CONSOLE_LOG | Both Airflow 2 and 3 | FALSE |
AIRFLOW__OPERATORS__DEFAULT_QUEUE | Both Airflow 2 and 3 | Worker queue defined as default, usually "default" |
AIRFLOW__SCHEDULER__STANDALONE_DAG_PROCESSOR | Both Airflow 2 and 3 | True/False depending on Scheduler Size |
AIRFLOW__WEBSERVER__SECRET_KEY | Both Airflow 2 and 3 | |
AIRFLOW__CELERY__RESULT_BACKEND | Both Airflow 2 and 3 | |
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN | Both Airflow 2 and 3 | |
AIRFLOW__EXECUTION_API__JWT_AUDIENCE | Both Airflow 2 and 3 | |
ASTRO_ORGANIZATION_ID | Both Airflow 2 and 3 | Your Astro Organization ID |
ASTRO_WORKSPACE_ID | Both Airflow 2 and 3 | Your Astro Workspace ID |
ASTRO_DEPLOYMENT_ID | Both Airflow 2 and 3 | Your Astro Deployment ID |
ASTRO_DEPLOYMENT_NAMESPACE | Both Airflow 2 and 3 | Your Astro Deployment Namespace |
Unsafe to Change
| Environment Variable | Description | Value |
|---|---|---|
OPENLINEAGE_NAMESPACE | Used internally by Astro for lineage and alerting. Do not change or overwrite. | |
AIRFLOW__OPENLINEAGE__NAMESPACE | Used internally by Astro for lineage and alerting. Do not change or overwrite. | |
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_TIMEOUT | The amount of time that the Celery executor waits for a response from the Celery backend before throwing an error. Both Airflow 2 and 3. | 30 |
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_CONNECT_TIMEOUT | The amount of time that the Celery executor will attempt to connect to the Celery backend before retrying. Both Airflow 2 and 3 | 5 |
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_KEEPALIVE | Whether the Celery executor will check whether the connection to the Celery backend is still alive. Both Airflow 2 and 3 | TRUE |
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__RETRY_ON_TIMEOUT | Whether the Celery executor will retry a connection to the Celery backend when the connection fails. Both Airflow 2 and 3 | TRUE |
AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET | Routes scheduler logs to stdout. Both Airflow 2 and 3 | stdout |
AIRFLOW__LOGGING__REMOTE_LOGGING | Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments. | TRUE |
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER | Location of remote logging storage. Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments. | baseLogFolder |
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID | ID of the connection that provides access to your remote logging location. Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments. | |
AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS | Class name that specifies the logging configuration. Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments. | astronomer.runtime.logging.logging_config |
AIRFLOW_CONN_ASTRO_S3_LOGGING | Connection URI for writing task logs to Astro’s managed S3 bucket. Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments. | |
AIRFLOW__LOGGING__ENCRYPT_S3_LOGS | Determines whether to use server-side encryption for S3 logs. Both Airflow 2 and 3. | FALSE |
AIRFLOW__WEBSERVER__BASE_URL | The base URL of the Airflow UI. Both Airflow 2 and 3 | https://${fullIngressHostname |
AIRFLOW__CORE__SQL_ALCHEMY_CONN | The SqlAlchemy connection string for the metadata database. Both Airflow 2 and 3. | dbConnSecret. |
AIRFLOW__WEBSERVER__UPDATE_FAB_PERMS | Determines whether to update FAB permissions on webserver startup. Airflow 2 Only. | TRUE |
AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX | Determines whether to enable werkzeug ProxyFix middleware for reverse proxy. Airflow 2 only. | TRUE |
AIRFLOW__API__ENABLE_PROXY_FIX | Determines whether to enable werkzeug ProxyFix middleware for reverse proxy. Airflow 3 only. | TRUE |
AIRFLOW__CORE__EXECUTOR | The executor class that Airflow uses. Astro supports the Celery and Kubernetes executor. Both Airflow 2 and 3. | executor |
AIRFLOW_HOME | The home directory for an Astro project. Both Airflow 2 and 3. | usr/local/airflow |
AIRFLOW__KUBERNETES__NAMESPACE | The Kubernetes namespace where Airflow workers are created. Both Airflow 2 and 3. | namespace |
AIRFLOW__CORE__HOSTNAME_CALLABLE | Path to a callable, which resolves to the hostname. Both Airflow 2 and 3. | airflow.utils.net.get_host_ip_address |
AIRFLOW__SCHEDULER__STATSD_ON | Determines whether StatsD is on. Both Airflow 2 and 3. | TRUE |
AIRFLOW__SCHEDULER__STATSD_HOST | The hostname for StatsD. Both Airflow 2 and 3. | statsd.Hostname |
AIRFLOW__SCHEDULER__STATSD_PORT | The port for StatsD. Both Airflow 2 and 3. | <statsd-port> |
AIRFLOW__METRICS__STATSD_ON | Determines whether metrics are sent to StatsD. Both Airflow 2 and 3. | TRUE |
AIRFLOW__METRICS__STATSD_HOST | The hostname for sending metrics to StatsD. Both Airflow 2 and 3. | statsd.Hostname |
AIRFLOW__METRICS__STATSD_PORT | The port for sending metrics to StatsD. Both Airflow 2 and 3. | <statsd-metrics-port> |
AIRFLOW__METRICS__STATSD_PREFIX | The prefix for sending the metrics to StatsD. Both Airflow 2 and 3. | airflow |
AIRFLOW__WEBSERVER__COOKIE_SECURE | Sets a secure flag on server cookies. Airflow 2 only. | TRUE |
AIRFLOW__WEBSERVER__INSTANCE_NAME | Shows the name of your Deployment in the Home view of the Airflow UI. Airflow 2. | <Deployment-Name> |
AIRFLOW__API__INSTANCE_NAME | Shows the name of your Deployment in the Home view of the Airflow UI. Airflow 3. | <Deployment-Name> |
AIRFLOW__CELERY__WORKER_CONCURRENCY | Determines how many tasks each Celery worker can run at any given time and is the basis of worker auto-scaling logic | <Max-Tasks-Per-Worker> |
AIRFLOW__WEBSERVER__EXPOSE_CONFIG | Exposes the Configuration tab of the Airflow UI and hides sensitive values. Airflow 2 | NON-SENSITIVE-ONLY |
AIRFLOW__API__EXPOSE_CONFIG | Exposes the Configuration tab of the Airflow UI and hides sensitive values. Airflow 3. | NON-SENSITIVE-ONLY |
AIRFLOW__USAGE_DATA_COLLECTION__ENABLED | Disables the Airflow usage data collection & reporting to Scarf. Both Airflow 2 and 3. | FALSE |
AWS_DEFAULT_REGION | (AWS clusters only) The region where your cluster is located. Both Airflow 2 and 3. | The region where you configured your cluster. |
AWS_SECRET_ACCESS_KEY | The key secret for accessing Astro’s managed S3 bucket¹. Both Airflow 2 and 3 | |
INSTANCE_TYPE | Provides the instance size of the node the dag is scheduled on. Both Airflow 2 and 3 | (v1:metadata.labels['beta.kubernetes.io/instance-type']) |
OPENLINEAGE_URL | The URL for your Astro lineage backend. The destination for lineage metadata sent from external systems to the OpenLineage API. Both Airflow 2 and 3 | |
OPENLINEAGE_API_KEY | Your OpenLineage API key. Both Airflow 2 and 3 |