System environment variables
On Astro, certain environment variables have special handling. There are three types of special handling:- Override variables have their value set by Astro, regardless of what you set in your Deployment.
- Default change variables have values that are different from the values in open source Apache Airflow. But, if you specify a value, your value takes precedence.
- Unsafe to change variables are environment variables used by Astro to function. Do not override variables listed as Unsafe to change, because it can break Astro functionality.
The Astro UI does not currently prevent you from setting the environment variables listed as Unsafe to change. Attempting to set them can result in unexpected behavior that can include access problems, missing task logs, and failed tasks.If you need to set one of these variables for a particular use case, contact Astronomer support.
Default changed
Override
Unsafe to change
XCom and state store backends
On Astro Hosted, Astro stores XCom values larger than 64 KB in an Astro-managed object storage bucket instead of the Airflow metadata database. On Astro Runtime 3.3 and later, Astro also stores task and asset state values larger than 64 KB in the same bucket. Values of 64 KB or less stay in the metadata database. To do this, Astro sets the following environment variables on Deployments that run Astro Runtime 3.0 or later:AIRFLOW__CORE__XCOM_BACKENDAIRFLOW__WORKERS__STATE_STORE_BACKEND, on Astro Runtime 3.3 and later only
How Astro stores values
- Astro moves only values that tasks write after Astro sets the backend. Values that tasks wrote earlier stay in the metadata database and remain readable.
- The bucket is in the same Data plane as your Deployment. Each Deployment can read only its own values.
- You can’t access the bucket directly. To read or change a value, use a task, the Airflow UI, or the Airflow REST API.
- The bucket doesn’t limit value size. The practical limit is the memory of the worker that runs your task, because the task holds the full value in memory while it reads or writes the value.
Messages in place of a value
When the Airflow UI or the Airflow REST API can’t show a value from the bucket, it shows one of the following messages instead. The message starts with the type of value:xcom, task state, or asset.
If you save one of these messages as a value, the Airflow API server refuses the change. Enter the value itself, or delete the entry.
Use your own XCom or state store backend
Astro keeps your own backend when you set the variable as an Astro environment variable, either on the Deployment or as a Workspace-level environment variable linked to the Deployment. If you set the variable anywhere in your Astro project image, such as with anENV instruction in your Dockerfile, Astro replaces it with its own backend.
To move a backend from your image to a Deployment environment variable:
- Open your Astro project
Dockerfileand find theENVinstructions that setAIRFLOW__CORE__XCOM_BACKENDorAIRFLOW__WORKERS__STATE_STORE_BACKEND. - Add each variable to your Deployment with the same value. See Use the Astro UI.
AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH, can stay in your Dockerfile. You can also keep the backend ENV instructions for local development, because the Deployment environment variable takes precedence on Astro.
To switch a Deployment back to the Astro backend, delete the variable from the Deployment and from any Workspace-level environment variables linked to it. The same risk to in-progress Dag runs applies.