- When you should use a custom XCom backend.
- How to set up a custom XCom backend using the Object Storage XCom Backend.
- How to use a custom XCom backend class with custom serialization and deserialization methods.
Assumed knowledge
To get the most benefits from this guide, you need an understanding of:- XCom basics. See Pass data between tasks.
- Basic knowledge of a cloud-based object storage service like AWS S3, GCP Cloud Storage or Azure Blob Storage.
Why use a custom XCom backend?
Common reasons to use a custom XCom backend include:- You need more storage space for XComs than the Airflow metadata database can offer.
- You’re running a production environment where you require custom retention, deletion, and backup policies for XComs.
- You want to access XComs without accessing the metadata database.
- You want to restrict types of allowed XCom values.
- You want to save XComs in multiple locations simultaneously.
How to set up a custom XCom backend
There are two main ways to set up a custom XCom backend:- Object Storage XCom Backend: Use this method to create a custom XCom backend when you want to store XComs in a cloud-based object storage service like AWS S3, GCP Cloud Storage, or Azure Blob Storage. This option is recommended if you need to store XComs in a single remote location, and the Object Storage XCom Backend threshold and compression options meet your requirements.
- Custom XCom backend class: Use this method when you want to further customize how XComs are stored, for example to simultaneously store XComs in two different locations.
Use the Object Storage XCom Backend
You can create a custom XCom backend using object storage. The Object Storage XCom Backend is part of the Common IO provider and can be defined using the following environment variables:AIRFLOW__CORE__XCOM_BACKEND: The XCom backend to use. Set this toairflow.providers.common.io.xcom.backend.XComObjectStorageBackendto use the Object Storage XCom Backend.AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH: The path to the object storage where XComs are stored. The path should be in the format<your-scheme>://<your-connection-id@<your-bucket>/xcom. For example,s3://my-s3-connection@my-bucket/xcom. The most common schemes ares3,gs, andabfsfor Amazon S3, Google Cloud Storage, and Azure Blob Storage, respectively.AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD: The threshold in bytes for XComs to be stored in the object storage. All objects smaller or equal to this threshold are stored in the metadata database. All objects larger than this threshold are stored in the object storage. The default value is-1, meaning all XComs are stored in the metadata database.AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION: Optional. The compression algorithm to use when storing XComs in the object storage, for examplezip. The default value isNone.
To use the Object Storage XCom Backend with task loops in Airflow 3.4 and later, you need a version of the Common IO provider later than 1.10.1. With version 1.10.1 or earlier, Airflow fails to parse Dags that use task loops.
Use a custom XCom backend class
To create a custom XCom backend, you need to define an XCom backend class which inherits from theBaseXCom class.
The following example MyCustomXComBackend class only allows JSON-serializable XComs and stores them in both Amazon S3 and Google Cloud Storage. It reads XComs back from the Amazon S3 bucket.
The Airflow metadata database stores a reference string to the XCom, which is displayed in the XCom tab of the Airflow UI. The reference string is prefixed with s3_and_gs:// to indicate that the XCom is stored in both Amazon S3 and Google Cloud Storage.
You can add any serialization and deserialization logic to the methods that store and read XComs. See Custom serialization and deserialization for more information.
- Airflow 3.4+
- Airflow 3.3 and earlier
In Airflow 3.4 and later, override the
store_value() and load_value() methods. This change supports task loops. If your XCom backend still stores values with serialize_value(), Airflow fails to parse Dags that use task loops.include directory of your Airflow project. Then, set the AIRFLOW__CORE__XCOM_BACKEND environment variable in your Airflow instance to the path of the custom XCom backend class. If you run Airflow locally with the Astro CLI, you can set the environment variable in the .env file of your Astro project. On Astro, you can set the environment variable in the Astro UI.
Custom serialization and deserialization
By default, Airflow includes serialization methods for common object types like JSON, pandas DataFrames and NumPy. If you need to pass data objects through XCom that aren’t supported, you have several options:- Register a custom serializer, see Serialization.
- Add a
serialize()anddeserialize()method to the class of the object you want to pass through XCom, see Serialization. - Use a custom XCom backend to define custom serialization and deserialization methods, see Use a custom XCom backend class.