Skip to main content

Configure XCOM backend for a Remote Execution Agent

Airflow 3 only
This feature is only available for Airflow 3.x Deployments.

XCom (cross-communication) enables you to move data between tasks, or cross communicate between tasks in Airflow. By default, Airflow uses the metadata database to store XComs, but you can configure a custom XCom backend. Because the Airflow metadata database exists in the Astro Orchestration Plane, not in the Execution Plane with the Remote Execution Agent, you must configure a custom XCom Backend to make optimal use of them in your environment.

This procedure describes how to set up the XCom Backend in the Remote Execution Agent components.

Basic configuration​

In your Agent's values.yaml file, set the following configuration.


xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

tip

You can use the recommendations described in the Strategies for custom XCom backends in Airflow with the Remote Execution Agent.

However, there is one difference. Instead of directly setting the AIRFLOW_CORE_XCOM_BACKEND environment variable, the xcomBackend field in your values.yaml file configures this setting for you.:

xcomBackend: "AIRFLOW__CORE__XCOM_BACKEND"

Example: Amazon S3 custom XCom backend configuration​

The following example shows the configuration you can use if you use Amazon S3 in your environment.

xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

commonEnv:
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
    value: "s3://<connection-id>@<bucket>/<path-to-xcom>"
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
    value: 0  # Zero means always store in the object store
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
    value: "zip"

Was this page helpful?