Watch the recording
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.
How to pass data between your Airflow tasks
Recorded on
Tamara FingerlinSenior Developer AdvocateAstronomer
Kenten DanasDirector, Developer RelationsAstronomer
This webinar provides a deep dive into passing data between your Airflow tasks. We cover everything you need to know to choose and implement a method for passing data based on your use case, infrastructure, scale, and more. Questions covered in this session include:
- What are some best practices to follow when using XComs, Airflow’s built-in cross-communication utility?
- How can I pass data between tasks and DAGs using TaskFlow and traditional operators?
- How does the Astro Python SDK use XComs to move data between relational stores and Python data structures, and how can it simplify my pipelines?
- How can I set up a custom XCom backend and implement custom serialization methods?
- How can I pass data between tasks that are run in isolated environments like the KubernetesPodOperator?
All code covered in this webinar can be found in this repo.
