Airflow Summit 2021 Highlights
Airflow Summit 2021 has just ended with a bang! We’ve reached 10K+ data-driven attendees from all around the world. The lineup covered topics from the first commits to the exciting future of Apache Airflow®, and we couldn’t be happier about being a part of that inspiring event.
Seven days of talks were hosted by the global network of meetups (shoutout to London, Warsaw, Bangalore, Tokyo, Bay Area, Tel Aviv, New York, Melbourne!) and moderated by the vibrant Airflow community. “Community over the code” motto was genuinely alive, and we felt the immense energy coming from the organizers, speakers, and attendees.
The conference pictured the whole Airflow ecosystem and the power of the #community. We were impressed by the diverse group of speakers from various industries and presenting the Airflow journeys from many perspectives. With Pinterest, Societe Generale, PayPal, Wise on stage, it was visible that Airflow has its momentum. All the talks are available, and we've also compiled a short guide to the Astronomer's ones if you missed them!
Astronomer on Stage
The Summit was opened by Kaxil Naik, the Airflow PMC member & Committer and Manager - Airflow Engineering at Astronomer. He shared his journey of becoming a leading contributor to Apache Airflow®. Interestingly, Kaxil's first PR to Airflow was a fixed typo, not yet knowing Python.
Ry Walker, the Co-Founder of Astronomer, had a great discussion with Jey Sen, Data Platform Architect at Paypal, on Airflow as the Foundation of a Multi-Faceted Data Platform. It focused on building solid integrations with Airflow — one of the main themes of Airflow Summit 2021.
If you want to know what's next for Apache Airflow® after the 2.0 release, you should watch the talk by two prominent Airflow PMC members, aizhamal-nurmamat kyzy and Ash Berlin-Taylor. The future looks bright, and we loved the combination of code and non-code engagements complementing each other and making Airflow even stronger!
Andrew Godwin, the Principal Engineer at Astronomer, highlighted the core concepts of Airflow architecture, including DAGs, Operators, and Tasks.
In his talk "Apache Airflow® and Ray: Orchestrating ML at Scale", Daniel Imberman showcased the power of Airflow integration with Ray, which provides a simple, universal API for building distributed applications.
Two of the Astronomer's Founders, Ry Walker and Greg Neiheisel, shed light on what we have worked on lately, the Next-Gen Astronomer Cloud. And talking about next-gen, Kaxil Naik, in his second talk, provided a guide for a smooth transition to Airflow 2.0., with an important pit-stop at Airflow 1.10.15 as a bridge release.
Ash Berlin-Taylor entered the virtual stage for the second time to give you a deep dive into the Apache Airflow® Scheduler and tips for taming that complex beast. In another technical talk, Plinio Guzman, Ecosystem Architect at Astronomer, discussed building secure, high-quality DAGs in the Airflow Ecosystem.
On the final day of the Airflow Summit 2021, Kenten Danas highlighted another great Airflow integration. This time for productionizing Machine Learning Pipelines with Airflow, Kedro, and Great Expectations.
Kaxil was joined by Jarek Potiuk, Technical Advisor at Astronomer and Independent Open-Source Contributor and Advisor, in his third appearance. They talked about the reasons why Airflow loves Kubernetes—starting from an official container image, through quick-start docker-compose configuration, culminating in April with the official Helm Chart for Airflow release.
In the last Astronomer talk at the Airflow Summit 2021, Vikram Koka & Ephraim Anierobi talked about customizing Xcom to enhance data sharing between tasks.
The second edition of the Airflow Summit 2021 has ended, but we’re not stopping here. Stay tuned for more!
