How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari

When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world.

Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber’s approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs.

Key Takeaways:

(02:03) Airflow as a service streamlines Uber’s data workflows.

(06:16) Serialization boosts security and reduces errors.

(10:05) Java-based scheduler improves system reliability.

(13:40) Custom recovery model supports emergency pipeline switching.

(15:58) No-code UI allows easy pipeline creation for non-coders.

(18:12) Backfill feature enables historical data processing.

(22:06) Regular updates keep Uber aligned with Airflow advancements.

(26:07) Plans to leverage Airflow’s latest features.

Resources Mentioned:

Shobhit Shah

Sumit Maheshwar

Apache Airflow

Airflow Summit

Uber

Apache Airflow Survey

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning