Airflow in Action: Model Training with PBs of New Data Weekly. DataOps & MLOps Insights from Ford
Ford Motor Company has long been an innovator in the automotive industry. From pioneering the assembly line to advancing safety features, Ford has played a pivotal role in shaping modern transportation.
In recent years, the company has increased investments in AI and ML to better assist drivers and improve customer experience, power its smart mobility strategy, and foster advances across the broader automotive industry. ML engineers from the company have recently shared two key use cases at Ford, both using Apache Airflow® and Astro, the managed service from Astronomer.
Advanced Driver Assistance Systems (ADAS)
Serjesh Sharma, Supervisor for ADAS MLOps at Ford recently featured on the Astronomer Data Flowcast podcast. There he shared insights into the intricate data processes supporting ADAS and how his team leverages Airflow and Astronomer to:
- Process 1+ petabyte of data weekly
- Run 300+ parallel workflows, balancing CPU- and GPU-intensive tasks for AI model development across a hybrid public / private cloud platform.
- Reduce errors while accelerating AI model development for autonomous driving systems.
Unlocking the Power of AI at Ford: A Behind-the-Scenes Look at Mach1ML and Airflow
Ford’s journey with AI and ML extends beyond advanced driver systems into other areas critical for company growth. At this year’s Airflow Summit, Ford shared how its multi-million dollar Mach1ML MLOps platform, powered by Airflow and Astronomer, equips data scientists and engineers with the tools to efficiently create, deploy, and manage ML models at scale. In the remainder of this blog post, we’ll recap key highlights from each session and provide you further resources to learn more.
Ford’s ML Journey
Ford's journey to AI and ML was shaped by significant organizational and technical hurdles. As a century-old company, it faced resistance to change, duplicated efforts across teams, and a lack of established MLOps practices. On the technical side, Ford's legacy infrastructure lacked the integration and tooling required for ML workflows, while regulatory constraints and unoptimized code further complicated adoption.
To address these issues, Ford established its Artificial Intelligence Advancement Center, bringing together over 100 experts to centralize ML efforts and key use cases. They also built the Data Factory to streamline data management and developed specialized platforms like FordLLM for large language models, showcasing Ford's commitment to AI-driven innovation.
Introducing Mach1ML and the MLOps Development Kit (MDK)
Mach1ML is Ford's comprehensive AI and ML platform designed to simplify and scale development and deployment for its teams. Supporting this ecosystem is the MLOps Development Kit (MDK), which standardizes end-to-end ML pipelines and ensures compatibility across hybrid environments like Google Cloud and HPC clusters. By focusing on tools and infrastructure, Mach1ML and MDK allow engineers to spend more time on model development rather than productionizing workflows.
Challenges of Kubeflow
The initial iteration of Mach1ML relied on Kubeflow for orchestration but encountered several issues. Ford's teams struggled with its steep learning curve and tight integration with Google Cloud, which reduced flexibility. Additionally, integrating third-party tools introduced unnecessary overhead.
These limitations highlighted the need for a robust, user-friendly orchestration tool that could handle hybrid setups and scale effectively.
Mach1ML 2.0: Powered by Airflow
Ford transitioned to Airflow for the second version of Mach1ML, addressing the shortcomings of its previous approach. Airflow's low barrier to entry, Python-based customizability, and prebuilt operators made it an ideal choice for a data scientist-heavy user base. With Astronomer powering production MLOps deployments, Ford streamlined its workflows and achieved seamless integration across on-premises, cloud, and hybrid environments.
The platform's architecture includes Airflow and Astronomer for orchestration, GitHub for version control, and tools like Vertex AI and Tekton for model deployment. Airflow also enabled Ford to adopt plug-and-play workflows, ensuring efficient feature engineering, training, and deployment.
Figure 1: Ford’s Mach1ML platform powered by Airflow, with Astronomer used for pipelines orchestrating feature engineering, model training, model storage, and model deployment . Image source.
To tailor Airflow to its needs, Ford developed custom operators for HPC integration, Vertex AI, and experiment tracking with Weights & Biases. A custom DAG decorator enhanced metrics tracking, error handling, and debugging, while reusable templates simplified common Gen AI and computer vision workflows. These innovations empowered Ford's teams to build reliable ML pipelines with minimal overhead.
Key Benefits: Driving Intelligent Vehicles and Customer Innovation
By integrating Airflow and Astronomer, Mach1ML has delivered faster time-to-value for ML solutions, reduced operational overhead, and accelerated innovation. As one of the speakers from the Ford team noted:
"Mach1ML, powered by Apache Airflow, is not just a tool—it’s a catalyst driving Ford’s future toward intelligent vehicles and cutting-edge customer experiences."
Dive into the details by watching the replay of the session Unlocking the Power of AI at Ford: A Behind-the-Scenes Look at Mach1ML and Airflow.
Wrap up and next steps
The move towards autonomous driving and smart mobility is already reshaping transportation for billions of people around the world. Ford is at the forefront of these innovations, driven by analytics and AI, and powered by data workflows orchestrated by Airflow and Astro.
The easiest way to scale your data pipelines and workflows is to run them on the Astro managed Airflow service. You can try Astro our for free.