Airflow in Action: How Panasonic Energy is Accelerating the EV Transition With Data Engineering

Panasonic Energy sets out each day to change the world by accelerating the transition to sustainable energy through the production of safe, high-quality lithium-ion batteries.

The company’s journey began in 2017 when Panasonic Energy of North America (PENA) began producing batteries for Tesla at the Gigafactory in Sparks, Nevada. Its operations have since grown to be one of the largest lithium-ion battery factories in the world with a growing workforce of more than 4,000 employees that has surpassed six billion cells delivered and counting. And this is just the beginning as PENA expands operations to meet growing Electric Vehicle (EV) demand across North America. Data and analytics are central to this expansion.

At this year’s Airflow Summit, Michael Atondo, Data Engineering Lead at PENA, shared the company’s transformative journey with Airflow. From manual, disconnected workflows to a robust orchestration system, the session walked through Panasonic’s challenges, innovations, and plans for the future.

In this blog post, we’ll recap Michael’s session and then provide resources to learn more.

In the Beginning: A Mess of Manual Processes

Before Apache Airflow®, Panasonic's data workflows were a patchwork of local scripts and unreliable processes. Windows Task Scheduler was paired with standalone Python scripts running on a physical PC—literally plugged into the factory floor. If someone unplugged the PC, workflows ground to a halt.

CSVs were manually parsed from file shares, emails were sent via Python, and data management lacked any centralization. Unsurprisingly the lack of standards and automation hindered the efficiency, reliability, and scalability of data workflows.

Iteration 1: Laying the Foundation with Airflow

Panasonic’s first foray into Apache Airflow brought much-needed automation and structure. This iteration introduced basic DAGs (Directed Acyclic Graphs) to handle recurring tasks and replaced ad-hoc scripts with containerized dashboards. Despite these advances, the setup was rudimentary. DAGs were sprawling, monolithic codebases that were difficult to debug and maintain.

Nevertheless, this foundational step marked a shift toward modern orchestration. It enabled the creation of live dashboards for operational analytics providing stakeholders with visibility into factory operations.

Iteration 2: Eliminating the Wild West

Recognizing the limitations of its first iteration, Michael and team refined the company’s workflows in Iteration 2. The focus shifted to improving standardization, breaking down monolithic DAGs into manageable Airflow task groups, and implementing GitLab CI/CD pipelines for consistent deployment.

By introducing automated cleanup scripts for XComs and database maintenance, they streamlined operations and avoided costly storage overruns. This phase also introduced daily failure monitoring dashboards, with daily reports on DAG failures sent to the respective owner via MS Teams, providing visibility into pipeline issues and enabling proactive resolution.

Iteration 3: Maturing and Scaling

In its current state, the Panasonic data engineering team has matured Airflow usage with well-defined standards and identity for task groups.

Debugging is now much easier, and the team has embraced best practices for building modular, maintainable workflows. Task groups offer clarity, and monitoring tools ensure no issue goes unnoticed. This maturity prepares Panasonic for the next phase: scaling with Kubernetes and streaming data capabilities.

Note that since the Airflow Summit, Astronomer has released Astro Observe as an additional monitoring option for Apache Airflow. Astro Observe brings rich visibility and actionable intelligence to data pipelines with proactive alerting, and the ability to set SLAs on data quality or freshness that makes the process of detecting DAG failures easier.

Integrating Airflow with Cutting-Edge Technologies

Today, Airflow is deeply embedded in Panasonic’s data ecosystem, integrating with several key technologies:

• Redis: For caching and optimizing data retrieval.
• MySQL: As the backend for metadata and task state management.
• Tableau: For interactive dashboards with 15-minute latency caps.
• Amazon Redshift: Enabling scalable, high-performance data warehousing.
• Foundry: Ensuring consistency and reliability in data workflows.
• Plotly Dashboards: Providing customized, live data visualizations in factory operations.
• GitLab CI/CD Pipelines: Streamlining DAG development and deployment.
• MS Teams: for alerts and notifications

The Road Ahead: Streaming and Kubernetes

Panasonic is now exploring the future of Airflow with Kubernetes and streaming workflows. Kubernetes will allow Airflow to operate across both public and firewalled networks, facilitating seamless data streaming from Manufacturing Execution System (MES) and MQTT brokers. The addition of Kafka pipelineswill further enable real-time data processing, while integration with cloud platforms like AWS ensures scalability for growing data demands.

Next Steps

You can follow along with Panasonic’s journey in the session replay How Panasonic Leverages Airflow.

The best way to get started with operational analytics is to orchestrate your data pipelines on Astro, the leading Airflow managed service. You can get started for free here.