Customer Story: Autodesk’s Data Engineering Transformation with Astronomer and Apache Airflow

Overview

Autodesk, a global leader in design software for industries like construction, manufacturing, and animation, is known for its flagship product, AutoCAD. The company’s data strategy is crucial, as Autodesk's data-driven decisions influence its operational and strategic direction.

Bhavesh Jaisinghani, a Data Engineering Manager at Autodesk, leads a team focused on licensing data. He shares how Autodesk leveraged Astronomer and Apache Airflow to streamline and scale their data workflows.

To keep pace with rapid growth and the increasing complexity of Autodesk’s data workflows, Bhavesh’s team needed a more efficient, scalable, and user-friendly orchestration solution. The company turned to Astronomer, powered by Apache Airflow, to transform their data engineering processes.

Challenges

Legacy System Limitations

Before adopting Airflow, Autodesk relied on Apache Oozie as its primary orchestration tool. However, this legacy system presented several challenges:

• Complex User Interface: Oozie's interface was not intuitive, making it difficult for engineers to manage and monitor workflows effectively.
• High Maintenance Overhead: Significant time and resources were spent on maintaining the system rather than developing new features or improving data pipelines.
• Scalability Issues: Scaling up or down based on project demands was challenging, limiting Autodesk’s data engineering flexibility.
• Development Bottlenecks: The existing setup hindered quick development, testing, and deployment of new workflows.

Bhavesh and his team realized that continuing with Oozie would impede their ability to deliver crucial data insights promptly, affecting stakeholders across the organization. The Autodesk team sought a solution that would provide a more intuitive user interface, reduce maintenance time, and enable seamless scaling of workloads across their complex infrastructure.

Solution: Migration to Astronomer and Apache Airflow

Why Astronomer and Apache Airflow?

After evaluating various solutions for its migration off of Oozie (read more in a previous case study)—including both self-hosted and managed services—Autodesk chose Astronomer for its robust, managed offering of Apache Airflow, Astronomer. Their decision was driven by:

• User-Friendly Interface: Astronomer's intuitive UI made it easier for engineers to visualize, monitor, and manage workflows. Astronomer's centralized interface simplified user and performance management, log access, and workflow monitoring. Admins could view and control all aspects of the data engineering pipeline from one platform.
• Flexibility: Astronomer’s managed Airflow solution allowed Autodesk to scale quickly, reducing dependencies on in-house operations.
• Managed Service: Offloading the operational aspects of Airflow allowed the team to focus on development rather than infrastructure maintenance.
• Scalability: Astronomer provided the flexibility to scale resources up or down seamlessly, accommodating varying project needs. The team could shift from a multi-repository (polyrepo) to a single-repository (monorepo) structure, streamlining code management and collaboration in the development process.
• Independent Development Environments: Astro enabled the creation of dedicated UAT (User Acceptance Testing) environments within production, allowing Autodesk’s engineers to test and deploy features autonomously. The ability to create individual deployments within a shared workspace empowered engineers to develop and test independently without affecting production.

Astronomer's support and tooling facilitated a smooth migration from Oozie to Airflow, minimizing disruptions. Astronomer allowed Autodesk to upgrade to new Airflow versions efficiently with Seamless Deployment and Upgrades. With Astronomer’s CLI, developers could test locally before deployment, making the migration experience smooth. Astronomer allowed Autodesk to scale deployments easily, facilitating the onboarding of additional teams across departments.

Outcomes

Operational Efficiency

The migration has empowered the data engineering team to manage workflows efficiently. The reduction in server maintenance and downtime has allowed Autodesk to focus more on innovation, with Bhavesh noting a marked improvement in team productivity and performance.

Increased Scalability

Astronomer and Airflow enabled Autodesk to scale its data infrastructure significantly, going from 25 teams at the start of the project to supporting over 50 teams today. This scalability has allowed Autodesk to support diverse workflows across various departments, enabling more of the business to take a data-driven approach.

Minimal Workflow Disruptions

Since migrating from Oozie to Airflow with Astronomer, Autodesk has reduced workflow disruptions. The improved stability has been instrumental in providing accurate and timely data insights for C-level decision-makers, ensuring that critical business decisions are data-informed.

Business Impact

• Timely Data Delivery: Reliable workflows ensured that critical data reached stakeholders on time, supporting informed decision-making at the highest levels.
• Enhanced Visibility: Managers gained better insights into team performance and workflow health through Astronomer's centralized dashboard.
• Empowered Engineers: The ability to manage individual deployments and environments fostered a culture of ownership and innovation among team members.

Conclusion

Autodesk’s transition to Astronomer and Apache Airflow exemplifies the importance of a strategic, managed orchestration solution in achieving scalability and operational efficiency. By partnering with Astronomer and migrating away from Apache Oozie to Airflow, Autodesk has optimized its data engineering processes, enabling robust, scalable, and highly manageable data workflows — leading to faster innovation and more informed, data-driven decisions at all levels of the organization.

"We started with 25 teams and now have over 50 teams using Airflow. Astronomer has really helped us scale. The fact that we're not seeing the failures we experienced with Oozie is a testament to how much smoother things are running now."

— Bhavesh Jaisinghani, Data Engineering Manager at Autodesk

Ready to transform your data workflows like Autodesk did? Signup for a Free Trial of Astro today to learn how we can help you scale your data engineering efforts with ease.