Datastax Doubles Down on Astro While Scaling Data Operations

Shortening the Path to Production for Generative AI

Datastax is at the forefront of the Artificial Intelligence revolution (and also in the Apache family). Their flagship product, Astra DB (not to be confused with Astro), is built on Cassandra and provides a highly versatile NoSQL and vector database, purpose-built for generative AI development. Datastax’s customers use Astra DB to scale petabytes of data for the delivery of enterprise-ready AI applications.

As a seasoned data engineering and platform leader, Ankush Gautam has transformed Datastax’s approach to data orchestration and scalability. Under his leadership, the team has adopted best-in-class practices that streamline operations and enhance productivity. His deep expertise in optimizing complex systems has been pivotal in enabling Datastax to stay ahead in the competitive world of generative AI and enterprise data solutions.

With Day-0 support for Airflow upgrades we’re now getting into a regular cadence of proactively upgrading our Airflow versions – which is typically accomplished within a couple of hours.

Ankush Gautam
Data Engineering and Platform Leader
Datastax

A Winding Road to Astro

Data orchestration is essential to Datastax’s product development and go-to-market motion; with pipelines powering a range of use cases from kicking off dbt workflows, triggering data science models, and transporting data from CRM tooling into downstream marketing applications.

When Gautam joined Datastax he recognized that various teams were leveraging disparate approaches to orchestration on Airflow – that spanned a historical mix of deployment options (on-prem and cloud) as well as managed services – including Astro Hybrid and Google Cloud Composer (GCC). Early on, he ascertained a degree of friction when it came to general maintenance and upgrades on GCC.

"Having worked with Apache Airflow extensively, I’ve seen firsthand how critical a robust orchestration framework is to an organization’s success,” said Gautam. “Our decision to move to Astro wasn’t just about solving immediate challenges—it was about future-proofing our operations and ensuring that we could scale effortlessly while maintaining operational excellence."

Faced with growing complexities in data workflows, including DAGs with over 150 tasks and intricate interdependencies, Ankush led a strategic overhaul of Datastax's orchestration framework. His forward-thinking approach ensured that the team not only resolved immediate pain points but also established a robust foundation capable of supporting exponential growth in data operations. Likewise, the team felt that GCC left a lot to be desired when it came to tracking and visibility, as the platform did not provide insight into critical metrics like CPU usage.

With Airflow usage only continuing to expand within their data operations, the Datstax team decided to double down on its use of Astronomer – eliminating the use of GCC across certain deployments, and migrating from an Astro Hybrid deployment to the fully hosted Astro.

[T]he migration to Astro resulted in a 40% reduction in maintenance overhead, enabling the team to redirect resources toward innovation.

Ankush Gautam
Data Engineering and Platform Leader
Datastax

Maintaining a Focus on Growth

Feeling liberated from routine Airflow maintenance and operational tasks, Gautam indicates that his team is now able to focus on doing business critical work. “With Day-0 support for Airflow upgrades we’re now getting into a regular cadence of proactively upgrading our Airflow versions – which is typically accomplished within a couple of hours. Having Astro is the equivalent of having one or two engineers that are solely focused on maintenance and upgrades, that are now able to spend their time working exclusively on roadmap items.”

Ankush has cultivated a team culture that thrives on innovation and continuous learning. His ability to simplify complex systems has ensured that new engineers, even those with no prior Airflow experience, can seamlessly integrate into the team. “The Astro UI provides the visibility and approachability we need, enabling new members to contribute meaningfully within their first week.”

Under Ankush’s guidance, the migration to Astro resulted in a 40% reduction in maintenance overhead, enabling the team to redirect resources toward innovation. Additionally, proactive version upgrades that previously took weeks are now completed within hours, ensuring Datastax stays on the cutting edge of orchestration capabilities."