Astronomer Begins Platform Expansion Beyond Airflow With dbt Support In Latest Astro Release
The leading platform for fully-managed Airflow will continue to add support for additional popular open-source technologies, providing a unified platform for data-driven business.
The latest Astro release also adds capabilities to improve pipeline resilience and automate infrastructure provisioning and management.
New York, New York, July 31, 2024 — Astronomer, the company behind Astro, the leading fully-managed service for Apache Airflow®, today announced its latest platform release. The latest additions to Astro include new capabilities that will allow enterprises to improve pipeline resilience and increase release velocity while also hardening security. Most notably, Astronomer is adding support for dbt on Astro, which marks the platform’s first expansion beyond Airflow.
“For years, Astro has given our customers the best way to run Airflow and many have told us they want us to bring unified orchestration and observability to other technologies that are critical to their data stack,” said Andy Byron, Chief Executive Officer, Astronomer. “If it adds value and lowers the cost of ownership for our customers, Astronomer will continue to build out a unified platform to support whatever enterprises are building, from analytics to AI.”
Unveiling a Unified Approach to Modern Orchestration
Airflow is the industry’s de-facto standard for expressing pipelines and orchestrating data flows as code. Widely recognized as the industry’s leading and most advanced workflow management and orchestration solution, Airflow provides over 1600 data integrations with most of the popular databases, applications, AI frameworks, and tools, as well as hundreds of cloud services — with more added each month.
Since its launch in 2023, Astro has been the best way to run Airflow, with enterprise-grade features that allows teams to focus on data instead of pipelines, consolidate their data stack, confidently upgrade and rollback versions of Airflow, and more.
The value customers have seen from Airflow orchestration and observability on Astro has led to a desire for Astronomer to extend support to other popular open-source software, building out a unified platform for every facet of the data-driven enterprise. Astronomer begins executing on that vision with this release, allowing customers to orchestrate and run dbt Core™ on Astro in a single pipeline with just a few lines of code.
Astronomer will continue to add support for additional open source software in the coming years.
dbt On Astro
dbt™ is an open source command line tool that enables data teams to transform data, and has quickly become a standard in the modern data stack. dbt allows teams to quickly and collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. dbt Core is downloaded more than 10M times per month.
In 2023, due to overwhelming demand from customers and the Airflow community, Astronomer released Cosmos, an open source package that allows users to integrate dbt projects into Airflow in a few lines of code. Just one year later, Cosmos is downloaded 1.3M times per month, making it the most popular way to orchestrate dbt with Airflow. Now, Astronomer customers have a fully-managed way to deploy and run dbt and Airflow together on Astro.
“Our open-source integration became incredibly popular very quickly, but it doesn’t solve all the challenges of running dbt and Airflow together,” said Julian LaNeve, Chief Technology Officer, Astronomer. “In a time where organizations are being asked to do the same or more with the same budgets, we’ve had customers come to us with concerns about the price of other approaches at the enterprise scale, and we’re seeing them move their dbt transformations onto Astro where it’s 10x cheaper while still solving the operational challenges.”
With dbt on Astro, customers can simplify data workflow management by managing dbt and Airflow from a single interface, reducing the complexity and fragmentation associated with handling multiple data tools. This unification reduces context switching and fosters a more streamlined user experience to enhance focus on data analysis and accelerate decision-making.
Improved Pipeline Resilience
The latest version of Astro includes features that will greatly improve pipeline resilience:
- Universal Metrics Export allows comprehensive exporting of metrics to the popular monitoring tool Prometheus, providing enhanced visibility across their entire data estate. This is essential for organizations monitoring complex deployments and seeking real-time data to proactively manage system health.
- Self-Healing Workers automatically find and stop idle processes, which can occur due to common Airflow infrastructure issues. This prevents cascading failures, reduces stale data risks and ensures system uptime, empowering data engineering teams to focus on strategic work
Simple and Secure Deployments and Infrastructure Management
The latest version of Astro includes features that help automate infrastructure provisioning and deployment management, allowing teams to deliver faster releases while improving security:
- Astro Terraform Provider simplifies the management of Airflow infrastructure, offering automation capabilities that ensure consistent, reproducible, and scalable deployments for organizations standardizing their infrastructure management practices with Terraform.
- Customer Managed Workload Identity on AWS enhances governance by enabling reliable and secure access to data services, minimizing security vulnerabilities, and ensuring compliance with security requirements. While this feature was previously released for GCP and Azure, it is now available for AWS and will reduce onboarding friction for teams migrating between cloud providers.
To learn more, visit the resources below or get started with Astro today with a 14-day free trial.
Further Resources:
- Learn more at the Astronomer blog
- Astronomer CTO reflects on the history of Airflow and dbt
- Download Astronomer’s white paper on Full-stack Orchestration in the Age of the Data Product
- Try Astro now with a 14-day free trial
About Astronomer
Astronomer designed Astro, a modern data orchestration platform, powered by Apache Airflow®. Astro enables data teams to build, run, and grow their mission-critical data pipelines on a single platform for all of their data flows. Astro dramatically reduces costs, improves developer productivity, and reliably delivers your most data-centric use cases. For more information, visit www.astronomer.io.
Apache® and Apache Airflow® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by the use of these marks. All other trademarks are the property of their respective owners.
Media Contact:
Inkhouse for Astronomer
