Orchestrate your dbt models with Astro, powered by Apache Airflow®
Provide a seamless and integrated experience for deploying and managing dbt on Astro, significantly reducing the complexity and fragmentation associated with handling multiple data tools.
Why are dbt and Astro better together?
dbt excels at transforming data using SQL, making it accessible to data analysts and engineers.
Astro, powered by Apache Airflow®, provides robust orchestration capabilities, scheduling, and observability.
dbt and Astro enable precise management of complex workflows and the consistent delivery of your data.
Introducing dbt on Astro
With dbt on Astro, you can unify your data workflows, reducing complexity and providing clear insights that improve team productivity.
Gain complete visibility into your dbt tasks within Airflow, making it easier to detect and troubleshoot issues. Astro’s robust deployment capabilities allow you to deploy dbt code independently from Airflow DAGs, simplifying your CI/CD processes and reducing deployment errors.
Checkout the two features that are a part of dbt on Astro below.
Enhance Visibility with Cosmos
Cosmos is a production-ready open-source solution developed by Astronomer, provides complete visibility of your dbt tasks within the Astro platform.
Key Benefits:
- Comprehensive Visibility: Gain complete visibility into your dbt projects within Airflow, allowing for easier monitoring and troubleshooting.
- Integrated Operations: Run dbt projects using Airflow connections instead of traditional dbt profiles, simplifying the integration.
- Leverage Airflow's Power: Utilize Airflow's native functionalities such as data-aware scheduling, connections, retries, alerts, datasets, observability, and lineage, ensuring your dbt workflows are robust and reliable.
- Advanced Capabilities on Astro: Maximize your data workflow efficiency with Astro's advanced features, including automated data lineage collection and seamless management of Python dependencies, to streamline your operations and enhance overall productivity.
Streamline Deployments with dbt deploys
An Astro-only dbt integration that provides a cohesive experience for deploying and managing dbt on Astro.
Key Benefits:
- Simplified Deployments: Easily deploy dbt models with the new astro dbt deploy command, without the need to restart workers, while enabling full deploy history visibility
- Enhanced Collaboration: View commit descriptions directly in the Astro UI, facilitating better communication and troubleshooting.
- Efficient CI/CD: Streamline your CI/CD processes by managing dbt and Airflow code from separate repositories, reducing complexity and context switching.
See why people love Cosmos
Having set up dbt-core on Airflow before, I had to create a lot of internal jobs to achieve what this project does out of the box. With Cosmos and the open source community, it's been great to see best practices in place and how forward-thinking the project is. dbt artifacts automatically spitting out open-lineage datasets to feed into data-aware scheduling? Nice! It's been great and I know it will get better.
Adam Underwood
Senior Data Engineer, Grindr
We conducted a PoC, running dbt with Mage.ai, dbt Cloud, GitHub Actions, and Astronomer Cosmos. In the end, our team voted for Cosmos.
Jensen Yap
Senior Data Engineer, NFTBank.ai
Astronomer Cosmos has allowed us to seamlessly orchestrate our dbt projects using Apache Airflow® for our start-up. The ability to render dbt models as individual tasks and run tests after a model has been materialized has been valuable for lineage tracking and verifying data quality. I was impressed with how quickly we could take our existing dbt projects and set up an Airflow DAG using Cosmos.
Justin Bandoro,
Senior Data Engineer, Kevala Inc.
Cosmos has sped up our adoption of Airflow for orchestrating our System1 Business Intelligence dbt Core projects without requiring deep knowledge of Airflow. We found the greatest time-saver in using the Cosmos DbtTaskGroup, which dynamically creates Airflow tasks while maintaining the dbt model lineage and dependencies that we already defined in our dbt projects.
Karen Connelly
Senior Manager, Business Intelligence, System1
With Cosmos, we could focus more on the analytical aspects and less on the operational overhead of coordinating tasks. The ability to achieve end-to-end automation and detailed monitoring within Airflow significantly improved our data pipeline’s reliability, reproducibility, and overall efficiency.
Péter Szécsi
Information Technology Architect, Data Engineer, Hungarian Post
