dbt is an open-source library for analytics engineering that is frequently used in combination with Airflow. Using Airflow to orchestrate and execute dbt models as DAGs ensures fine-grained control over dbt tasks, observability over every step, and a reliable, scalable environment to run your models.
There are several ways to simplify the integration of Airflow and dbt. Cosmos is an open source package from Astronomer that is the easiest way to run your dbt Core projects as Airflow DAGs and Task Groups with just a few lines of code. Astronomer customers also have access to dbt on Astro, which allows for seamless deployment of dbt projects to production.
In this webinar, we discuss Cosmos and Astro features in depth, including:
- Native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow
- Utilization of Airflow’s data-aware scheduling to run models immediately after upstream ingestion
- Transformation of each dbt model into a task/task group complete with retries, alerting, etc.
- Deployment of dbt models to Astro for increased observability
