Introducing a single pane of glass for your data products
The modern data stack and the workflows that it powers is becoming more complex, and data professionals are forced to grapple with a growing sprawl of dependencies, spanning across multiple teams and deployments. Likewise, as data products play an increasingly critical role in supporting business objectives, most organizations have lost any tolerance for failure or delay in pushing quality data for internal and external consumption.
Today, we’re excited to announce Astro Observe, which brings new and more robust data observability capabilities to users on Astro, OSS Airflow, Amazon Managed Workflows for Apache Airflow (MWAA), and Google Cloud Composer (GCC). Astro Observe is being offered to select design partners as part of a private preview. Users across these platforms can request access to the preview, and receive early access to a centralized dashboard for the management of data products, in addition to a dependency graph that provides a complete view of up-and-downstream dependencies within the data supply chain.
Moving from reactive to proactive interventions
Astro Observe’s Data Products Dashboard gives teams a central platform to assign ownership, manage, and monitor their data products. It creates a clear picture of accountability and enables teams to take action when necessary. Here, teams can set and track SLAs against data freshness and delivery time; and receive alerts when an asset within a data product is at risk of breaching its SLA.
Pictured: Data Product Overview in Dashboard
Pictured: SLA Evaluations view in Data Products Dashboard
With Dependency Graphs, teams have access to a complete view of lineage and ownership of dependencies across the entire supply chain for a data product; and are empowered with the necessary context to pinpoint the cause of issues and deliver remediation quickly – with the foresight to avoid outages or degradation of service. Users can also develop a historical context of dependencies with the version histories, which surfaces a view of lineage at an exact date and time, providing valuable insight for debugging, auditing, and managing changes.
Pictured: Dependency Graph in Data Products Dashboard
While Astro Observe is agnostic across open source Airflow and other managed services, Astro users enjoy the added benefit of having a single destination for data orchestration and observability. By centralizing all activity to a single interface, teams can understand activity and take action within Airflow without switching between platforms.
Next Up: Public Preview and General Availability
As we move towards GA, Astro Observe users can expect more capabilities, such as a recommendation engine for proactively surfacing actions that teams can take within their for pipelines to improve efficiency and mitigate risk. We’ll also be introducing functionality around anomaly detection and cost optimization, further allowing users to deliver proactive interventions to improve operations and avoid failures.
In the coming months, Astro Observe will become available to all customers as we move into Public Preview, with general availability planned for early 2025. If you’re interested in participating in the preview, you can request access now, and we’ll be in touch to discuss your specific deployments and configuration. We’re excited to invite Airflow users on this journey as we firmly believe that with strong data observability practices comes greater reliability and trust of data products – as well as faster innovation, lower costs, and more productive platform and data engineering teams as critical data assets are better secured and governed.
Additional Resources
