Astro Observe overview

Astro Observe gives you visibility into the health and performance of your data pipelines and their assets. By default, you can view at-a-glance summaries about the health of your data pipelines in your Astro Organization.

You can also get high-detail visibility by creating data products, which are pipeline-driven assets that captures the data lifecycle of a pipeline. Assets monitored in Observe can include tasks, datasets, warehouse tables, and local files. Through Observe, you can define business-level SLAs on your data products, receive alerts when these SLAs are at risk, and get broader insights into the performance of your pipelines.

Health​

Before you create a Data Product or configure specific alerts or SLAs, you can use the Data Pipelines Health Overview page to see a high-level view of operational and business health across your pipelines and data products. The dashboard includes performance metrics, like failed task runs and task retries, alongside business-facing metrics, like SLA evaluations and warehousing costs, to help you identify potential issues at a glance and understand the overall health of your data ecosystem. This aggregate view of your pipelines enables you to quickly identify data resources that might be good candidates to include in a Data Product.

Additionally, The Health page shares usage data, including a section about your Snowflake credit use, split between views about DAGs and Assets. It also identifies the specific Airflow Operators that your Organization uses, and the number of DAGs that use them.

Refer to Enable Snowflake cost metrics on Observe to set up cost metrics reporting.

Data products​

In Observe, a data product is a pipeline-driven asset that captures the data lifecycle of a pipeline. Data products in Observe can include data assets from different DAGs and Airflow deployments to capture the full scope of a complex data pipeline. For example, a set of five DAGs that each populate an executive dashboard with product analytics might be grouped together into a single data product in Observe.

• Dependencies. A responsive graph displays the relationships between the final assets in a data product and the upstream assets that feed into it.
• SLA evaluations. You can create custom business-level Service Level Agreements for evaluating the on-time delivery of a data product or its freshness. You can also view all SLAs in effect for the data product along with their current statuses at a glance.
• Metrics. You can see key metrics for assets across your data product, like task failures and retries. Observe also supports custom metrics, like the accuracy of a machine learning model, relevant to your specific data product.

See Create a Data Product to refer to steps to create your own data products. Or, to follow a quickstart, see Get started with Astro Observe.

Asset Catalog​

Assets include information about the attribute details and historical data regarding task runs. In the Astro UI, when you click Observe and then Asset Catalog in the Organization menu, you can view and filter the different data assets in your Workspace.

In this view you can see the following information about your Assets, including:

• Name
• Asset type
• Namespace
• DAG ID
• Owner

You can click on any asset to view more detailed information about it, such as historical metrics and metadata. To learn more about using assets and accessing metrics, see Work with assets.