Astro Observe

Astro Observe gives you visibility into the health and performance of your data pipelines and their assets. You can get this visibility by creating data products, which are sets of one or more assets and the upstream DAGs, tasks, and datasets that these assets depend on. Assets monitored in Observe can include tasks, datasets, warehouse tables, and local files, where you can configure notifications when asset performance or access meets certain criteria. Through Observe, you can define business-level SLAs on your data products, receive alerts when these SLAs are at risk, and see broader insights into the performance of your pipelines.

Data products​

In Observe, a data product is a set of one or more data assets, like DAGs, tasks, and datasets, that have importance to your business. For example, a set of five DAGs that each populate an executive dashboard with product analytics might be grouped together into a single data product in Observe. Data products in Observe can include data assets from different DAGs and Airflow deployments to capture the full scope of a complex data pipeline.

• Dependencies. A responsive graph displays the relationships between the final assets in a data product and the upstream assets that feed into it.
• SLA evaluations. You can create custom business-level Service Level Agreements for evaluating the on-time delivery of a data product or its freshness. You can also view all SLAs in effect for the data product along with their current statuses at a glance.
• Metrics. You can see key metrics for assets across your data product, like task failures and retries. Observe support custom metrics, like the accuracy of a machine learning model, relevant to your specific data product.

See Create a Data Product for detailed steps to create your own data products. Or, to learn more about business applications and use cases for data products, see Leveraging data products for health and performance benefits.

Asset Catalog​

In the Astro UI, when you click Observe and then Asset Catalog in the Organization menu, you can view and filter the different data assets in your Workspace. Assets include information about the attribute details and historical data regarding task runs.

In this view you can see the following information about your Assets, including the following:

• Name
• Asset type
• Namespace
• DAG ID
• Owner

You can click on any asset to view more detailed information about the asset, such as historical metrics and metadata.

To learn more about the data quality of your assets, and how this impacts your Airflow performance, see Data quality and Airflow.