Prerequisites
- An Astro Deployment running the Astro Executor on:
- Hosted execution mode with Astro Runtime 3.1+, or
- Remote Execution mode on Astro Runtime 3.1+, using a Remote Execution Agent 1.5.0+ and Helm chart 2.0.0+. See Helm chart versioning for the Agent-to-chart compatibility matrix.
What you’ll see
A Resource Metrics tab appears in the Airflow UI with three views:- Dag: Aggregated CPU and memory usage across all tasks in a Dag.
- Task: CPU and memory usage for a specific task across its recent runs.
- Task Instance: CPU and memory usage for a single task run, mapped to the worker that ran it.
- Which tasks are consuming the most CPU or memory?
- Which tasks are likely causing out-of-memory (OOM) errors?
- Which Dags are driving infrastructure cost?
- Which tasks are good candidates for right-sizing or optimization?
How Astro collects task-level metrics
Astro samples task-level metrics on their own interval, separate from the interval used for Deployment-wide worker metrics. For long-running tasks, samples can lag up to five seconds behind the task. This sampling shapes what each metric represents:- CPU values account for all work the task performs, but averaging across the sampling interval smooths very short-lived peaks.
- Memory values reflect usage at each sample, so a spike shorter than the sampling interval on an otherwise idle task doesn’t appear.