Skip to main content
Almost every data engineer uses AI to debug Airflow failures, from Dag import errors to failed tasks to Airflow infrastructure issues. Pasting a stack trace to an agent with “fix this” works some of the time, but there are better, systematic ways to let AI find the root cause of issues in your Airflow pipelines. This guide explains how to use AI agents to debug systematically in loops, or automatically as soon as a Dag running on Astro fails.

​
Assumed knowledge

To get the most out of this guide, you should have:
  • A local Airflow environment. See Run Airflow locally.
  • Basic familiarity with an AI coding agent harness, such as Claude Code or Cursor.

​
Otto

LabsOtto is in Labs.
Debugging is one of the tasks where Otto has the biggest advantage over a general-purpose agent. Otto investigates with access to the operational history in Astro: component logs, recent deploys, connections, lineage, and a compatibility knowledge base drawn from Astronomer’s experience running Airflow at scale. A general-purpose harness only sees what’s in your local repository and whatever you added to the context manually. You can ask Otto to investigate a failure interactively from the Astro CLI, trigger an investigation from the Astro UI, or automatically using an Astro alert and Otto’s API endpoint. See Investigate with Otto and How to use Otto to automatically investigate Dag failures and PR a fix for more information. An investigation reports a root cause and a suggested fix, among additional information such as estimated severity, a confidence score, and the evidence behind the diagnosis. The following example shows the same failure investigated from different locations: a copy_to_staging task that fails because Snowflake can’t cast the percentage string "88.1%" into a numeric column.
An Otto investigation in the Astro UI for a failed Dag run, labeled HIGH, P2, and PERMANENT. Otto reports 95 percent confidence that the task fails because Snowflake can't cast the percentage string 88.1% into a numeric column, and suggests emitting a numeric value instead. Dag-level checks rule out a Dag run timeout and a recent deploy as causes.

​
Debug systematically

Using AI doesn’t change what good debugging looks like. When you use an agent to debug a Dag, use the same systematic approach you would use when debugging manually. Astronomer recommends the following debugging ladder:
  1. Infrastructure: Is the scheduler running, is the database reachable, and are the worker queues sized adequately?
  2. Dag parsing: Does the Dag file import without errors? Check with astro dev parse.
  3. Dag and task run: Is the Dag schedule correct, and are the tasks running with the correct dependencies?
  4. Logic: Is the task code doing what it should?
  5. Prevention: After you fix the bug, what test or check would have caught it earlier?
To see an example of the debugging ladder in practice, watch the recording of the Best practices for debugging your Airflow Dags webinar.
Encode this ladder as a skill or rule file for your harness, so every debugging session follows the same steps without you repeating them in each prompt.

​
Give your agent context about the failure

For debugging, your agent needs access to the context surrounding a failure:
  • Task logs: The full output of the failed task run, if applicable, including the exception, stack trace, and surrounding log lines.
  • Run history and task state: Whether this is the first failure, whether upstream tasks actually succeeded, and what schedule and trigger rule the task uses.
  • Source history: What changed in the Dag file recently, for example using git log and git blame.
  • Lineage: What data sources upstream of the Dag might influence its functioning, and which downstream assets can be affected. When running Airflow on Astro, Otto can access information gathered as part of Astro Observe.
  • Additional variables: Sometimes Dags depend on values that aren’t defined in code. This could be environment variables, Airflow variables, Airflow connections, or configuration information fetched at runtime from a third-party service. If possible gather, the values for the failed Dag run and make them available to your agent.

​
Gather task logs

For a local project in container mode, Airflow writes task logs to $AIRFLOW_HOME/logs/ inside the scheduler container, so your agent needs a shell in the container to read them. You can use astro dev bash to open one. Use astro dev logs for component logs from the scheduler, triggerer, API server, and Dag processor. For a Dag that ran on an Astro Deployment, Astronomer recommends using the astro-airflow-mcp as a wrapper around the Airflow REST API. See Airflow MCP Plugin for more information.
Commands such as astro dev run tasks test <dag-id> <task-id>, and triggering a Dag run, execute real task code against whatever connections the task uses. Before you let an agent run tasks while debugging, make sure running tasks is safe, for example by pointing the project at a test environment.
For stepping through task logic line by line, see Debug with dag.test().

​
Debug in a loop

Debugging in a loop works the same way as writing Dags in a loop. Turn the bug into a regression test first: a test that reproduces the failure and fails until the fix is correct. Then let the agent iterate against that test the same way it would against any other success criteria. The same safety guardrails apply here: scope tool permissions, require approval for anything you can’t easily undo, and commit frequently so errors can be reverted.