Skip to main content
Running AI agents for local engineering work is, in essence, a context engineering problem. The prompt, your local files, installed skills, available tools, and the output of every command the agent runs all become context, and the code the agent writes is only as good as the information you give it access to. This Learn section covers the basics of using AI to write Apache Airflow® pipelines:
  • Foundational concepts and terms: The core terms in agentic data engineering, plus what to know about safety and cost of using AI agents.
  • Run Airflow locally: How to use the Astro CLI to spin up a local Airflow project so your agent can run Dags to test them in a realistic environment.
  • IDE setup for data engineering: How to attach your code editor to the same container your Dags run in, so autocomplete, type checking, and debugging match the Airflow and provider versions in your Airflow project.
  • AI context for data engineering: How to give your AI agents access to advanced knowledge about Airflow to write better Dags.
  • Develop Dags with AI: How to write instructions for AI agents that allow them to iteratively develop Airflow pipelines.
  • Debug Dags with AI: How to systematically debug failed tasks, Dags, and other Airflow issues with AI.
Otto, Astronomer’s data engineering agent, has advanced capabilities for local data engineering with a focus on writing Dags, debugging, and upgrading Airflow.To work with Otto locally, install the Astro CLI, sign in to your Astro account with astro login (a free trial is available), and then run astro otto.
For a hands-on demo of using AI agents in local data engineering, watch the recording of the Local data engineering in the agentic era webinar.

​
The mental shift from typing code to orchestrating agents

The way people get machines to do what they want has moved through increasingly abstract layers: machine code, assembly, high-level languages, and now natural language prompts. The move from typing code to directing an AI is a bigger jump than the ones before it. Every earlier layer still had you writing deterministic instructions. An AI model generates those deterministic instructions through a non-deterministic process. Developers adopting AI for code often follow a similar progression: first copying snippets out of a chat based AI application, then working with an assistant inline while coding, then directing a single agent, then coordinating several at once. Each step moves the work away from writing code by hand and toward two other jobs: engineering the agent’s context and writing a spec for the outcome you want. Deciding what to build in the first place, and whether it’s worth building at all, is still your job.

​
Basic concepts in agentic engineering

There are several core concepts you should understand when using AI in your local engineering processes:
  • Model: The LLM (large language model) itself. Your model runs either in the cloud (for example, Claude Opus 4.8) or on your computer (for example, a Llama variant). The model takes in input (context) and produces output.
  • Harness: The code around the model that decides what to send to it as input and how to handle different outputs. The harness determines which files are read automatically, which tools an AI has access to, and which AI outputs require human approval. The model running in the harness, plus all available tools, is your AI agent.
  • Interface: How you interact with your AI agent. Most developers use a terminal interface, an integrated development environment (IDE), or a combination of both.
  • Context window: Everything the model receives as input for a single call, including tool descriptions, conversation history, and any files or command output the agent pulled in. It has a fixed maximum size.
Information (context) given to your AI agent comes in several forms:
  • Auto-loaded context: Most agent harnesses automatically load some files into the context window, for example AGENTS.md files.
  • Prompt: What you directly type and paste into your AI agent interface, plus what the AI harness decides to send with your input, for example the conversation history.
  • Skills: Step-by-step instructions or workflows the agent can load on explicit demand or automatically based on keywords in the prompt. Skills are Markdown files with structured frontmatter. Astronomer maintains the astronomer/agents skills, a set of skills covering Airflow and data engineering best practices.
  • Available MCP servers: Structured interfaces to other systems. An MCP server exposes a set of tools the agent can call, for example your data warehouse’s MCP. The astro-airflow-mcp server exists for Airflow Deployments running on Astro. Tool descriptions from MCP servers are part of context the moment the server connects, so connected MCP servers cost tokens even when you don’t use their tools.
  • Available tools: Any script can be a tool. Both the tool description and the tool output can be part of AI context. Tools are what let an agent act on your computer instead of only producing text, which also makes them where most of the risk sits. See Safety.

​
Safety

A local agent running on your computer can be a safety issue if left running without supervision.
  • Tools can be destructive. As soon as an agent has access to a tool and no approval step, it can use that tool. An agent with unrestricted Bash access can wipe your entire hard drive while it thinks it’s deleting files inside a container. Restrict which commands your agent can run without asking. See Restrict agent commands for how to set those restrictions.
  • Incoming context can be wrong, or even deliberately malicious. Anything your agent reads can act as an instruction rather than as data. A web page can tell your agent to ignore what you asked and hand over your credentials, and the agent has no reliable way to separate that from a legitimate request. Treat anything retrieved from the web or another external source as untrusted.
  • Anything in the context window can leave it. Credentials, connection strings, and query results all become part of context the moment the agent reads them, and an agent that can reach the internet can send them elsewhere, be that by accident or because of a malicious prompt injection on a website.
  • Your Agent might spill your secrets.. For that reason you shouldn’t give your agents plain credentials and rather sign into to a CLI yourself and let the agent run commands through the session you authenticated. Where the system supports it, give your agent its own account with restricted permissions. Otto works this way: you run astro login yourself, and Otto uses that authenticated session to interact with Astro Deployments.
To guard against these issues you have three options:
  • Restrict permissions: Allow only specific CLI sub-commands, and deny unrestricted web access entirely.
  • Require human review for potentially destructive commands and actions.
  • Run in a sandboxed environment: Running agents in an environment separate from your local machine allows you to be more permissive, for example allowing file deletion, as long as deletion is only possible within the sandbox.
What you can’t do is enforce a rule by writing it into a prompt. Models are non-deterministic and might not follow your instructions, and compaction can drop the instruction from context partway through a long session.
For more on what can go wrong with AI agents, see the ContextOops chapter in the AI Context Engineering with Apache Airflow® eBook.

​
Cost

Agentic loops can read and write a lot of tokens, which can get expensive over time. A few strategies to lower cost are:
  • Choose the right model for the task. Use a more capable model for planning and for problems that need reasoning across a whole project. Smaller, cheaper models handle well-scoped individual tasks.
  • You pay for what you send, as well as for what the model writes. Skills, rules files, MCP tool descriptions, and pasted documentation all form part of the input on every turn. Keep your skills and instructions short and specific.
  • Avoid paying to generate the same code twice. If your agent keeps writing the same script, move that script into your project so the agent can call it instead of regenerating it each time.
  • Use a Dag-as-a-tool. Many skills are step-based workflows with some configuration. If you often need the same workflow, move it into an Airflow Dag and have your agent trigger that Dag and use its output. Running a Dag-as-a-tool is more reproducible, more robust, and cheaper, because the pipeline keeps the context needed for each step small, and deterministic steps can be accomplished using deterministic code. See Skills vs. Pipeline: Two Ways to Build the Same AI Workflow by Vikram Koka for more information on skills as Airflow Dags.