Skip to main content

Get started with Airflow using the Astro CLI

With the Astro CLI, you can run Airflow on your local machine. Follow this quickstart to build an Airflow project from the Learning Airflow template and run it in a local Airflow environment with just a few commands. At the end of the tutorial, you'll have all of the files and components you need to develop and test Airflow DAGs locally.

Step 1: Install the CLI​

brew install astro

Step 2: Create an Astro project​

An Astro project contains the set of files necessary to run Airflow, including dedicated folders for your DAG files, plugins, and dependencies. This set of files builds an image that you can run both on your local machine with Airflow and deploy to Astro.

Use astro dev init with the --from-template flag to create the project based off of the Learning Airflow template.

astro dev init --from-template learning-airflow

This command generates all of the project files you need to run Airflow locally, including an example DAG that you can run out of the box. Different templates generate different example DAGs. See Create an Astro project for more information about the default project structure.

Step 3: Run Airflow locally​

Running your project locally allows you to test your DAGs before you deploy them to a production environment. While this step is not required for deploying and running your code on Astro, Astronomer recommends always using the Astro CLI to test locally before deploying.

  1. To start running your project in a local Airflow environment, run the following command from the learning-airflow project directory:

    astro dev start

    This command builds your project and spins up 4 Docker containers on your machine, each for a different Airflow component:

    • Postgres: Airflow's metadata database
    • Webserver: The Airflow component responsible for rendering the Airflow UI
    • Scheduler: The Airflow component responsible for monitoring and triggering tasks
    • Triggerer: The Airflow component responsible for running Triggers and signaling tasks to resume when their conditions have been met. The triggerer is used exclusively for tasks that are run with deferrable operators

  2. After your project builds successfully, open the Airflow UI in your web browser at https://localhost:8080/.

  3. Find your DAGs in thedags directory in the Airflow UI.

    In this directory, you can find an example DAG, example-astronauts, which was generated with your Astro project. To provide a basic demonstration of an ETL pipeline, this DAG shows a simple ETL pipeline example that queries the list of astronauts currently in space from the Open Notify API and prints a statement for each astronaut. The DAG uses the TaskFlow API to define tasks in Python, and dynamic task mapping to dynamically print a statement for each astronaut.

    Example DAG in the Airflow UI

info

The Astro CLI uses port 8080 for the Airflow webserver and port 5432 for the Airflow metadata database by default. If these ports are already in use on your local computer, an error message might appear. To resolve this error message, see Run Airflow locally.

Step 3: Develop locally with the CLI​

Now that you have a locally running project, you can start to develop your Astro project by adding DAGs, dependencies, environment variables, and more. See Develop your project for more details on how to modify all aspects of your Astro project.

Most changes you make, including updates to your DAG code, are applied automatically to your running environment and don't require rebuilding your project. However, you must rebuild your project and restart your environment to apply changes from any of the following files in your Astro project:

  • packages.txt
  • Dockerfile
  • requirements.txt
  • airflow_settings.yaml

To restart your local Airflow environment, run:

astro dev restart

This command rebuilds your image and restarts the Docker containers running on your local machine with the new image. Alternatively, you can run astro dev stop to stop your Docker containers without restarting your environment, then run astro dev start when you want to restart.

Next Steps​

After you have finished Getting Started with the CLI, you can configure your CLI to locally debug your Airflow environment, authenticate to cloud services to test your DAGs with data stored on the cloud, or you can learn more about developing DAGs with Astro.

Was this page helpful?