Skip to main content

Create Airflow variables in the Astro UI

This feature is available only on Astro Hosted.
public preview
This feature is in Public Preview.

You can create and manage Airflow variables for Deployments with the Astro Environment Manager in the Astro UI. The Environment Manager uses an Astro-managed secrets backend to store Airflow variable key-value pairs as Kubernetes Secrets.

Using the Environment Manager, you can quickly and securely create Airflow variables once and share them to multiple Deployments without having to set up your own secrets backend. You can also create a variable once and use it across multiple Airflow Deployments.

For example, you can configure a variable with the access token credentials for a sandbox or development environment. Then, you can later configure your variable to be applied to all Deployments in the workspace by default. This means that when you create new Deployments, they automatically have access to your development environment. Later, you can override the variable to use your production credentials in production Deployments by using a field override or edit.

Compared to creating a variable in the Airflow UI, when you create a variable in the Astro UI, you can:

  • Share the variable with multiple Deployments within the Workspace.
  • Override the variable value for individual Deployments.
  • Use variables in branch-based deploys and PR previews.

Workspace Owners and Operators can create and assign variables, while Workspace Authors can view configured variables and use them in Deployments.

How variables are stored​

When you create an Airflow variable in the Environment Manager, Astro stores Airflow variable details in an Astronomer-hosted secrets manager, and then applies variables to Deployments as Kubernetes Secrets. Specifically the following steps occur:

  • Astro stores the variable value in a secure secrets manager hosted by Astronomer.
  • When a variable is assigned to a Deployment, Astro uses Airflow's provided local filesystem secrets backend to mount your Airflow variables as Kubernetes Secrets.
  • When your DAGs use your variables, Airflow reads the details from the filesystem using the Airflow local filesystem secrets backend.

This process occurs every time you create or update a variable.

Prerequisites​

Create an Airflow Variable​

You can create Airflow variables both at the Deployment and Workspace level. When you create a variable at the Deployment level, the variable details are available only to that specific Deployment. When you create a variable at the Workspace level, you can apply the variable to several Deployments and override its value as needed for each Deployment.

To create a variable at the Workspace level:

  1. In the Astro UI, click Environment in the left menu to open the Airflow Variables page.
  2. Click + Airflow Variable to add a new variable.
  3. Choose whether you want your variable to be Automatically link to all deployments by setting the toggle to On. See Link variables to Deployments for more information.
  4. Enter your information in the required fields.
  5. Choose whether you want your variable to be Not Secret or Secret, when working with it in the future. This allows you to store API keys, tokens, or other secrets in an Airflow Variable without those credentials being available to future editors.
  6. Click Create Airflow Variable.
  7. If you haven't linked your variable to all deployments, make it accessible to individual Deployments. See Link variables to Deployments.

To create a variable at the Deployment level:

  1. In the Astro UI, select a Deployment, then click the Environment tab within the Deployment menu.
  2. Click + Airflow Variable to add a new variable.
  3. Enter your information in the required fields.
  4. Choose whether you want your variable to be Not Secret or Secret, when working with it in the future. This allows you to store API keys, tokens, or other secrets in an Airflow Variable without those credentials being available to future editors.
  5. Click Create Airflow Variable to make your new variable.

After you create a variable, you can reference its Airflow Variable Key from DAG code like you would with any Airflow variable created through the Airflow UI.

After you create a variable at the Workspace level, you can link it to multiple Deployments. Linking variables is useful for standardizing external resource usage across your entire team.

For the most flexibility, you can set default variables and override the variable details per-Deployment based on details like the Deployment's usage and environment type (production or development).

  1. In the Astro UI, click Environment in the left menu to open the Airflow Variables page.
  2. Click the variable you want to link to a Deployment.
  3. Click + Link Deployment.
  4. Choose a Deployment from the list that appears.
  5. (Optional) Click More options and then add any value overrides for this Deployment. For example, if your variable provides a development token by default, you can override its details to instead provide a production token.
  6. Click Link Airflow Variable.

Configure variable sharing for a Workspace​

You can configure Astro to link Workspace-level variables to all Deployments in the Workspace by default.

This is useful, for example, when you need to configure a variable for development environments that all Deployments in a Workspace should start with. Then, when you create new Deployments, they automatically have a default variable to your development resources.

When you're ready to connect your Deployments to production resources, you can either replace the variable or override the variable field values with your production resource information.

If you toggle the Automatically link to all deployments setting from Off to On, Astro respects any variable value overrides that you might have configured for existing linked Deployments.

  1. Click Environment in the main menu to open the Airflow Variables page.
  2. Click the variable that you want to add per-Deployment field overrides to.
  3. Toggle the Automatically link to all deployments setting to choose either:
    • Off: Only share the variable individually to Deployments.
    • On: Link to all current and future Deployments in this Workspace.
  4. (Optional) Change the default variable value.
  5. Click Update Airflow Variable to save.

Override variable values​

If you create a variable at the Workspace level and link it to a Deployment, you can later edit the variable within the Deployment to specify a value override. When you override the variable's value, you specify the value that you want to use for one Deployment, but not for others. This way, you can configure the variable a single time, but still have the flexibility to customize variables at the Deployment level.

For example, you might have created a variable that points to a dev token, and then add a value override to specify a staging or production token for your staging and production Deployments to use.

  1. Click Environment in the main menu to open the Airflow Variables page.
  2. Click the variable that you want to add per-Deployment field overrides to.
  3. Click Edit to open the variable configurations for a specific linked Deployment.
  4. Switch the Override value toggle from No Override to Override.
  5. Add the override value.
  6. Click Update Airflow Variable link.

Was this page helpful?