Set up Azure Key Vault as your secrets backend

This topic provides setup steps for configuring Azure Key Vault as a secrets backend on Astro. If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.

Prerequisites

A Deployment.
The Astro CLI.
An Astro project.
An existing Azure Key Vault linked to a resource group.
Your Key Vault URL. To find this, go to your Key Vault overview page > Vault URI.
(Remote Execution Only) Helm installed
(Remote Execution Only) The values.yaml file from the Register Agents modal in your Deployments>Agents page.

If you don’t already have Key Vault configured, read Microsoft Azure documentation.

Step 1: Register Astro as an app on Azure

Steps 1 and 2 are only required if you are using service principal (client secret) authentication. If you prefer to use managed identity authentication, skip to Step 3 and follow the Managed Identity tab instructions.

Follow the Microsoft Azure documentation to register a new application for Astro. At a minimum, you need to add a secret that Astro can use to authenticate to Key Vault. Note the value of the application’s client ID and secret for Step 3.

Step 2: Create an access policy

If you use a managed identity to authenticate to Key Vault, skip to Step 3. Ensure your managed identity has an access policy or Azure RBAC role that grants it access to your Key Vault secrets.

Follow the Microsoft documentation to create a new access policy for the application that you just registered. The settings you need to configure for your policy are:

Configure from template: Select Key, Secret, & Certificate Management.
Select principal: Select the name of the application that you registered in Step 1.

Step 3: Set up Key Vault locally

Astro
Remote Execution

In your Astro project, add the following line to your requirements.txt file:

requirements.txt

apache-airflow-providers-microsoft-azure

Add the following environment variables to your .env file. Choose the option that matches your authentication method:Client secret authentication:

AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "tenant_id": "<your-tenant-id>", "client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}

For client secret authentication, find your client ID in Azure Portal at App Registration page > Application (Client) ID. To find your tenant ID, go to App Registration page > Directory (tenant) ID. To find your client secret, go to App Registration Page > Certificates and Secrets > Client Secrets > Value.Managed identity authentication:Before using managed identity authentication, you must configure your Deployment with a workload identity. See the Azure tab in Authorize a Deployment to cloud resources using workload identity to set up your managed identity and authorize it to your Deployment.

AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "managed_identity_client_id": "<your-managed-identity-client-id>", "workload_identity_tenant_id": "<your-tenant-id>"}

Add the Azure Key Vault Backend to your project by updating your values.yaml file. Choose an authentication method:

Service Principal
Managed Identity

Add the following to your values.yaml file to set the secrets backend class to use the Vault provider and configure your secrets backend kwargs:

values.yaml

secretBackend: "airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend"

commonEnv:
  - name: AIRFLOW__SECRETS__BACKEND_KWARGS
    value: '{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "tenant_id": "<your-tenant-id>", "client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}'

You need to run the Remote Execution Agent with your Azure credentials to fetch from your secrets manager.

Managed identity authentication is more secure because it eliminates the need to manage client secrets.

Prerequisites

Before updating your Helm values, configure a managed identity with access to your Key Vault:

Create a user-assigned managed identity in Azure, or use an existing one.
Grant the managed identity access to your Key Vault. In your Key Vault, create an access policy with Get and List permissions for secrets, and select your managed identity as the principal (see Step 2).
Configure Azure AD Workload Identity to allow your Kubernetes service accounts to use the managed identity.

Update Helm values

Add the following to your values.yaml file:

values.yaml

secretBackend: "airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend"

commonEnv:
  - name: AIRFLOW__SECRETS__BACKEND_KWARGS
    value: '{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "managed_identity_client_id": "<your-managed-identity-client-id>", "workload_identity_tenant_id": "<your-tenant-id>"}'

labels:
  azure.workload.identity/use: "true"

annotations:
  azure.workload.identity/client-id: "<your-managed-identity-client-id>"

Replace:

<your-vault-url>: Your Azure Key Vault URL
<your-managed-identity-client-id>: Client ID of your managed identity
<your-tenant-id>: Your Azure tenant ID

For managed identity authentication, find your managed identity client ID in Azure Portal at Managed Identities > your identity > Client ID. To find your tenant ID, go to Microsoft Entra ID > Overview > Tenant ID. This configuration tells Airflow to look for variable information at the airflow/variables/* path in Azure Key Vault and connection information at the airflow/connections/* path. You can now run a dag locally to check that your variables are accessible using Variable.get("<your-variable-key>"). By default, this setup requires that you prefix any secret names in Key Vault with airflow-connections or airflow-variables. If you don’t want to use prefixes in your Key Vault secret names, set the values for sep, "connections_prefix", and "variables_prefix" to "" within AIRFLOW__SECRETS__BACKEND_KWARGS.

Step 4: Deploy configuration

Astro
Remote Execution

Run the following commands to export your environment variables to Astro.
astro deployment variable create --deployment-id <your-deployment-id> --load --env .env
In the Astro UI, mark AIRFLOW__SECRETS__BACKEND_KWARGS as Secret. See Set environment variables in the Astro UI.
Run the following command to push your updated requirements.txt file to Astro:
astro deploy --deployment-id <your-deployment-id>
(Optional) Remove the environment variables from your .env file, or store your .env file so that your credentials are hidden, for example with GitHub secrets.

Run the following command to update your Remote Execution Agent with your new configurations.

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Overview

Get started

Develop DAGs

AI features

Deploy code

Manage Deployments

Automation & CI/CD

Platform observability

Administration

Release notes

Best practices

Reference

Airflow 3

Set up Azure Key Vault as your secrets backend

Prerequisites

Step 1: Register Astro as an app on Azure

Step 2: Create an access policy

Step 3: Set up Key Vault locally

Prerequisites

Update Helm values

Step 4: Deploy configuration

​Prerequisites

​Step 1: Register Astro as an app on Azure

​Step 2: Create an access policy

​Step 3: Set up Key Vault locally

​Prerequisites

​Update Helm values

​Step 4: Deploy configuration

Prerequisites

Step 1: Register Astro as an app on Azure

Step 2: Create an access policy

Step 3: Set up Key Vault locally

Prerequisites

Update Helm values

Step 4: Deploy configuration