Set up Google Cloud Secret Manager as your secrets backend
This topic provides setup steps for configuring Google Cloud Secret Manager as a secrets backend on Astro.
If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.
Prerequisites
- A Deployment.
- The Astro CLI.
- An Astro project.
- Cloud SDK.
- A Google Cloud environment with Secret Manager configured.
- A service account with the Secret Manager Secret Accessor role on Google Cloud.
- (Optional) A JSON service account key for the service account. This is required to provide access to a secrets backend from a local machine, or when you're not using Workload Identity.
- (Remote Execution Only) Helm installed
- (Remote Execution Only) The
values.yaml
file from the Register Agents modal in your Deployments>Agents page.
Step 1: Create an Airflow variable or connection in Google Cloud Secret Manager
To start, create an Airflow variable or connection in Google Cloud Secret Manager that you want to store as a secret. You can use the Cloud Console or the gcloud CLI.
Secrets must be formatted such that:
- Airflow variables are set as
airflow-variables-<variable-key>
. - Airflow connections are set as
airflow-connections-<connection-id>
.
For example, to add an Airflow variable with a key my-secret-variable
, you run the following gcloud CLI command:
gcloud secrets create airflow-variables-<my-secret-variable> \
--replication-policy="automatic"
For more information on creating secrets in Google Cloud Secret Manager, read the Google Cloud documentation.
Step 2: Set up GCP Secret Manager locally
- Astro
- Remote Execution
-
Copy the complete JSON service account key for the service account that you want to use to access Secret Manager.
-
Add the following environment variables to your Astro project's
.env
file, replacing<your-service-account-key>
with the key you copied in Step 1:AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": "<your-service-account-key>"} -
(Optional) Run
Variable.get("<your-variable-key>")
to run a DAG locally and confirm that your variables are accessible.
In your Astro project, add the Google Cloud Secret Manager Backend to your project by adding the following to your values.yaml
file to set the secrets backend class to use the Vault provider and configure your secrets backend kwargs:
secretBackend: "airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend"
commonEnv:
- name: AIRFLOW__SECRETS__BACKEND_KWARGS
value: '{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": "<your-service-account-key>"}'
You need to run the Remote Execution Agent with GCP credentials to fetch from your secrets manager.
Step 3: (Astro Only) Configure Secret Manager on Astro using Workload Identity (Recommended)
-
Set up Workload Identity for your Airflow Deployment. See Connect Astro to GCP data sources.
-
Run the following commands to set the secrets backend for your Astro Deployment:
$ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
$ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "project_id": "<your-secret-manager-project-id>"} -
(Optional) Remove the environment variables from your
.env
file or store your.env
file in a safe location to protect your credentials inAIRFLOW__SECRETS__BACKEND_KWARGS
.
To ensure the security of secrets, the .env
variable is only available in your local environment and not in the Astro UI . See Set Environment Variables Locally.
Step 4: Configure Secret Manager on Astro using a service account JSON key file
- Astro
- Remote Execution
-
Set up the Secret Manager locally. See Set up GCP Secret Manager locally.
-
Run the following command to set the
SECRET_VAR_SERVICE_ACCOUNT
environment variable on your Astro Deployment:astro deployment variable create --deployment-id <your-deployment-id> SECRET_VAR_SERVICE_ACCOUNT="<your-service-account-key>" --secret
-
(Optional) Remove the environment variables from your
.env
file or store your.env
file in a safe location to protect your credentials inAIRFLOW__SECRETS__BACKEND_KWARGS
.
- Run the following command to update your Remote Execution Agent with your new configurations.
helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml