Set up AWS Systems Manager (SSM) Parameter Store
In this section, you'll learn how to use AWS Systems Manager (SSM) Parameter Store as a secrets backend on Astro.
If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.
Prerequisites
- A Deployment.
- The Astro CLI.
- An Astro project with version 5.1.0+ of
apache-airflow-providers-amazon. See Add Python and OS-level packages. - An IAM role with access to the Parameter Store that your Astro cluster can assume.
- (Remote Execution Only) Helm installed
- (Remote Execution Only) The
values.yamlfile from the Register Agents modal in your Deployments>Agents page.
Step 1: Create Airflow secrets directories in Parameter Store
Create directories for Airflow variables and connections in Parameter Store that you want to store as secrets.
Variables and connections should be stored in /airflow/variables and /airflow/connections, respectively. For example, if you're setting a secret variable with the key my_secret, it should be stored in the /airflow/connections/ directory. If you modify the directory paths, make sure you change the values for variables_prefix and connections_prefix in Step 2.
For instructions, see the AWS Systems Manager Console, the AWS CLI, or the Tools for Windows PowerShell documentation.
Step 2: Set up Parameter Store locally
- Astro
- Remote Execution
Add the following environment variables to your Astro project's .env file:
AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "role_arn": "<your-role-arn>", "region_name": "<your-region>"}
You can now run a DAG locally to check that your variables are accessible using Variable.get("<your-variable-key>").
In your Astro project, add the AWS Systems Manager (SSM) Parameter Store to your project by adding the following to your values.yaml file to set the secrets backend class to use the provider and configure your secrets backend kwargs:
secretBackend: "airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend"
commonEnv:
- name: AIRFLOW__SECRETS__BACKEND_KWARGS
value: '{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "role_arn": "<your-role-arn>", "region_name": "<your-region>"}'
You need to run the Remote Execution Agent with AWS credentials to fetch from your secrets manager.
For secure production environments, you can store sensitive Kwargs containing secret ID and app role ID in a secret:
commonEnv:
- name: AIRFLOW__SECRETS__BACKEND_KWARGS
valueFrom:
secretKeyRef:
name: airflow-secret-backend
key: '{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "role_arn": "<your-role-arn>", "region_name": "<your-region>"}'
Step 3: Deploy configuration
- Astro
- Remote Execution
-
Run the following commands to export your secrets backend configurations as environment variables to Astro.
$ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
$ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "role_arn": "<your-role-arn>", "region_name": "<your-region>"}' --secret -
(Optional) Remove the environment variables from your
.envfile or store your.envfile in a safe location to protect your credentials inAIRFLOW__SECRETS__BACKEND_KWARGS.
- Run the following command to update your Remote Execution Agent with your new configurations.
helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml