This feature is only available for Airflow 3.x Deployments.
AWS PrivateLink enables private connectivity between your Remote Execution Agents and the Astro orchestration plane without exposing traffic to the public internet. This guide explains how to set up a VPC Endpoint in your AWS environment to establish secure communication with Astro.
By default, Remote Execution Agents communicate with the Astro orchestration plane over the public internet. With AWS PrivateLink, you can route this traffic through a private connection within AWS, which provides enhanced security and can simplify network configurations for organizations with strict security requirements.
The setup involves creating a VPC Endpoint in your AWS account that connects to Astronomer’s VPC Endpoint Service. Once configured, your Remote Execution Agents can communicate with Astro through this private connection.
Before you can create a VPC Endpoint, Astronomer must configure the VPC Endpoint Service on the Astro side. Contact Astronomer Support with the following information:
Astronomer Support will provide you with the VPC Endpoint Service name, Service region, and supported Availability Zones required to create your VPC Endpoint.
If your Remote Execution Agents run in a different AWS region than the Astro orchestration plane, inform Astronomer Support. Additional configuration may be required on the Astro side, such as adding your region to the VPC Endpoint Service cross-region configuration or adding your AWS account to the allowed principals list.
After receiving the VPC Endpoint Service name from Astronomer Support, create a VPC Endpoint in your AWS account.
Set the following values:
astro-privatelink.Subnet selection shows your subnets in the Availability zones supported by the VPCe Service. If there is a mismatch, you must create subnet(s) in the zones provided by Astronomer Support.
After creating the VPC Endpoint, configure DNS so that your Remote Execution Agents resolve the Astro orchestration plane hostname to the private endpoint IP addresses.
After completing the configuration, verify that your Remote Execution Agents can communicate with Astro through the private endpoint. Validate in the Astro UI that the agents are heart beating and reporting a Healthy status. You can also verify from within your network using the below instructions.
The response should show the private IP addresses assigned to your VPC Endpoint rather than public IP addresses.
The expected response is 404 page not found. If the connection is successful, your Remote Execution Agents will use the private endpoint for all communication with the Astro orchestration plane.
If you have multiple Remote Execution Agents across different VPCs, you can either create a VPC Endpoint in each VPC, or use a single VPC Endpoint and configure network routing between VPCs.
The following table summarizes the actions required based on your configuration:
If you previously created a Route53 private hosted zone, you can associate additional VPCs with the same hosted zone rather than creating new zones for each VPC.
After verifying that the private endpoint works correctly, you can optionally configure your Remote Execution Agents to only allow traffic through the VPC Endpoint. This ensures that all communication with Astro uses the private connection.
To restrict traffic:
This configuration ensures that only traffic coming through the VPC Endpoint Service can reach the Deployment.
The VPC Endpoint Service may require manual acceptance of endpoint connections, if still in pending state after 5 minutes. Contact Astronomer Support to approve your endpoint connection request.
Verify that your Route53 private hosted zone is correctly configured and associated with the VPC where you are testing.
Check that the security group attached to the VPC Endpoint allows inbound traffic on port 443 from the subnets where your Remote Execution Agents are running.