Skip to main content

Astro API best practices

The Astro API is Astronomer's REST API for managing resources on Astro, for example to create a Deployment. This guide provides the following best practices for general development, and specifically development with the Astro API, to ensure a safe and optimal experience:

  • Considerations for using the Astro API, Astro CLI, and Astro Terraform provider
  • Error handling
  • Authenticating scripts using API tokens
  • Using a graphical REST API client for development
  • Handling rate limiting with exponential backoff retries
  • Reusing HTTP connections

Feature overview​

This guide highlights the following Astro features:

Considerations for using the Astro API, Astro CLI, and Astro Terraform provider​

Astronomer provides several tools to manage Astro resources. Each tool offers benefits for specific use cases:

Astro API​

  • Has structured input and output, which allows for convenient automation
  • Allows you to use any programming language or framework that can make HTTP requests
  • Requires you to implement status checking mechanisms yourself, such as polling the API to check for Deployment creation completion

Astro CLI​

  • Provides a local development environment
  • Produces human-readable output such as text or table format, therefore less suitable for automation
  • Makes deploying Airflow code to Astro convenient

Astro Terraform provider​

  • Allows you to use Terraform, the industry standard for managing infrastructure as code
  • Requires Terraform development familiarity
  • Uses declarative language, which means you define your desired end state and, don't have to think about details, such as polling for infrastructure creation status

The tool that suits you best depends on factors such as your technical experience, existing knowledge in your organization, and use of Astronomer. It's common to use all available tools.

Error handling with the Astro API​

It's a best practice to ensure your code handles unexpected situations properly, such as errors. Because Astro's tooling generally distinguishes between client-side and server-side errors, you can automate a process for determining the cause of an error. All HTTP 4XX code indicate a client-side error and all HTTP 5XX status codes indicate a server-side error.

For example, Deployment names are unique within an Astronomer Workspace. If you create a second Deployment with the same name as an already existing Deployment, the Astro API returns an HTTP 400 error.

To ensure your code handles errors correctly, wrap requests in a try/except code block:

import requests

organization_id = "<your-organization-id>"
workspace_id = "<your-workspace-id>"
astro_api_token = "<your-astro-bearer-token>"

try:
    # Create a Deployment
    # https://www.astronomer.io/docs/api/platform-api-reference/deployment/create-deployment
    response = requests.post(
        f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments",
        headers={"Authorization": f"Bearer {astro_api_token}"},
        json={
            "astroRuntimeVersion": "12.0.0",
            "defaultTaskPodCpu": "0.25",
            "defaultTaskPodMemory": "0.5Gi",
            "executor": "CELERY",
            "isCicdEnforced": False,
            "isDagDeployEnabled": False,
            "isHighAvailability": False,
            "name": "my_deployment",  # <== this will fail if "my_deployment" already exists
            "resourceQuotaCpu": "10",
            "resourceQuotaMemory": "20Gi",
            "schedulerSize": "SMALL",
            "type": "STANDARD",
            "workspaceId": workspace_id,
        },
    )
    response.raise_for_status()
except requests.exceptions.HTTPError as e:
    print("Failed creating deployment. Reason: " + e.response.json()["message"])
    raise e

The statement, response.raise_for_status(), raises an exception on any HTTP response code that's not 2XX, which are all non-successful HTTP response codes. In the except clause, you can handle this exception however you want.

In case of an error, the Astro API returns a reason in the response body, with the key message. For this specific example, it returns HTTP code 400. This code example prints the error reason. Without the reason, you can't know why a request failed. In the example of a duplicated Deployment name, the logs show the following:

Failed creating Deployment. Reason: Invalid request: Deployment name 'my_deployment' already exists in this workspace

The complete error response structure is:

{
    "message": "Invalid request: Deployment name 'my_deployment' already exists in this workspace",
    "requestId": "f004d12a-29c8-40d8-b239-2b1b615ea45b",
    "statusCode": 400
}

For traceability purposes, you can also include the requestId in your logs, which is an internal Astronomer identifier that Astronomer support can use to track down your request.

Authenticating scripts using API tokens​

In automated scripts, such as CI/CD pipelines, you can query the Astro API with an API token. API tokens grant access to certain Astronomer resources, so it's important to keep the token safe. Do not hardcode the token in code. Instead, store the token in a secret and expose it as an environment variable, ASTRO_API_TOKEN. Storing API tokens in a system dedicated for storing secret values, like GitHub Actions Secrets, ensures secret values are not visible to humans and only referenced by code when needed.

Additionally, a best security practice is the Principle of Least Privilege, where you grant only the permissions necessary to perform an action. This reduces the attack surface, or the number of ways a bad actor could cause damage, in case of a leaked API token.

Astronomer provides three levels of API tokens, from least to most privilege:

This feature is available only if you are on the Enterprise tier or above.

Consider custom Deployment roles for configuring Deployment-level roles with only the necessary permissions.

Use a graphical REST API client for development​

The Astro API documentation provides a convenient web interface to try out the API:

Screenshot of the Astro API documentation

A graphical REST API client can be a helpful addition when developing with any REST API. Popular tools include Postman and Insomnia. The Astro API documentation provides downloads to the API specifications, which are YAML files that define the API structure, that you can load into your tool of choice. Graphical REST API clients often provide convenience features over web-based documentation including query history, value sharing with variables, the ability to define environments with different values, and chained requests, which lets you use results from one query in a follow-up query.

Handle rate limiting with exponential backoff retries​

The Astro API limits requests in case the request rate passes certain thresholds, depending on the type of the request. When a request is rate limited, the API returns an HTTP 429 status code.

It's a best practice to apply an exponential backoff strategy to not overload the server side for rate-limited requests. An exponential backoff strategy gradually increases the time between requests to allow for the server to recover and respond correctly, for example, by waiting 1, 2, 4, or 8 seconds between consecutive requests.

Consider the scenario of waiting for a Deployment to receive status HEALTHY after creation. Creating a Deployment can take a moment, so repeatedly requesting the status from the Astro API without any pause between requests can cause rate limiting.

The following example checks the HTTP response code and handle HTTP 429 separately from other response codes. While this script won't hit any rate limit threshold on the Astro API since the code waits 5 seconds between requests, running multiple scripts simultaneously can quickly increase requests. When receiving an HTTP 429 response, it calculates a waiting period of 2 ** (timeout_attempts - 1) seconds and then increases the period depending on the attempt number.

import datetime
import requests
import time

organization_id = "..."
deployment_id = "..."
astro_api_token = "..."

timeout_secs = 600
timeout_attempts = 0
start = datetime.datetime.now()
while True:
    try:
        response = requests.get(
            f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}",
            headers={"Authorization": f"Bearer {astro_api_token}"},
        )
        response.raise_for_status()
        timeout_attempts = 0

        deployment_status = response.json()["status"]
        if deployment_status == "HEALTHY":
            print("Deployment is healthy")
            break
        if (datetime.datetime.now() - start).total_seconds() > timeout_secs:
            raise Exception("Timeout")
        else:
            print(f"Deployment status is currently {deployment_status}. Waiting...")
            time.sleep(5)

    except requests.exceptions.HTTPError as e:
        if e.response.status_code == 429:
            timeout_attempts += 1
            sleep_duration = 2 ** (timeout_attempts - 1)
            print(f"Request was rate limited. Sleeping {sleep_duration} seconds and trying again.")
            time.sleep(sleep_duration)
        else:
            print("Failed fetching deployment status. Reason: " + e.response.json()["message"])
            raise e

In the previous example, the code defines custom logic for handling rate limits and exponential backoffs. While this works, Python's requests library comes with several built-in utilities you can use to simplify the code. For example, you can avoid defining your own exponential backoff logic by using the Python requests library.

requests.session() creates a persistent session between requests and replaces requests.get(...) with session.get(...) to use the settings configured in the session. This way you don't have to duplicate the same if e.response.status_code == 429 business logic for every request. The code example below configures the urllib3.Retry exponential backoff logic for HTTP status code 429 and up to 10 attempts. This means it waits 1, 2, 4, ..., 128, 256, and 512 seconds in between the ten attempts.

import requests
from requests.adapters import HTTPAdapter
from urllib3 import Retry

session = requests.session()
ratelimit_retry = Retry(status_forcelist=[429], backoff_factor=1, total=10)
session.mount(prefix="https://api.astronomer.io", adapter=HTTPAdapter(max_retries=ratelimit_retry))
response = session.get(...)

You can set a default value for headers using Python's request library as a way to simplify your code. While you can write headers={"Authorization": f"Bearer {astro_api_token}"} with every request, it's cleaner to define this value once and then automatically apply it to every request using the session object:

import requests

session = requests.session()
session.headers = {"Authorization": f"Bearer {astro_api_token}"}

# Before
requests.get(
    f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}",
    headers={"Authorization": f"Bearer {astro_api_token}"},
)

# After
session.get(f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")

Reuse HTTP connections​

For repeated requests to the Astro API, or any REST API, it's a best practice to reuse connections. That means you (client) keep a connection open to the Astro API (server) for multiple requests, instead of opening and closing a connection for every request. This reduces latency and CPU usage. Reusing HTTP connections is also referred to as HTTP persistent connections or HTTP keep-alive, where keep-alive refers to the Keep-Alive header that used to be transmitted with the message.

Using Python's requests library again, requests.session object reuses connections:

import requests

session = requests.session()
session.headers = {"Authorization": f"Bearer {api_token}"}
session.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")
session.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")

A single connection then handles the two GET requests, instead of recreating a connection. This becomes visible when configuring DEBUG logging:

import logging
import requests

logging.basicConfig(level=logging.DEBUG)

# Before
requests.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}", headers={"Authorization": f"Bearer {astro_api_token}")
requests.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}", headers={"Authorization": f"Bearer {astro_api_token}")

# DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.astronomer.io:443
# DEBUG:urllib3.connectionpool:https://api.astronomer.io:443 "GET /platform/v1beta1/organizations/clkvh3b46003m01kbalgwwdcy/deployments/clw9bhr2n0d9801gp7zuigsqn HTTP/11" 200 None
# DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.astronomer.io:443
# DEBUG:urllib3.connectionpool:https://api.astronomer.io:443 "GET /platform/v1beta1/organizations/clkvh3b46003m01kbalgwwdcy/deployments/clw9bhr2n0d9801gp7zuigsqn HTTP/11" 200 None

# After
session = requests.session()
session.headers = {"Authorization": f"Bearer {api_token}"}
session.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")
session.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")

# DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.astronomer.io:443
# DEBUG:urllib3.connectionpool:https://api.astronomer.io:443 "GET /platform/v1beta1/organizations/clkvh3b46003m01kbalgwwdcy/deployments/clw9bhr2n0d9801gp7zuigsqn HTTP/11" 200 None
# DEBUG:urllib3.connectionpool:https://api.astronomer.io:443 "GET /platform/v1beta1/organizations/clkvh3b46003m01kbalgwwdcy/deployments/clw9bhr2n0d9801gp7zuigsqn HTTP/11" 200 None

In the logs, you can see the first example creates two connections. The second example uses requests.session, which reuses connections, so it only creates one connection.

See also

Was this page helpful?