Introducing the Astro Platform Release, Q4 2023

Welcome to the launch of the latest Astro Platform release! We are thrilled to bring you a suite of powerful features designed to simplify connectivity, secure and optimize your data platform, and empower your teams to unlock the full potential of their data.

The three enhanced features included in this release are:

• Simplify connection setup across deployments with Connection Management, the Astro-native secrets backend that you don’t have to manage
• Upgrade with confidence and ensure business continuity with rollbacks and upgrade utilities
• Enable cost savings by scheduling zero-cost downtime during development deployment with Scale to Zero Deployment

In this article, we’ll dive into these key features and explore how they can benefit your organization.

Connection Management: A Single Way to Create, Secure, and Share Connections Across Environments

The centerpiece of this release is Connection Management, an Astro-native secrets backend that you don’t have to manage. This feature offers a secrets backend on Astro, providing an out-of-the-box, native solution that makes it easier for you to manage Airflow connections in one central location. No more hassle of setting up connections multiple times across deployments, or local development environments.

In Airflow, a connection is specific to one Airflow instance and if you want to use it in another, you have to set it up all over again. Connection Management allows teams to securely share connections across deployments and local development environments. You only need to create the connection once and choose which deployments to share it with. We know that setting up connections (even just once) can be a time-consuming and tedious task for developers. That’s why Astro Connection Management offers purpose-built integration forms that prompt you for exactly what you need, making the process effortless. You’ll no longer need to search for documentation on the exact properties required for a given type of Connection again.

With an intuitive built-in inheritance model, Connection Management allows you to override fields based on each deployment, ensuring flexibility. For example, you can set up a connection to your database provider once, and point to your production instance from your production deployments, and your development instance from your development deployment. The ability to share connections across deployments means you can spin up branch-based deployments effortlessly, equipped with the necessary credentials from an existing deployment.

Onboarding new engineers with their local environments becomes a breeze with connection sharing through Astro’s Connection Management. The process requires near-zero environment setup time, enabling you to quickly get new engineers up and running. Astro's authorization model allows you to manage credential access from day one, ensuring a seamless and secure experience.

Finally, Connection Management allows you to centralize control of your connections. Astro gives you complete trust and visibility over how your connections are used on the platform. Connection Management shows you every service that is connected to Airflow in one place, so you don’t need to look into each individual instance to understand your exposure. With a single source of truth, you can govern your credentials effectively and maintain security.

To learn more about the capabilities of Connection Management, get detailed instructions in the documentation or try a step-by-step tutorial. If you are new to using connections, check out the in-depth Connections 101 Module on Astronomer Academy and join our upcoming webinar on how to manage connections in Airflow.

Upgrade Airflow with Confidence: Introducing Deployment Rollbacks and Upgrade Utilities

Upgrading Airflow should be a confident stride forward, not a leap of faith. This platform release introduces Deployment Rollbacks and Upgrade Utilities to ensure a smooth journey upgrading Airflow.

Upgrading major and minor versions of Airflow can pose significant challenges, potentially causing environments and DAGs to break, resulting in lost or incomplete data. The most common issues include incompatible dependencies and API changes, making the upgrade process time-consuming and error-prone.

The new Local Upgrade Test command in the Astro CLI will proactively identify these common potential issues with an upgrade. The new utility includes version testing to identify Airflow Providers and Python packages that have changed, categorizing them by additions, removals, and updates. Also included is testing to identify DAGs not importing due to compatibility issues, providing you with a list of DAGs with import errors. You can then test upgrades locally, mitigating the risk of deploying potentially problematic changes to their production environments running on Astro.

Upgrade Utilities play a crucial role in efficient environment management, addressing compatibility issues before things break. The Local Upgrade Test command offers a user-friendly solution to tackle the challenges of upgrading Airflow, streamlining the process and empowering users to deploy updates with the confidence (and passed tests) they need to upgrade safely.

With Deployment Rollbacks, you can effortlessly revert deployments to a known "good" state in case of any issue during an upgrade. We know that the fear of potential disruptions often hinders the adoption of the latest Airflow versions, limiting the exploration of new features and improvements. Deployment Rollbacks eliminate this barrier, allowing data engineers to embrace the latest Airflow releases with confidence, knowing they can revert to a stable state should any issues arise. This, in turn, empowers data engineers to focus more on the development of data workflows and less on the potential disruptions that come with upgrades.

This feature is particularly relevant for teams managing multiple Airflow environments, where high availability and reliability of deployments are essential. Platform engineers can now confidently navigate upgrades. This not only streamlines their tasks but also enhances the overall efficiency of managing multiple different Airflow versions, reducing organizational complexity. Secure and reliable in-place upgrades become a reality when you can confidently upgrade production deployments, knowing you can roll back to a stable state if issues arise, and minimize the risk of data downtime.

We’re excited about how these features enhance overall reliability, reduce the risk of costly errors, and empower users to confidently embrace the latest Airflow versions, fostering a culture of continuous improvement and innovation.

Read our documentation for more information.

Scale to Zero Development Deployments: Optimizing Efficiency and Cost Savings

To deliver on cost-effective scaling, we’re introducing a new deployment type, a Development Deployment, for which you can set a Hibernation Schedule. In the specified timeframe of the Hibernation Schedule, all components of the Development Deployment will be scaled down to zero – from the Airflow system components to scheduler to workers – and incur zero costs. With Scale-to-Zero Development Deployments, you only pay for what you use, when you want to use them.

For example, you can choose to spin down your Development Deployment on the weekends when your team is not developing or testing, regardless of DAG schedules. Or, keep your Development Deployment in Hibernation Mode until you want to test something. Turning Hibernation Mode on or off is easily done through the Astro UI or the Astro API by users with Operator role access. You can now take advantage of cost savings by scaling down all development deployments when they're not in use, optimizing costs without impacting production.

This feature is currently in Private Preview. Read our documentation for further details and contact us if you'd like to be included in the Private Preview.

Summary

In this release, we build upon the innovative feature set of previous releases. We're continuing our commitment to streamline processes, reduce complexities, and enhance efficiency in your Airflow data operations — allowing you to focus on data, not infrastructure.

For a more comprehensive overview of the enhanced features included in this release, join our webinar where we'll explore each in detail, complete with live demos.

If you’re ready to get started, try Astro now with our 14-day free trial.