New Google Cloud Platform regions
You can now create an Astro cluster on GCP in the following regions:asia-northeast1(Tokyo)asia-northeast2(Osaka)asia-northeast3(Seoul)asia-south1(Mumbai)europe-central2(Warsaw)europe-west6(Zurich)northamerica-northeast1(Montreal)us-west3(Salt Lake City)
Additional improvements
- You can now search for Organization members by name, email address, and role in the People tab of the Organization view in the Astro UI. You can also search for members in the Access tab of the Workspace view.
Bug fixes
- Fixed an issue where you could not use the KubernetesPodOperator to execute tasks in a Kubernetes cluster outside of your Astro cluster. See KubernetesPodOperator.
New Google Cloud Platform regions
You can now create an Astro cluster on GCP in the following regions:asia-southeast1(Singapore)australia-southeast1(Sydney)europe-west1(Belgium)europe-west2(England)europe-west3(Frankfurt)southamerica-east1(São Paulo)us-west2(Los Angeles)us-west4(Nevada)
Submit Support Requests in the Astro UI
Support requests can now be created and submitted in the Astro UI. You no longer need to open an account on the Astronomer support portal to reach the Astronomer team. To streamline the request process, the Submit Support Request form auto-populates your currently selected Workspace and Deployment in the Astro UI.Parallelism Now Autoscales with a Deployment’s Worker Count
To better scale concurrent task runs, Astro now dynamically calculatesparallelism, which is an Airflow configuration that determines the maximum number of tasks that can run concurrently within a single Deployment.A Deployment’s parallelism is now equal to the current number of workers multiplied by the worker_concurrency value. This change ensures that your task runs won’t be limited by a static parallelism limit as workers autoscale in your Deployment. See Worker Autoscaling Logic for more information.Note that you can still use a static parallelism value by setting AIRFLOW__CORE__PARALLELISM as an environment variable.Bug Fixes
- Fixed a rare issue where some user emails would be associated with the wrong username.
- Fixed an issue where you could not properly sort entries in the People tab by name.
Update Deployment configurations with the Astro CLI
You can now programmatically update the configurations for your Astro Deployments using Deployment API keys and the Astro CLI. Updating a Deployment with an API key doesn’t require manual user authentication, meaning that you can now add Deployment configuration steps to automated processes such as CI/CD pipelines.Specifically, you can now run the following commands with Deployment API keys:astro deployment listastro deployment updateastro deployment variable createastro deployment variable listastro deployment variable update
Bug fixes
- Fixed an issue where a Deployment’s logs wouldn’t load in the Astro UI if it was the only Deployment in the Workspace
Support for the us-east4 GCP region
You can now create an Astro cluster on GCP in the us-east4 region, which is located in northern Virginia, USA.New Datasets page in the Astro UI
You can now use the new Datasets page in the Lineage tab to view a table of datasets that your dags have read or written to. This information can help you quickly identify dataset dependencies and data pipeline access requirements.Click the name of a dataset to show its lineage graph.Bug fixes
- Fixed an issue where the Astro Runtime field of the Astro UI listed the running version as Unknown for Deployments using an unsupported version of Astro Runtime
Data lineage Is now available on Astro
We are excited to introduce data lineage to Astro. You now have access to a new Lineage view in the Astro UI that visualizes data movement across datasets in your Organization based on integrations with Airflow, Apache Spark, dbt, Great Expectations, and more.Built around the OpenLineage open source standard, the data lineage graphs and metadata in the Astro UI can help you better understand your ecosystem and diagnose issues that may otherwise be difficult to identify.For example, if an Airflow task failed because the schema of a database changed, you might go to the Lineage page on Astro to determine which job caused that change and which downstream tasks failed because of it.To learn more about data lineage and how you can configure it on Astro, see:- Integrate Airflow and OpenLineage
- Enable data lineage for External Services
- OpenLineage Compatibility Matrix
This functionality is still early access and under active development. If you have any questions or feedback about this feature, reach out to Astronomer support.
Support for Astro on Google Cloud Platform (GCP)
Astro now officially supports Astro clusters on Google Cloud Platform (GCP). This includes support for an initial set of GCP regions as well as Workload Identity for secure connection to other GCP data services in your ecosystem.For more information about the installation process and supported configurations, see Install Astro on GCP and Resource Reference GCP.Support for Organization-Level user invites
You can now invite users to an Astro Organization without having to first invite them to a specific Workspace. Users invited to an Organization will receive an activation email which brings them directly to the Organization view of the Astro UI.Additional improvements
- Improved the templated emails sent out for user invites with clear instructions for how to get started on Astro
- Improved error messaging behavior on the DAGs and Usage pages of the Astro UI
- New user accounts must now be verified via email before they can access Astro
New AWS node instance types available
To widen our support for various use cases and levels of scale, we’ve expanded the types of AWS node instances that are supported on Astro. You can now create clusters with:To modify an existing Astro cluster to use any of these instance types, see Modify a Cluster.Additional improvements
- Improve the error message that renders in the Astro UI if you try to create a worker that is too large for the Deployment’s node instance type to support. This error message now specifies a clear call to action
Feedback in Astro UI on worker size limits
The Astro UI now renders an error if you try to modify the Worker Resources to a combination of CPU and memory that is not supported by the node instance type of the cluster that the Deployment is hosted on. This validation ensures that the worker size you request is supported by the infrastructure available in your Astro cluster, and minimizes silent task failures that might have occurred due to invalid resource requests.If your Astro cluster is configured with them5d.8xlarge node type, for example, the Astro UI will show an error if you try to set Worker Resources to 350 AU. This is because the maximum worker size an m5d.8xlarge node can support is 307 AU.Additional improvements
- The data plane now connects to various AWS services via AWS PrivateLink. This ensures that traffic to AWS services is kept private and does not traverse the NAT and Internet gateways, reducing the risk of exposing your resources to the internet.
Bug fixes
- Fixed an issue where you could not add a new user to a Workspace if the user had an email address that contained uppercase characters
New analytics page in Astro UI to monitor Deployments
The Astro UI now includes a dedicated Analytics page that contains various Deployment-level metrics. These metrics are collected in real time and can provide insight into how your data pipelines are performing over time.For more information about accessing the Analytics page and the available metrics, see Deployment Analytics.Lineage backend upgrade scheduled for all Organizations
As part of Astronomer’s acquisition of Datakin, data lineage features are coming soon to Astro. The first step in enabling these features is to implement lineage backends for existing Astro customers.Starting on March 31st and continuing over the next couple of weeks, all Astro Deployments on Runtime 4.2.0+ will be upgraded to emit lineage events. As a result of this change, you might start seeing lineage-related scheduler logs such as the following:- You can ignore any lineage logs that indicate an error or failed process, such as the first line in the example logs above. These logs will more accurately reflect the state of your lineage functionality once lineage features are launched on Astro.
- Deployments on Runtime 4.2.0+ will be updated to emit data lineage events only after you push code. Until you do so, this change will not be applied.
- Because Astronomer is upgrading each customer individually over time, the exact date that you will start seeing these logs will vary.
- When you push code to a Deployment on Runtime 4.2.0+ and trigger this update, all other Deployments on Runtime 4.2.0+ in the same Workspace will also restart in order to receive the lineage backend update. If you plan to push code to any Deployment affected by this change, then we recommend doing so at a time where you can tolerate some Airflow components restarting. For more information about expected behavior, see What Happens During a Code Deploy.
New AWS regions available
You can now create new Clusters in:af-south-1(Cape Town)ap-east-1(Hong Kong)ap-northeast-3(Osaka)me-south-1(Bahrain)
Additional improvements
Maximum node count is now configurable per Cluster
As of this release, Maximum Node Count is now a configurable setting for new and existing clusters. On Astro, maximum node count represents the total number of EC2 nodes that your cluster can support at any given time. For an Astro cluster on AWS, EC2 nodes are the primary unit of infrastructure required to run a Deployment and its components, including workers and the Airflow scheduler. New clusters have a maximum node count of 20 by default, but the setting can be modified to any value from 2 to 100 at any time.Previously, maximum node count was a fixed, global setting that applied to all customers on Astro and could not be configured per cluster. Now, your organization can modify this setting as your workloads evolve and more Deployments are created. Once the limit is reached, your cluster will not be able to auto-scale and worker pods may fail to schedule.To update this setting for an existing cluster, reach out to Astronomer support and provide the name of your cluster and the desired maximum node count.Additional improvements
- In Resource Settings, the maximum allowed value for Worker Resources has been increased to 400 AU.
Export task usage as a CSV file
In the Astro UI, you can now export your task usage data from the Usage tab as a CSV file to perform more complex data analysis related to your Airflow usage and costs. For example, you can use the file as the basis for a pivot table that shows total task usage by Workspace.To export your task usage data as a CSV file, click the Export button in the Usage tab.Bug fixes
- Fixed an issue where saving new environment variables in the Astro UI would occasionally fail
Running Docker image tag in Airflow UI
The Docker image that is running on the Airflow webserver of your Deployment is now shown as a tag in the footer of the Airflow UI. Depending on how your team deploys to Astro, this tag is either a unique identifier generated by a CI tool or a timestamp generated by the Astro CLI onastro deploy. Both represent a unique version of your Astro project.
While it is a good proxy, the tag shown in the Airflow UI does not forcibly represent the Docker image that is running on your Deployment’s scheduler, triggerer, or workers.This value is also distinct from the Docker Image that is shown in the Deployment view of the Astro UI, which displays the image tag as specified in the Cloud API request that is triggered on
astro deploy. The image tag in the Airflow UI can be interpreted to be a more accurate proxy to what is running on all components of your Deployment.If you ever have trouble verifying a code push to a Deployment on Astro, reach out to Astronomer support.Additional improvements
- The threshold in order for bars in the Worker CPU and Worker Memory charts to appear red has been reduced from 95% to 90%. This is to make sure that you get an earlier warning if your workers are close to hitting their resource limits.
Bug fixes
- Fixed an issue where malformed URLs prevented users from accessing the Airflow UI of some Deployments on Astro
- Fixed an issue where Astro Runtime 4.0.11 wasn’t a selectable version in the Astro Runtime menu of the Deployment creation view in the Astro UI
Bug fixes
- Removed the Teams tab from the Astro UI. This view was not yet functional but coming back soon
- Fixed an issue where the number of users per Workspace displayed in the Organization view of the Astro UI was incorrect
- Fixed an issue where if a secret environment value was updated in the Astro UI and no other values were modified, the change was not applied to the Deployment
Introducing Astro and a new look
This week’s release introduces a reimagined Astronomer brand that embraces Astro as a rename of Astronomer Cloud. The rebrand includes a new Astronomer logo, color palette, and font.The new Astronomer brand is now reflected both in the Astro UI as well as in the main Astronomer website and documentation.In addition to visual changes, we’ve renamed the following high-level Astro components:- Astronomer Cloud CLI is now Astro CLI
- Astronomer UI is now Cloud UI
- Astro Runtime is now Astro Runtime
New Organization roles for users
The following Organization-level roles are now supported on Astro:- Organization Member: This role can view Organization details and membership. This includes everything in the People, Clusters, and Settings page of the Astro UI. Organization members can create new Workspaces and invite new users to an Organization.
- Organization Billing Admin: This role has all of the Organization Member’s permissions, plus the ability to manage Organization-level settings and billing. Organization Billing Admins can access the Usage tab of the Astro UI and view all Workspaces across the Organization.
- Organization Owner: This role has all of the Organization Billing Admin’s permissions, plus the ability to manage and modify anything within the entire Organization. This includes Deployments, Workspaces, Clusters, and users. Organization Owners have Workspace Admin permissions to all Workspaces within the Organization.
Create new Workspaces from the Astro UI
All users can now create a new Workspace directly from the Overview tab of the Astro UI:When you create a new Workspace, you will automatically become a Workspace Admin within it and can create Deployments. For more information about managing Workspaces, see Manage Workspaces.Bug fixes
- Fixed an issue where authentication tokens to Astro weren’t properly applied when accessing the Airflow UI for a Deployment. This would result in an authenticated user seeing
Error: Cannot find this astro cloud userin the Airflow UI. - Fixed an issue where long environment variable values would spill out of the Value column and onto the Updated column in the Environment Variables view of a Deployment in the Astro UI.
Monitor dag runs across all Deployments in a Workspace
You can view key metrics about recent dag runs through the new DAGs page in the Astro UI. Use this page to view dag runs at a glance, including successes and failures, across all Deployments in a given Workspace. You can also drill down to a specific dag and see metrics about its recent runs.For more information about the DAGs page, see Deployment metrics.Additional improvements
- All resource settings in the Deployment view of the Astronomer UI now show exact CPU and Memory usage to the right of every slider, previously shown only in Astronomer Units (AUs). This makes it easy to know exactly how many resources you allocate to each component.
- A banner now appears in the Astronomer UI if a Deployment is running a version of Astro Runtime that is no longer maintained. To make the most of features and bug fixes, we encourage users to upgrade to recent versions as much as possible.
- Added more ways to sort pages that utilize card views, such as the Deployments page
- Added user account avatars next to usernames in several places across the Astro UI
Bug fixes
- Removed the Environment field from the Deployment view of the Astronomer UI. This field is not currently functional and will be re-added as soon as it is.
Support for third-party identity providers
You can now integrate both Azure AD and Okta as identity providers (IdPs) for federated authentication on Astro. By setting up a third-party identity provider, a user in your organization will be automatically logged in to Astro if they’re already logged in via your IdP. By adding new Astro users through your IdP’s own user management system, Workspace Admins can automatically add new users to their Workspace without those users needing to individually sign up for Astro.For more information about this feature read Set up an identity provider.Support for the Astro CLI
The Astro CLI (astro) is now generally available as the official command-line tool for Astro. It is a direct replacement of the previously released astro executable and comes with various significant improvements. We encourage all customers to upgrade.For more information on the Astro CLI, see CLI Release Notes. For install instructions, read Install the CLI.Multiple authentication methods for a single user account
Astro now supports multiple authentication methods for a single user account. This means that as long as you’re using a consistent email address, you now have the flexibility to authenticate with GitHub, Google, username/password, and/or an external identity provider (idP) at any time. Previously, a single user account could only be associated with one authentication method, which could not be changed after the account was created.This also means that all Organizations now have GitHub, Google, and username/password authentication methods enabled by default for all users.Additional improvements
- Changed the default RDS instance type for new clusters from
db.r5.xlargetodb.r5.large, which represents a monthly cost reduction of ~50% for newly provisioned clusters. Customers with existing clusters will need to request a downscale via Astronomer support
Identity-based login flow
Astro now utilizes an identity-based login flow for all users. When you first log in via the Astro UI, you now only need to enter the email address for your account. Astro assumes your Organization and brings you directly to your Astro Organization’s login screen.This change serves as a foundation for future SSO and authentication features. In upcoming releases, users will be able to authenticate via custom identity providers like Okta and Azure Active Directory.Additional improvements
- Significant improvements to the load times of various Astro UI pages and elements.
- In the Astro UI, the tooltips in the Resource Settings section of a Deployment’s page now show the definition of 1 AU. This should make it easier to translate AU to CPU and Memory.
- Scheduler logs in the Astro UI no longer show
DEBUG-level logs. - To ensure that all workers have enough resources to run basic workloads, you can no longer allocate less than 10 AU to Worker Resources.
Improvements to “Scheduler Logs” in the Astro UI
The Scheduler Logs tab in the Astro UI has been updated to make logs easier to read, separate, and parse. Specifically:- You can now filter logs by type (
DEBUG,INFO,WARN, andERROR). - The page now shows logs for the past 24 hours instead of the past 30 minutes.
- The page now shows a maximum of 500 logs instead of a lower maximum.
- When looking at a Deployment’s logs, you can return to the Deployment’s information using the Deployment Details button.
Removal of worker termination grace period
The Worker Termination Grace Period setting is no longer available in the Astro UI or API. Previously, users could set this to anywhere between 1 minute and 24 hours per Deployment. This was to prevent running tasks from being interrupted by a code push. Today, however, existing Celery workers don’t have to terminate in order for new workers to spin up and start executing tasks. Instead, existing workers will continue to execute running tasks while a new set of workers gets spun up concurrently to start executing the most recent code.To simplify Deployment configuration and reflect current functionality:- The worker Termination Grace Period was removed from the Astro UI
- This value was permanently set to 24 hours for all Deployments on Astro
Additional improvements
- Removed Kubernetes Version column from the Clusters table. This value was previously inaccurate and is not needed. The Kubernetes version of any particular Astro cluster is set and modified exclusively by Astro as part of our managed service.