Astro release notes

Astronomer is committed to continuous delivery of both features and bug fixes to Astro. To keep your team up to date on what's new, this document will provide a regular summary of all changes released to Astro.

Latest Astro CLI Version: 1.30.0 (Release notes)

Latest Astro Runtime Version: 12.4.0 (Release notes)

November 19, 2024​

Additional improvements​

• Improved error messaging when an Organization Owner email is not supplied, preventing you from creating default Deployment alerts.
• Updated the onboarding flow so that newly invited users see a signup page, instead of the login page.

Bug fixes​

• Fixed a bug in Observe where selecting a dataset with only downstream dependencies did not fully update the asset lineage graph. Now, all dependencies are correctly displayed.

November 12, 2024​

Network egress management​

You can now enable Private Network Egress or configure Customer Managed Egress for DAG Workloads for dedicated AWS clusters to ensure security and compliance, and to provide a data loss protection architecture to secure against unauthorized data transfer.

Additional improvements​

• Added the ability to work with the Astro Environment Manager through the Astro API, by working with Environment Objects. Environment Objects are the API representation of the different functionalities the Environment Manager supports such as Universal Metrics Export, Airflow Variables, and Connections.
• Added the ability to filter assets list by asset type, namespace, and DAG.
• Updated Deployment configuration page.
• Removed Airflow Variable Key from UI when editing Airflow Variable.
• Moved the IP Access List from being a sub-section of the Authentication settings page to a dedicated page within Organization Access Management.

Bug fixes​

• Fixed a bug where only organization owners were considered as fallback emails for Deployment Health Alerts alerts. With this fix, Workspace owners will be considered first and Organization owners will only be considered as a final fallback if no human Workspace owners exist.

November 5, 2024​

Proactive Alerts for Deployment health​

Astro now proactively alerts you on Deployment-level health issues when infrastructure components aren’t running as expected. New Deployments now receive four Deployment health alerts by default. These alerts provide insight into infrastructure-level issues and suggest specific actions to remediate the problem. Alerts notify the Contact Emails associated with a Deployment by default and are fully customizable. See Astro Alerts.

Create data products with Astro Observe​

You can now use Astro Observe to create Data Products, which allow you to configure alerts and receive insights into the performance of the pipelines where you integrate them. See Create a Data Product.

Additional improvements​

• Added validation of environment variable keys in deployment create and update requests. Environment variables must now match the following regex: ^[a-zA-Z_]+[a-zA-Z0-9_]*$.
• Upgraded Azure clusters to Kubernetes 1.29
• Improved descriptions for scheduler sizes and High Availability in advanced section of cluster creation.
• Improved description of Development Mode in advanced section of Deployment creation.
• Removed a warning that displayed in the Astro UI when a user's Deployment was using CI/CD Enforcement.

Bug fixes​

• Fixed an issue where Open Airflow button was disabled, even when the Deployment was not getting created or hibernating.
• Fixed a bug where the Airflow webserver would restart for DAG-only deploys.

October 29, 2024​

Airflow mini-scheduler turned off by default​

The mini-scheduler config (AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION) is now disabled by default. Previously enabled to optimize performance at scale, the mini-scheduler sometimes led to unexpected task state changes and failures, especially when the main scheduler got out of sync. This change reduces debugging complexity, but users with very large DAGs, many downstream dependencies, and the need to run over 10,000 tasks per hour can re-enable it for potential performance benefits by setting AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION to True.

Bug fixes​

• Fixed a bug where only users with Organization-scoped roles could create or update Deployment alerts, even if they had Deployment alert creation and update permissions enabled.

October 22, 2024​

Bug fixes​

• Fixed a bug affecting dbt deploys for Deployments using the Kubernetes executor.

October 15, 2024​

Additional improvements​

• The Scope and Scope Entity ID columns of the Alerts and Notification Channels lists are now consolidated to the Scope column, which displays the entity name and links to the Deployment or Workspace the alert is enabled for. If the scope of the Alert or Notification Channel is the entire Organization, the Scope column value is This Organization. This enhancement allows you to navigate directly to the Deployment or Workspace from the list.

Bug fixes​

• Fixed a bug where the Default Pod Size field was showing incorrect sizes.

October 9, 2024​

Updates and changes to Astro Alerting UI​

• You can now define whether a notification channel is available to specific Deployments, an entire Workspace, or to an entire Organization.
• You can create and edit alerts and notification channels of any type through the Organization Alerting page, whether they are Deployment or Organization alerts. If you do not have user permissions with privileges to make a particular type of alert, you can still access the Alerting editor in the Organization menu to create, edit, and manage both alerts and notification channels.
• Previously, you could access and edit Workspace alerts from the Alerting page in the main navigation of the Astro UI. Now, you can create and manage your Organization alerts through the Alerting page in the Organization menu, or your Deployment-specific alerts in your Deployment's Alerts page.

Bug fixes​

• Fixed a bug where editing Airflow variable links in the Environment Manager would delete any links to Deployments. Now, editing Airflow variables in the Environment Manager does not affect linked Deployments.

October 1, 2024​

Additional improvements​

• Added Updated By and Created By columns to the Workspaces list.
• Enhanced the onboarding experience so you can use one of the template Deployments without connecting to your GitHub account.

Bug fixes​

• Fixed an issue that showed incorrect pagination numbers in the table footer when searching the Alerts list.
• Resolved an issue where the globalFilter search paramameter would not clear if you deleted the input value entirely.
• Fixed an issue with the dbt deploy rollback functionality where dbt bundles weren't updated to the rollback deploy version's bundle state.

September 24, 2024​

Additional improvements​

• The Universal Metrics Exporter is now generally available to Team-tier level customers. See Export metrics for information about how to export infrastructure metrics about your Airflow use on Astro to your preferred third-party observability tools.
• Updated the Alerts tab in the Astro UI to improve bulk-selecting alerts, increase the number of characters you can use when naming an alert, and allow auto-generated alert names to include the alert type in the alert name.
• Improved the display of long strings to have a truncated display, which expands to show the full string when you hover over or copy the value.

Bug fixes​

• Fixed an issue where you couldn't submit Log Summary feedback for different task runs that had the same task ID.

September 17, 2024​

Additional improvements​

• Updated the Organization Dashboards' Cost Breakdown tab, which now displays top-level Network and Cluster costs.

Bug fixes​

• Fixed an issue which prevented KPO task Pods from terminating correctly.

September 9, 2024​

Scale Deployments Reliably with XL Deployment size and standalone DAG processor​

As your team and Airflow use cases grow, scaling up your Airflow environment reliably can be a challenge. Complex, dynamically-generated DAGs, sub-optimal DAG parsing practices, or a growing business that requires a larger data pipeline can strain DAG processing and threaten your Airflow scheduler’s availability.

Astro Hosted Deployments now support high-scale environments more reliably by separating the DAG processor from the scheduler. You can take advantage of this Airflow best practice in Astro Hosted Deployments with Medium and larger sizes. By separating the DAG processor from the scheduler, Astro protects your core scheduling function from competing with the DAG processor for resources. This separation ensures scheduler availability for your most complex use cases and improves overall DAG parsing performance.

In addition to this change, you can now create Extra Large size Deployments to confidently run your largest workloads on Astro. The Extra Large Deployment provides 8 vCPU and 16 GiB of memory and comes with two DAG processors to support your largest and most complex workloads, ranging from dynamically-generated DAGs to high-scale pipelines that can grow alongside your business.

This update is available on Runtime version 9.7.0 and greater; if you are running a lower Runtime version, upgrade your Runtime version. To learn more about this feature, read more about Deployment scheduler resources.

Astro UI improvements​

Improved the navigation and design for managing clusters, editing settings, and accessing analytics for Organizations and Deployments.

These changes include:

• A new section in the main menu to include an Organization section with links to important resources for administering your Astro Organization. These include links to your Organization Settings, Organization Dashboards, and quick access to your cluster settings.
• A dedicated Deployment Analytics page in your detailed Deployment information, which now includes improvements to the tooltips and metrics visualization design. See Deployment metrics for more information about the available metrics.
• Your Deploy History now appears on the Overview page of your Deployment. See Deploy history for more information about how to view your code deploy history.

Create and manage Airflow variables with the Environment Manager​

You can now create and manage Airflow variables for Deployments across your Workspace through the Astro Environment Manager. This allows you to quickly and securely create Airflow variables once and share them to multiple Deployments without having to set up your own secrets backend.

See Create Airflow variables in the Astro UI for more information.

September 3, 2024​

Additional improvements​

• Made updates to the Astro UI for editing Astro Alerts, to clarify whether you are discarding your edits to an alert, or discarding the alert.

Bug fixes​

• Fixed an issue that blocked some users from pushing project images to the Docker registry.

August 27, 2024​

Additional improvements​

• You can now give Airflow Dataset Create and Delete permissions to custom roles. This capability is provided to Deployment Admin, Workspace Author, Workspace Operator, and Workspace Owner default roles. Previously only Workspace Owner and above had access to this permission set. See more information about User Permissions and Custom Deployment Roles.
• If your Astro Deployments run on AWS or GCP clusters, you can now see the Account ID of the clusters in the Cluster Details page in your Organization Settings.
• For Deployment alert emails, the sender now shows as Astro. It previously showed the sender as Postmaster.
• You can now see Deployment details while Astro creates your Deployment, where only the specific actions that are not available until after creation is complete are disabled.

August 20, 2024​

Additional improvements​

• Added the ability to specify Custom Workload Identity at Deployment creation time when creating an AWS Deployment.
• Improved the descriptions of default hibernation schedules in the UI.
• Added more detail in descriptions for code deploy failures in the Deploy History page.

Bug fixes​

• Fixed an issue where you could not use global text search for DAGs in your Alerts.
• Resolved a problem where search query parameters might be deleted when using pagination.

August 13, 2024​

Additional improvements​

• Improved the descriptions of Deployment Health Alerts to provide more specific information about performance thresholds.

Bug fixes​

• Fixed an issue where DAG lists didn't update when you switched Deployments in the Astro UI.
• Fixed a bug where the cursor would disappear while typing in certain fields in the Astro UI.

August 7, 2024​

Customize labels on your metrics export​

You can now add Key:Value pair labels to metrics that you export using the Universal Metrics Exporter. This allows you to tag and record metrics coming from specific Deployments or Workspaces. See Export metrics from Astro for more information.

Additional improvements​

• Added a new template demonstrating how to run dbt projects on Astro using Cosmos when you're onboarding. See Get started on Astro to set up a trial account and try out a template.

Bug fixes​

• Fixed an issue where a user could not bulk select alerts if they had different names but the same configurations.

July 30, 2024​

New alerts for Deployment health incidents​

This release introduces four new alert types that correspond to four of the existing Deployment health incident types. These new Deployment Health Alerts allow Astro to proactively notify you when Deployment health issues arise.

Using Deployment health alerts, you can:

• Improve alerting coverage beyond DAG and task failures to address infrastructure-level incidents that are otherwise difficult to monitor.
• Proactively monitor Deployment health and take immediate remediation actions through email, Slack, and PagerDuty to reduce mean time to resolution.
• Share Deployment health visibility across teams.

See Astro Alerts: Trigger Types and Deployment Health Incidents for more information.

Additional improvements​

• Now, when you link directly to your Astro or Airflow UI, any link previews successfully have improved visuals and include metadata information.
• Improved the Astro Alerts UI to streamline creating notification channels and alerts across DAGs and Deployments. See Astro Alerts for setup information.
• Added an example Universal Metrics Export dashboard configuration file for Grafana Cloud. See Export metrics from Astro for setup instructions and Grafana example for a configuration example.
• Organization Dashboards are now generally available to Enterprise-tier customers. See View Organization Dashboards for more information.

Bug fixes​

• Fixed a bug where the Astro UI would show an error in your Deploy History instead of your Runtime Version, if the Runtime version is yanked.
• Fixed an issue when creating an alert where you couldn't select tasks with the same task_id across multiple selected DAGs.

July 24, 2024​

Run your first DAG on Astro with GitHub Integration​

Now, when you first set up your Astro account, you can choose to connect your GitHub account and run a sample Astro project template and DAG. Previously, to run your first DAG on Astro, you needed to either download and use the Astro CLI or work with GitHub Actions. Now, the onboarding process allows you to customize your first experience with Astro to focus on your use case, whether that's Business Operations, Generative AI, or learning Airflow, and set up a GitHub Integration to quickly clone and deploy the example DAGs. Try it now with Start Astro.

Work with dbt projects on Astro​

You can now deploy dbt code to Astro quickly and independently of your DAG or image deploys. Data build tool (dbt) core is an open source tool for data transformation that uses SQL to transform your data instead of Python in your DAGs. You can also use both Cosmos, an open source tool that integrates dbt models into Airflow and treats them like DAGs, and dbt deploys on Astro to have unparalleled visibility into your dbt tasks with the Airflow UI and a streamlined code deploy process.

See Deploy dbt projects for more information about how to get started.

Additional improvements​

• Improved how the GitHub Integration in the Astro UI links directly to the Astro Project in a GitHub repository.
• (Astro Hosted only) The customer managed workload identity setting for Deployments is now generally available. This allows you to grant Astro Deployments all of the permissions of an AWS IAM role. See Attach an IAM role to your Deployment for detailed information.
• The ability to deploy automatically from GitHub using the official Astro GitHub integration is now generally available. See Deploy code with GitHub for setup steps.
• Astro Runtime version 11 is now classified as Long Term Support (LTS). This means that instead of being supported until October 2024, Astro will support Runtime version 11 until October 2025. See the Astro Runtime maintenance and lifecycle schedule for more information, including information about restricted versions.

Bug fixes​

• Fixed some bugs in the Astro UI for defining ephemeral storage to show storage limits, fix maximum pod size for storage, and add storage for default pod size.

July 16, 2024​

Additional improvements​

• Added the following metrics to Metrics Export:
  • The new airflow_executor_open_slots, airflow_dagrun_dependency-check, and airflow_dagrun_dependency-check.<dag_id> metrics allow you to collect metrics about your DAGs and executor status.
  • Monitor task execution with kube_pod_container_resource_limits. This new metric enables you to track resource use against the configured limits so you can understand if task execution meets your configured CPU, memory, or storage limits for your Celery Workers or Kubernetes Executor and KubernetesPodOperator pods.
• Added new suporting documentation for the Astro Terraform Provider, including a Getting Started guide and code examples for common uses. Read Astro Terraform Provider for more information.

Bug fixes​

• Fix an error where start times in the UI were different from actual DAG run trigger times.

July 9, 2024​

Additional improvements​

• Fixed a bug that prevented adding a date input for a DAG filter in the Astro UI.

July 2, 2024​

Export metrics about your Astro Deployments to observability tools​

You can now export comprehensive, operational metrics about the performance of your Astro Deployments to third-party observability tools, such as New Relic, using the new Universal Metrics Exporter. This new feature allows you to configure a Prometheus endpoint to export metrics using the Prometheus data model, which means you can integrate your Astro observability metrics directly into your existing monitoring tools. See Export metrics from Astro for setup instructions.

Added ephemeral storage metrics to Astro Deployment Analytics​

To assist you in determining the amount of custom ephemeral storage to configure for your workers and schedulers, you can now use Deployment Analytics to see relevant usage metrics. These include:

• Ephemeral Storage Usage metric that shows a % of usage against the configured limit for your Celery Workers, KubernetesPodOperator, Kubernetes Executor, and Scheduler.
• Dynamic Y-axis Scaling to the Celery Worker, KPO/KE, and Scheduler Deployment analytics panel to include dynamic zooming.

See Deployment Analytics for more information.

Additional improvements​

• The Scheduler heartbeat not found Deployment health incident is downgraded to Warning from Critical.

June 25, 2024​

Customer managed workload identity for AWS​

The Customer Managed Identity Deployment setting is now available on AWS. This means that you can now assign an existing workload identity and AWS IAM role to your Airflow Deployments on Astro. When you use this setting, your Deployment uses the identity to assume the permissions of your IAM role and gain secure access to your data services. With this feature, you can:

• Re-use or share a customer managed identity across many Deployments, either ephemeral or static.
• Leverage existing identities when migrating from MWAA or open source Airflow environments

This can reduce friction when migrating to Astro. See Attach an IAM role to your Deployment for detailed information.

Self-healing workers automatically address stuck queued tasks​

A new improvement to the Astro Data Plane now automatically identifies when Celery workers are online and healthy, but not actually processing new tasks. When this happens, it looks like many tasks are stuck in a queued state. With this new feature, you will experience a lower frequency of tasks stuck in a queued state, which causes performance and reliability issues for your Airflow implementation.

Self-healing workers use the following process:

• It first identifies workers that both have tasks stuck in a queued state for an extensive time period and are in Deployments where the concurrency available means that tasks should not be queued.
• The healer then kills catatonic workers that are not running tasks or shifts workers that are still running tasks into a warm shutdown period.
• After the catatonic worker is shut down, a new healthy, worker comes online and resumes tasks.

This feature is automatically enabled for the Astro Hosted infrastructure and does not require any action.

Additional Improvements​

• Improved the formatting for how IP addresses are listed in the Astro UI to make it easier to copy and paste them.

Bug fixes​

• europe-west6 is no longer available as a region for dedicated clusters on GCP.

June 18, 2024​

Additional Improvements​

• Moved the Environment variable tab in Deployment Settings to the Environment tab, and renamed it to Environment Variables. This helps disambiguate between Environment Variables and Airflow Variables in the Astro UI. Refer to Manage environment variables for more information about the different methods to set up and manage Environment variables.

Bug fixes​

• Fixed a bug where a duplicate Customer managed identity option displayed when configuring Deployment settings.

June 11, 2024​

Restrict Astro access to specific IP Address ranges with IP Access list​

Organization owners can now limit the IP addresses of users to control who can access the Astro UI or programmatically work with the Astro API for their organization. This allows you to restrict access to your Astro Organization to users on a VPN or to specific network ranges. See Set up IP Access list.

Additional improvements​

• When using the GitHub integration, you can now retry a failed deploy or trigger a Git deploy directly from your Deployment settings page in the Astro UI. See GitHub Integration for more information.
• Deployments at the end of an Astro trial now automatically hibernate and are not immediately deleted. If your trial has ended, add a credit card number to your Organization to access your Workspaces and wake your hibernating Deployment. While in hibernation, Deployment settings and configurations are preserved. If you do not enter a payment method within 30 days, your hibernating Deployments and all corresponding data are deleted. See Start a trial for more information.
• The ability to create, update, and delete Deployment API tokens is restricted to users with the Workspace Owner role. Read Workspace user permissions for more information.

Bug fixes​

• Fixed an issue where the Organization Settings Dashboard failed to load correctly.

June 6, 2024​

API keys are no longer supported​

As of June 1, 2024, Deployment API keys are no longer supported. Replace your API keys with Deployment API tokens, confirm successful operation with your API tokens, and then delete your API keys.

Configure ephemeral storage on worker Pods​

You can now customize the amount of ephemeral storage for data intensive workloads on Celery, Kubernetes, and KubernetesPodOperator workers. Previously, to accommodate larger workloads, you needed to integrate an external object storage or database tool to process large datasets within a single task. Now, you can customize the ephemeral storage when creating or updating your Deployment so that all data processing happens directly in your task Pod.

You are only charged for requested resources which are greater than the minimum defaults for each worker type:

• Celery worker: 10 GiB minimum by default. 100 GiB maximum.
• Kubernetes executor/ Kubernetes pod operator: 0.25 GiB minimum by default. 100 GiB maximum.

Automate Airflow, resource, and infrastucture management with the Astro Terraform Provider​

You can now use Terraform to automate managing resources and changes to large organizations and Airflow infrastructure with the Astronomer Terraform provider package. The provider is available through both the Terraform Registry and a public Github repository, where you can review the provider's code, make issues, and create pull requests.

Refer to the Astronomer Terraform Provider docs in the Terraform registry or the GitHub repository for more information.

Additional improvements​

• The Astro UI now validates cron expressions for development Deployment hibernation schedules and shows commonly used hibernation schedules that you can enable or disable. Refer to Create a hibernation schedule for more information.
• If you don't already have a GitHub repository, you can now create one when you authorize the GitHub Integration from the Astro UI. Previously, you could only connect Astro to existing repositories.

Bug fixes​

• Fixed an issue where you could not use Airflow connection testing with Azure Managed Identities on Astro.

May 30, 2024​

Additional improvements​

• Dedicated clusters are now available only to Team tier customers and above.
• Added a centralized docs reference page that lists all open source Apache Airflow provider packages and their versions for each Astro Runtime version. See Provider package reference docs.

Bug fixes​

• Fixed an issue where you could make changes to clusters and Deployments on an inactive Organization.
• The default storage resources for the Kubernetes executor and KubernetesPodOperator Pods are now enforced with 0.25Gi instead of 10Gi in some cases. You can still customize the resource allocation for Kubernetes pods depending on your needs. See Configure Kubernetes Pod Resources

May 22, 2024​

The Astro GitHub integration is now in Public Preview​

The ability to deploy automatically from GitHub using the official Astro GitHub integration is now in Public Preview.

The Astro GitHub integration is a new way to automatically deploy code from a GitHub repository to Astro by merging pull requests or making commits directly to specific branches, without needing to configure a GitHub Action. Additionally, the GitHub integration displays Git metadata directly in the Astro UI, including Git commit descriptions and gives you greater visibility into the status and logs of individual code deploys.

See Deploy code with the Astro GitHub integration for more information.

Bug fixes​

• Fixed an issue where users with custom roles could see roles that could not be assigned in the Astro UI.

May 15, 2024​

Additional improvements​

• Airflow connections that you configure through the Astro UI environment manager are now mounted to Deployment schedulers, meaning that scheduler processes can now make use of these Airflow connections.

Bug fixes​

• Fixed an issue where you couldn't update users who were added through SCIM but didn't belong to an Organization.
• Fixed an issue where a user with a custom role could create API Tokens, users, or teams with greater permissions than their own.

May 8, 2024​

Updates to address ranges for dedicated clusters on Google Cloud Provider​

Astro on GCP Dedicated Clusters uses source network address translation (SNAT) that performs many-to-one IP address translations for connections to your data sources and defaults secondary ranges to RFC 6598 address space (non-standard Private IP addresses), to minimize the risk and concern with IP overlap and exhaustion. Your target data sources will see connections from Astro using the VPC Subnet Range when using private networking, like VPC Peering or VPN. If you want to configure private connectivity, ensure the default subnet and peering ranges do not overlap with your target data source network when you're creating your dedicated cluster. See Create a dedicated Astro cluster for more details.

Improvements to Astro performance​

As part of continued investment in the reliability, performance, and scalability of Astro, Astronomer is embarking on a migration of public and private image registries. Astro Runtime clusters will benefit from more performant, globally distributed and geo-replicated image registries, with built-in registry resilience if a regional outage occurs. This is in addition to the previous release of the registry cache local to every Astro cluster.

No end user or task runtime change or impact is expected as part of the backend cutover during the week of May 6th, 2024.

Additional improvements​

• You can no longer create Deployments using Astro Runtime versions marked as yanked in https://updates.astronomer.io/astronomer-runtime, even if your Organization has enabled creating Deployments with deprecated Runtime versions. These versions of the Astro Runtime have known issues and should not be used. For more information, see Restricted Runtime Versions.

Bug fixes​

• Fixed a bug where multiple users could not access Organization Dashboards simultaneously.

April 30, 2024​

Deploy automatically from GitHub using the official Astro GitHub integration​

The Astro GitHub integration is a new way to automatically deploy code from a GitHub repository to Astro without needing to configure a GitHub Action. Compared to using GitHub Actions, the Astro GitHub integration:

• Allows you to enforce software development best practices without maintaining custom CI/CD scripts.
• Enables developers to iterate on DAG code quickly.
• Shows Git metadata directly in the Astro UI, including Git commit descriptions.
• Gives you greater visibility into the status and detailed logs of an individual deploy.

See Deploy code with the Astro GitHub integration for more information.

April 23, 2024​

Restrict a custom Deployment role to specific Workspaces​

You can now restrict the use of a custom Deployment role to specific Workspaces. Use Workspace role restriction when some Workspaces in your Organization have different requirements for how users interact with Deployments. See Restrict a custom Deployment role to specific Workspaces for setup steps.

Additional improvements​

• The custom Deployment roles feature is now generally available.

• You can now promote a development Deployment to a production Deployment by switching off the Development Mode toggle in the Deployment's configuration.

• Workspace Members can now see and use custom Airflow menu items. To give a custom role this permission, you can add deployment.airflow.customMenu.get to the role's permissions list. This permission works only on Deployments running Astro Runtime 9 or later.

  Note that you might have to modify the code for your menu item plugins to make them work on Astro. See Appbuilder menu items for more information.

• You can now filter the Workspaces and clusters lists in the Astro UI by name.

Bug fixes​

• To improve the reliability of data lineage for customers who leverage it, data lineage is now a private preview feature that can be enabled upon request. To reenable data lineage for an Astro Organization, reach out to your account team.

April 16, 2024​

Bug fixes​

• Fixed an issue where you couldn't grant a custom Deployment role to a Deployment API token using the Astro API.

April 9, 2024​

Additional improvements​

• You can now manage custom Deployment roles using the Astro API.
• Improved the time it takes to load Deployment analytics in the Astro UI.
• You can now view info-level incidents from the Deployment health status indicator in the Astro UI.
• It is now possible for workers to use up to 6400 CPUs and 12800 GiB of memory on a single Deployment.

Bug fixes​

• Fixed an issue where you couldn't configure boolean values for Airflow connections in the Astro UI.
• Fixed an issue where Airflow connections configured through the Astro UI did not work with deferrable tasks.

April 3, 2024​

Additional improvements​

• Deployment health incidents are now generally available.
• Deployment hibernation is now in public preview.

Bug fixes​

• Fixed an issue where connections configured in the Astro UI were not mounted to triggerer Pods, resulting in failed runs for tasks that use deferrable operators.
• Fixed an issue where a field in the Snowflake - Private Key (Content) connection type was not parsed correctly.
• Fixed an issue where Organization Members could delete and update Astro alerts.
• Fixed an issue where the Astro UI didn't show the correct amount of available ephemeral storage for the default worker queue.
• Removed additional dependencies to make Astro more resilient to Quay outages.

March 26, 2024​

Refactored Astro API documentation​

Astro API documentation is now hosted at https://www.astronomer.io/docs/api. In the new API documentation center, you can:

• Format and test API requests directly in your browser.
• Export requests to Python, Javascript, and curl.
• View weekly changelogs for the API.

Create Deployments with deprecated versions of Astro Runtime​

You can now use the Astro API to create Deployments with deprecated versions of Astro Runtime. Using deprecated Astro Runtime versions is sometimes necessary if you're migrating existing Airflow environments to Astro, or if you need to maintain deprecated environments for testing purposes.

Note that this feature is disabled by default. To use this feature, reach out to your account team and request for the feature to be enabled. See Run a deprecated Astro Runtime version for more information.

New GCP database instance types available​

You can now use the following node instance types for database instances in GCP clusters:

• XLarge Compute Optimized (24 CPU, 48 GiB MEM)
• XXLarge Compute Optimized (32 CPU, 64 GiB MEM)

See GCP Hybrid cluster settings for a list of all available database instance types.

Additional improvements​

• The Astro UI now includes Learning Bytes for features that are not yet configured within your Organization.
• The Astro UI now loads the status for DAG and task runs more quickly.

Bug fixes​

• When you change a worker type for an existing worker queue, the Astro UI no longer resets the worker queue's concurrency configurations.
• Fixed an issue where the Astro API did not return the correct value for IsHibernating when you queried Deployment information.

March 19, 2024​

New Azure regions available on Astro Hosted​

You can now create Hosted dedicated clusters in the following Azure regions:

• centralus
• westus3
• southcentralus

See Astro Hosted resource reference for more information.

Custom Deployment roles are now in Public Preview​

The ability to customize Deployment-level permissions with Deployment roles is now in Public Preview.

You can additionally use the new Workplace Accessor and Deployment Admin roles to define which users have access to specific Deployments in your Workspace. See more in User permissions reference and and Create and assign custom Deployment roles.

Export data from reporting dashboards using webhooks​

In addition to exporting report data with downloads or email, you can now export reporting data using webhooks. Use webhooks to send your reporting data to services such as Segment, Airtable, or Marketo.

Note that webhook exports are a Sigma feature in beta and might experience behavior changes.

Additional improvements​

• You can now use the new Credits tab in the Organization Billing page to see your credit balance.
• On Astro Hybrid, the maximum worker concurrency on a worker queue has increased from 64 to 256.

March 7, 2024​

Reporting dashboards are now in public preview​

Organization dashboards are now in Public Preview to use for examining key metrics across your Organization.

You can also export data from dashboards in the format of your choice. Exports can be triggered on a regular schedule or as an alert when specific criteria are met in your data. Export reporting data to share with other team members or to keep a record of key performance indicators. See Export reporting data for more information.

Customize Deployment-level permissions using Deployment roles​

Custom Deployment roles are a new way to define granular permissions for Astro users. For the first time, you can set a user's permissions at the Deployment level and define which specific parts of a Deployment they can access or modify. Use custom Deployment roles to have users collaborate in the same Workspace with only the minimum permissions they require. See Customize Deployment roles for more information.

New Deployment registry cache to improve resiliency​

Deployments now include a cache of Astronomer's image registry that stores the current Astro Runtime image for your Deployment. Because Deployments now always have access to their running image, image registry outages should no longer result in failed DAG runs.

Additional improvements​

• Removed nonfunctional network usage per Pod metrics from the Deployment Analytics page.
• The end-of-life date for Deployment API keys is June 1, 2024.
• The Cloud UI has been renamed to the Astro UI across all help text and documentation.
• Due to a minor change to Astronomer cluster architecture, you can no longer add custom tags to Hybrid clusters on AWS.

Bug fixes​

• Fixed an issue where CPU usage per Pod metrics did not render correctly in the Deployment Analytics page.
• Fixed an issue where the Astro UI didn't show all available Teams when selecting Teams to add to a Workspace.

February 27, 2024​

Use a custom service account to authorize Deployments to GCP​

You can now attach a custom GCP service account to your Deployment to grant the Deployment all of the service account's permissions to your cloud. Using a custom service account provides the greatest amount of flexibility for authorizing Deployments to your cloud. For example, you can use existing service accounts on new Deployments, or your can attach a single service account to multiple Deployments that should all have the same level of access to your cloud. For setup steps, see Authorize Deployments to your cloud.

Bug fixes​

• Fixed an issue where the Astro API failed to list Deployments after you deleted a hibernation override setting.

February 21, 2024​

New worker types​

Astro Hosted Deployments now support A120 and A160 workers, which include enough CPU and memory to handle the most resource-intensive tasks in your DAGs. See Astro Hosted resource reference for more information about each worker type.

New platform variables to improve Celery executor reliability​

Astro Deployments now have the following environment variables set by default. This is to ensure that the Celery executor doesn't freeze when it attempts to connect with a Pod in the Celery backend that has unexpectedly terminated.

AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_TIMEOUT=30
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_CONNECT_TIMEOUT=5
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_KEEPALIVE=True
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__RETRY_ON_TIMEOUT=True

For more information about each of these variables, see Platform variables

Ephemeral storage limit on schedulers​

Astro now limits that amount of ephemeral storage in a scheduler to 5Gi. If a scheduler attempts to use more than 5Gi of ephemeral storage, it will be terminated.

It is rare for schedulers to require more than 5Gi of ephemeral storage. If your schedulers start to terminate after this update, ensure the following:

• Your Deployment is not producing large amounts of temporary files without cleaning them up.
• If you're using the BingAds Python SDK, your Deployment uses version 13.0.14 or later. Earlier releases include a bug that generates large amounts of temporary files.

Bug fixes​

• You can no longer create Teams using the Astro API in an Organization that has SCIM provisioning enabled.
• Astro API responses have been standardized to always return cloud provider names in uppercase (for example, AWS).

February 13, 2024​

New Astro reporting dashboards show metrics for Deployments across your Organization​

The new Dashboards page includes a suite of dashboards that you can use to asses the performance of Deployments and DAGs across your entire Organization. Each dashboard focuses on a different aspect of your data pipelines to show you opportunities for cost and performance improvements. You can additionally configure Astro to send you alerts when a given metric reaches a specific threshold. See Organization Dashboards for summaries of each available dashboard.

Additional improvements​

• When you submit a support request from the Astro UI, you must now define an Active Engagement Period when you or a member of your team can engage with a member of Astronomer support.
• Workspace Members can now access the Clusters view in the Airflow UI for a Deployment.

Bug fixes​

• Fixed an issue where network connections between clusters could be disrupted occasionally.
• When you retrieve information about a Deployment through the Astro API, the API now returns an empty value for EnvironmentVariables if the Deployment has no environment variables.
• Deleting a Workspace through the Astro API now deletes all Astro Cloud IDE projects associated with the Workspace.
• Fixed an issue where you could not clear optional fields in a Deployment's configuration using the Astro API.

February 6, 2024​

Bug fixes​

• Fixed an issue where all Deployment task logs included the error Not exporting configs to configmap...

January 30, 2024​

New messages for Deployment health status​

Astro now automatically monitors Deployments and notifies you when a Deployment isn't running as expected, such as when it can't detect a heartbeat in a scheduler. These notifications, known as Deployment incidents, appear in your Deployment's health status in the Astro UI.

See Deployment health incidents to learn more about each available incident type and how to address them.

Additional improvements​

• You can now access Ask Astro from the Astro UI Help menu:

  

• User role titles are now consistently formatted across the Astro UI.

January 25, 2024​

Self-service VPC peering and route management for AWS​

You can now configure a network connection between Astro and an AWS VPC without contacting Astronomer support. Astro automatically handles creating a connection request and provides instructions for completing the setup yourself. After you create a VPC connection, you can configure routes whenever you need to connect to an additional service in your external VPC. See Create a private connection between Astro and AWS.

Additional improvements​

• There is no longer a minimum on the amount of CPU and memory that you can request for Kubernetes Pods.

Bug fixes​

• Fixed an issue in Astro Hybrid where refreshing the browser could occasionally reset a worker queue's worker type setting.

January 16, 2024​

Additional improvements​

• The Astro UI Deployment analytics page now shows CPU Usage Per Pod (%) and Memory Usage Per Pod (MB) as a percentage of your total available resources rather than the resources of a single worker Pod, such that these metrics will never show Deployment resources usage as exceeding 100%.
• The maximum value for worker queue Max # of workers has increased from 30 to 100.

January 9, 2024​

Additional improvements​

• When you create a Deployment, the Astro UI now shows the Airflow version of your Deployment instead of the equivalent Astro Runtime version.
• Enabled point-in-time restore (PITR) on Hybrid GCP clusters to improve resiliency to outages. Note that this might result in increased costs for cloud storage.

Bug fixes​

• Fixed an issue where you occasionally couldn't access the Airflow UI for a Deployment with the error "No healthy upstream".
• Fixed an issue where deleting a user with no Workspace membership from an Organization would affect how their Workspace membership appeared in other, unrelated Organizations.
• Fixed an issue where the Open in Airflow button on the DAG details page in the Astro UI did not open the Airflow UI as expected.

December 20, 2023​

Bug fixes​

• Fixed an issue where creating an alert for a DAG through the Astro API would apply the alert to the incorrect Deployment.
• Fixed an issue where Deployment worker Pods could crash when running the Kubernetes executor.
• Fixed an issue where Deployment API key expiration dates were not applied correctly if you configured multiple API keys at once.
• Fixed an issue where logging features could be disrupted if you set AZURE_CLIENT_ID as an environment variable. Note that this fix applies only to Astro Runtime 10 and later.

December 12, 2023​

Bug fixes​

• Fixed an issue where the Astro UI would produce an error if you updated an environment variable on an Astro Hybrid Deployment running the Kubernetes Executor.

December 6, 2023​

Additional improvements​

• The Astro Environment Manager is now generally available. This feature allows you to create and manage Airflow connections in the Astro UI.

Bug fixes​

• Fixed an issue where DAG code that appeared in the Airflow UI did not roll back when you rolled back a Deployment, even though the running code was successfully rolled back.
• Fixed an issue where you could not view billing information from the Astro UI when you installed Astro through the Azure Marketplace.
• Fixed an issue where the Astro UI would produce a console error when a user accessed their Workspace list.

November 30, 2023​

Support for Microsoft Entra Workload ID​

You can now use Microsoft Entra Workload ID to authorize Deployments to resources in Azure. Workload identity is a simple and secure way to authorize access external resources, as it doesn't require creating or storing long-term credentials. To set up Microsoft Entra Workload ID, see Authorize Deployments to cloud resources.

Bug fixes​

• Removed the ability to create Hybrid Azure clusters in eastasia because some workload identity features aren't supported in this region.

November 16, 2023​

Install Astro from the Azure Marketplace​

Astro is now available as an Azure Native ISV Service. If your team is considering Astro and you use Azure, Astronomer recommends installing Astro from the Azure Marketplace because:

• You can manage billing from the Azure Portal.
• Microsoft Entra ID is pre-configured for all Organizations.
• It's easier to create Astro resources and get started directly from Azure.

See Install Astro from the Astro marketplace for setup steps. To learn more about Astronomer's partnership with Microsoft, see Introducing Apache Airflow™ on Astro – an Azure Native ISV Service.

Create Airflow connections in the Astro UI and link them to Deployments​

You can now create Airflow connections in the Astro UI through the new Environment Manager menu. The Environment Manager lets you create Airflow connections directly in the Astro UI and stores all connections in an Astro-managed secrets backend. You can then share connections between Deployments and set default connections so that your team members always have access to external resources when they create new Deployments. See Create Airflow connections in the Astro UI.

Note that this feature is currently available only for Deployments running the Celery executor.

Trigger a DAG from an Astro alert​

You can now configure Astro alerts to trigger any DAG in your Workspace through the Airflow REST API. You can configure the triggered DAG to complete any action, such as sending an alert through a custom communication channel or writing data about the incident to a table.

New Azure regions available on Astro Hosted​

You can now create Hosted dedicated clusters in the following Azure regions:

• eastus
• canadacentral
• uksouth
• brazilsouth
• centralindia
• francecentral
• japaneast

See Astro Hosted resource reference for more information.

November 7, 2023​

Roll back Deployments to previous versions of your code​

Astro now maintains snapshots of your past deploys, including your Deployment image and DAG code, for the previous three months. If you need to quickly revert a Deployment back to a working version of your code, you can roll back to a past deploy from the Deploy History page in the Astro UI.

Deploy rollbacks are a powerful safety mechanism to ensure that your production pipelines continue to run when something unexpected happens after a deploy. See Roll back to a past deploy for more information and configuration steps.

Bug fixes​

• Fixed an issue where, you could inadvertently open the support request window if you opened a DAG that included "Support" in its name from the DAGs view. As a result of this change, the support request window URL has been updated from https://cloud.astronomer.io/support to https://cloud.astronomer.io/open-support-request.
• Fixed an issue where the Deployment configuration menu in the Astro UI didn't always show your Deployment's current configuration.
• Fixed an issue where you could not create a Deployment with some stable Runtime versions using the Astro Platform API.

October 31, 2023​

Deployment API keys are now deprecated​

Deployment API keys have been officially deprecated in favor of Deployment API tokens. This means:

• You can't create new Deployment API keys.
• If a Deployment has no configured Deployment API keys, the API keys tab will not appear.
• You can continue using existing Deployment API keys until a future end-of-support date.

Additional improvements​

• Workspace Operators can now create, update, and delete Deployment API tokens.

Bug fixes​

• Fixed an issue where a Deployment's Updated By field was not updated if you transferred the Deployment.

October 24, 2023​

New Azure regions available on Astro Hosted​

You can now create Deployments in standard clusters in the following Azure regions:

• eastus2
• westus2
• westeurope

See Astro Hosted resource reference for more information.

Bug fixes​

• Fixed an issue where Deployment API tokens weren't deleted after their associated Deployment was deleted.
• Fixed an issue where you could not create a Deployment with some available Runtime versions using the Astro Platform API.

October 17, 2023​

Additional improvements​

• You can now view deploy history for both Hosted and Hybrid Astro Deployments in the Astro UI. For more information, see View deploy history.

Bug fixes​

• Modified behavior for Astro Hosted so that KubernetesExecutor and KubernetesPodOperator pods running in-cluster have equivalent resource requests and limits. If you do not configure them to have equivalent resource requests and limits, Astro modifies them to the become the limits. Previously, the DAG deploy would fail if resources did not equal limits.

October 10, 2023​

Deployment API tokens​

Deployment API tokens are now generally available and replace Deployment API keys as the most secure and customizable way to programmatically update Deployments. This includes using them to deploy code and update environment variables.

See Deployment API tokens to learn how to create and manage Deployment API tokens.

New Edit Deployments in the Astro UI​

Editing Deployments in the Astro UI has a new, consolidated flow. All of the configuration options are now editable in a single form, similar to Deployment creation, instead of spread across multiple forms. See Deployment Settings for a detailed description of how to create, update, and configure your Deployment options.

October 3, 2023​

Additional Improvements​

• Added a DAG Success alert so you can now set up an alert for successful completion events. See how to set up Astro alerts.

Bug Fixes​

• Fixed a problem in the Astro UI where a warning about Deployment Health was displayed when a Workspace had zero Deployments.

September 26, 2023​

Introducing the Astro API​

You can now use the Astro API to create applications and scripts to programmatically interact with Astro. The Astro API is a standard REST API that includes endpoints for interacting with all key resources and components on Astro.

Using the Astro API, you can create robust and secure applications for managing Deployment resources, updating user permissions, and performing many other key Astro operations. To make your first API call, see Get started with the Astro API.

September 19, 2023​

Manage Deployments programmatically using Deployment API tokens​

Deployment API tokens replace Deployment API keys as the most secure and customizable way to manage Deployments programmatically. You can use Deployment API tokens to perform all of the same actions as a Deployment API key, including:

• Pushing code to a Deployment.
• Updating a Deployment's environment variables.
• Making requests to update your Deployment's Airflow environment using the Airflow REST API.

Unlike Deployment API keys, you can set an expiration date for Deployment API tokens and rotate them to better manage access to your Deployment. See Deployment API tokens to learn how to create and manage Deployment API tokens.

Additional improvements​

• When you create a new Deployment, the Astro UI now presents new options and suggestions for running your first DAG.
• You can now retrieve a Workspace's ID from the Astro UI. To find a Workspace's ID, open the Workspace in the Astro UI and go to Workspace Settings > General.

September 12, 2023​

Per-Deployment IAM workload identities on AWS​

Astro Hybrid clusters on AWS now support per-Deployment IAM workload identities, meaning that you can now limit your trust policies to authorize only specific Deployments to your cloud resources.

{ "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DeleteLoadBalancer" }

To migrate from using cluster workload identities to Deployment workload identities:

1. In the AWS Management Console, go to the Identity and Access Management (IAM) dashboard. Identify all of your trust policies that specify your cluster workload identity. They should look similar to the following trust policy:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": [
                       "arn:aws:iam::<dataplane-AWS-account-ID>:role/AirflowS3Logs-<cluster-ID>"
                   ]
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }
   

2. For each trust policy, add the workload identities for any Deployments that you want to access the related resource. To locate your Deployment workload identity, open the Deployment in the Astro UI and copy the Workload Identity from the Details page. Your trust policy should now look like the following:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": [
                       "arn:aws:iam::123456789876:role/AirflowS3Logs-cl6zcnlc641hr0voibivf21jh",
                       "arn:aws:iam::<dataplane-AWS-account-ID>:role/astro-<namespace>"
                   ]
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }
   

3. For each Deployment that you specified in your trust policies, open the Deployment in the Astro UI and click Details, then click Edit Details. In the Workload Identity section, select the new Deployment identity from the dropdown list, then click Update. To avoid disruption to tasks, do not complete this step until you have added the Deployment workload identity to all of the trust policies it needs for access.

4. Upgrade to the latest Astro CLI release, which includes support for Per Deployment IAM Workload Identity.

5. After you've tested the policies with your Deployment workload identities, remove the cluster workload identity from your trust policies

Bug fixes​

• Fixed an issue where Billing Admins could view task usage on the Usage page only for Workspaces that they belonged to. Now, Billing Admins can view usage for all Workspaces regardless of their Workspace role.

Additional improvements​

• The Astro UI Usage page now shows task usage for deleted Deployments. If you're an Astro Hybrid Billing Admin, this means that task usage metrics now better reflect your billable usage.
• When you create a Deployment through the Astro UI and choose an Astro Runtime version, you can now select only the most recent supported patch for each major version of Astro Runtime.
• You can now filter task logs by log level or source from the DAGs page in the Astro UI.

September 6, 2023​

View deploy history in the Astro UI​

When you view a Deployment in the Astro UI, you can now open the Deploy History tab to view a table of all code deploys. The table shows who made deploys, when they made the deploys, and what Astro Runtime image they used for the deploy.

You can also now use the Astro CLI to specify an optional description for your deploys using the --description flag. Deploy descriptions appear in the Deploy History table and are useful for telling other Workspace members why you made a deploy or what changes it contains. For more information, see View deploy history.

View rendered data figures in the Astro Cloud IDE​

Python cells that generate figures using Matplotlib, Plotly, or any libraries that extend these tools now show the figures they render in the Astro Cloud IDE. When you run a Python cell, any figures it generates appear in a new Figures tab.

August 29, 2023​

Changes to Workspace user roles​

To increase granularity and better serve each user persona on Astro, Workspace roles have been updated with new names and permissions:

• The Workspace Author role is a new role for users who primarily write and deploy DAGs. Users with this role can push code changes, but they can't update Deployment or Airflow settings such as Airflow variables, Astro environment variables, or connections.
• The Workspace Admin role has been renamed to Workspace Owner. Users with this role are responsible for administrating membership to the Workspace.
• The Workspace Editor role has been renamed to Workspace Operator. In addition to pushing code changes, Workspace Editors can now also edit Airflow objects such as variables, connections, and XComs. Users with this role are responsible for managing the environments that DAGs run in.
• The Workspace Viewer role has been renamed to Workspace Member. Users with this role only need viewing permissions for a Deployment and don't have permissions to make any code or configuration changes.

For more information about these role changes, see User permissions reference and Enhanced Astro Workspace Roles for more granular permissions

Additional improvements​

• The lifespan of the personal user access token you can retrieve from cloud.astronomer.io/token has been reduced from 24 hours to 1 hour.
• The DAGs view of the Astro UI now shows your configured dependency edge labels in the graph view.
• The Astro UI now shows more detailed instructions for deploying code when you create a new Deployment.
• The Deployment Analytics page in the Astro UI has been renamed to Overview.

Bug fixes​

• Fixed an issue where Deployments using the Kubernetes executor could not run DAGs with lower resource requests than the Default Pod Size. Minimum requests are now hard-coded and decoupled from default requests.

August 21, 2023​

Additional improvements​

• You can now configure task log forwarding to Datadog at the Deployment level.
• In the DAGs view of the Astro UI, you can now double click a task run node in the graph view to view the task run's logs and mapped tasks.
• The A50 worker type has been renamed to A60 to make it consistent in scale with other worker types.
• The max possible CPU quota and Memory quota for a Deployment running in a Hosted dedicated cluster has increased to 1600 vCPU/ 3200 GiB respectively.

August 15, 2023​

Additional improvements​

• You can now see how many Astro alerts you've configured for a DAG in the DAGs page of the Astro UI.

Bug fixes​

• Fixed an issue where you couldn't run Astro Cloud IDE pipelines that included a Markdown cell.
• Fixed an issue where an Organization's SSO bypass link was formatted incorrectly in the Astro UI.

August 8, 2023​

New Hosted worker type​

You can now configure Deployments with the A50 machine type, which has 12 vCPU and 24 GiB. See Astro hosted resource reference.

Additional improvements​

• When you create a new Astro Cloud IDE project, you can now specify whether you want the project to include an example pipeline.
• You can now access Organization-level settings in the Astro UI only through the Organization Settings link. Additionally, some Organization settings have been moved to the top level of navigation so that there is no longer a Settings menu.
• You can now use commas, apostrophes, and ampersands in Workspace and Organization names.
• The Workspace list view in the Astro UI has been redesigned so that Organization Owners can now edit and delete Workspaces directly from the list.

Bug fixes​

• Fixed an issue where the Astro UI showed incorrect CPU and memory limits in bar charts on the Deployments list and Details page.

August 1, 2023​

Hosted Deployments have DAG-only deploys enabled by default​

New Astro Hosted Deployments now have DAG-only deploys enabled by default. When DAG-only deploys are enabled, some workflows for your Deployment, including image-based deploys, are different compared to when DAG-only deploys are disabled. For more information about how code and image deploys work when DAG-only deploys are enabled, see What happens during a project deploy. To disable DAG-only deploys, see Enable/ disable DAG-only deploys on a Deployment.

Teams now have Organization-level roles​

All Teams on Astro now have an Organization role. Existing Teams have been given the Organization Member role, which doesn't result in any additional automatic permissions.

Coupled with SCIM user groups, you can now manage your Organization Owners and Billing Admins from your identity provider. See Manage teams for more information.

Additional improvements​

• The Astro UI now shows how many Workspaces, DAGs, clusters, and Astro Cloud IDE projects you have in the left sidebar.
• You can now create Deployments in standard clusters hosted in europe-west2 on GCP and eu-central-1 on AWS.
• The default metadata database instance type for new Deployments on GCP clusters in Astro Hybrid has been reduced to Small General Purpose with 2 vCPUs and 8GiB. See GCP Hybrid cluster settings.

Bug fixes​

• Because some regions don't support specific machine types that Astro Hosted uses, you can no longer create Hosted dedicated clusters in the following AWS regions:

  • af-south-1
  • ap-east-1
  • ap-northeast-2
  • ap-northeast-3
  • ca-central-1
  • eu-south-1
  • eu-west-3
  • me-south-1

July 25, 2023​

Additional improvements​

• The templates for Astro alert messages have been updated to include more information about the Deployment that the alert was triggered in, including a link to the DAG that triggered the alert.

Bug fixes​

• Fixed an issue where Azure AD single sign-on (SSO) connections were incorrectly labeled as SAML connections in the Astro UI.

July 18, 2023​

Configure default Pod sizes​

You can now configure the default minimum CPU and memory for tasks that you run with the Kubernetes executor or KubernetesPodOperator. If you don't specify CPU or memory in a task definition, Astro runs the task in a Pod that uses your default resource configurations. Configure default minimum resources to ensure that tasks always have enough CPU and memory to run successfully. See Deployment settings.

New regions available on Astro Hosted​

You can now create Hosted dedicated clusters in the following regions:

• AWS

  • af-south-1 - Africa (Cape Town)
  • ap-east-1 - Asia Pacific (Hong Kong)
  • ap-northeast-1 - Asia Pacific (Tokyo)
  • ap-northeast-2 - Asia Pacific (Seoul)
  • ap-northeast-3 - Asia Pacific (Osaka)
  • ap-southeast-1 - Asia Pacific (Singapore)
  • ap-southeast-2 - Asia Pacific (Sydney)
  • ap-south-1 - Asia Pacific (Mumbai)
  • ca-central-1 - Canada (Central)
  • eu-central-1 - Europe (Frankfurt)
  • eu-south-1 - Europe (Milan)
  • eu-west-1 - Europe (Ireland)
  • eu-west-2 - Europe (London)
  • eu-west-3 - Europe (Paris)
  • me-south-1 - Middle East (Bahrain)
  • sa-east-1 - South America (São Paulo)
  • us-east-1 - US East (N. Virginia)
  • us-east-2 - US East (Ohio)
  • us-west-1 - US West (N. California)
  • us-west-2 - US West (Oregon)

• GCP

  • asia-east1 - Taiwan, Asia
  • asia-northeast1 - Tokyo, Asia
  • asia-northeast2 - Osaka, Asia
  • asia-northeast3 - Seoul, Asia
  • asia-south1 - Mumbai, Asia
  • asia-south2 - Delhi, Asia
  • asia-southeast1 - Singapore, Asia
  • asia-southeast2 - Jakarta, Asia
  • australia-southeast1 - Sydney, Australia
  • australia-southeast2 - Melbourne, Australia
  • europe-central2 - Warsaw, Europe
  • europe-north1 - Finland, Europe
  • europe-southwest1 - Madrid, Europe
  • europe-west1 - Belgium, Europe
  • europe-west2 - England, Europe
  • europe-west3 - Frankfurt, Europe
  • europe-west4 - Netherlands, Europe
  • europe-west6 - Zurich, Europe
  • europe-west8 - Milan, Europe
  • europe-west9 - Paris, Europe
  • northamerica-northeast1 - Montreal, North America
  • northamerica-northeast2 - Toronto, North America
  • southamerica-east1 - Sau Paolo, South America
  • southamerica-west1 - Santiago, South America
  • us-central1 - Iowa, North America
  • us-east1 - South Carolina, North America
  • us-east4 - Virginia, North America
  • us-east5 - Columbus, North America
  • us-south1 - Dallas, North America
  • us-west1 - Oregon, North America
  • us-west2 - Los Angeles, North America
  • us-west3 - Salt Lake City, North America
  • us-west4 - Nevada, North America

See Astro Hosted resource reference for all available configurations.

Bug fixes​

• You can no longer create Hybrid GCP clusters in eu-north-1.

July 11, 2023​

Send Astro alerts to email​

You can now send Astro alerts to multiple email addresses. Sending Astro alerts to email requires no configuration outside of Astro, which makes it a quick option to improve your alerting infrastructure. See Configure Astro alerts for setup steps.

Configure SCIM provisioning for Azure AD​

If your Organization uses Azure for single sign-on (SSO), you can now set up SCIM provisioning for Astro. SCIM provisioning simplifies user management by allowing you to add and remove Astro users from Okta based on your existing user groups. See Set up SCIM provisioning for more information.

Additional improvements​

• On Astro Hosted deployments, the astronomer_monitoring_dag has been paused for all image-based Deployments and removed entirely from all Deployments with DAG deploys enabled. It has been replaced with an implementation that allows workers on Deployments to fully scale to 0.

July 5, 2023​

Configure SCIM provisioning for Okta​

If your Organization uses Okta for single sign-on (SSO), you can now set up SCIM provisioning for Astro. SCIM provisioning simplifies user management by allowing you to add and remove Astro users from Okta based on your existing user groups. See Set up SCIM provisioning for more information.

See pricing estimate when creating a Deployment​

The Deployment creation page in the Astro UI has been reorganized to make it easier to focus on specific configurations for your Deployment. Each configuration is now collapsible and includes guidance for different environment sizes. Additionally, the page now shows cost estimates for a Deployment before you create it.

Additional improvements​

• You can now configure Deployments with the A40 machine type, which has 8 vCPU and 16 GiB. See Astro hosted resource reference.
• You can now access your Organization settings from a Workspace by clicking the name of your Organization/ Workspace.
• On Astro Hybrid, the default Azure DB instance is now Standard D2ds_v4.
• The Deployment creation screen for Astro Hybrid has received several updates that were previously only available on Astro Hosted.

Bug fixes​

• Fixed an issue where DAGs that used the KubernetesPodOperator and had tasks with in_cluster=False could not be parsed.

June 27, 2023​

Support for dedicated clusters on Azure​

You can now create a dedicated cluster in the following Azure regions:

• australiaeast
• eastus2
• northeurope
• westeurope
• uswest2

See Astro Hosted resource reference for more information.

Additional improvements​

• The Astro UI now shows how many Workspaces each Team belongs to in Settings > Access Management > Teams.
• You can now create dedicated clusters in us-west1 on GCP.

June 20, 2023​

Additional improvements​

• You can now add a new Astro user to Workspaces before the user has accepted their invite.
• If you have Organization Owner permissions, you can now add a user to a Workspace even if the user hasn't been added to your Organization. Users added to Workspaces this way are automatically added to your Organization as an Organization Member.
• The Astro UI now shows your Team IDs in Settings > Access Management > Teams. Use Team IDs to add Teams to Workspaces using the Astro CLI.
• A Team's Updated At and Updated By values are now updated when you change the Team's permissions in a Workspace or Organization.

Bug fixes​

• Fixed an issue where a Workspace descriptions were incorrectly required when creating a new Workspace through the Astro CLI.

June 13, 2023​

Manage billing and track usage for Astro Hosted​

Use the new Billing page in the Astro UI to see both high-level and detailed metrics about your spend in Astro Hosted. You can also use this page to configure your billing details and view invoices. See Manage billing for more details.

New cell type for using Airflow operators in the Astro Cloud IDE​

You can now use any Airflow operator available on the Astronomer Registry in your Astro Cloud IDE pipeline. Operator cells apply formatting and checks for parameter inputs, making it easy to configure operators as part of your pipeline. See Use Airflow operators in the Astro Cloud IDE.

Additionally, you can configure custom cells to use your team's custom operators in a pipeline. See Create custom operator cells.

IMDSv2 is now enforced on AWS clusters​

Astronomer now enforces IMDSv2 on all AWS clusters. Any requests for resources on your clusters are now session-based and muse include a token in the request body.

Additional improvements​

• Trial Deployments now have DAG-only deploys enabled by default.
• The Astro UI now shows your Organization Short Name and Astro SAML Connection Name in the Astro UI.
• You can now view mapped tasks from the DAGs page in the Astro UI.

Bug fixes​

• Fixed an issue where worker node pools in Hosted dedicated clusters on Azure were not being updated correctly.
• Fixed an issue the Astro UI would reset a Deployment's Min Worker Count from 0 to 1 after you edited the Deployment in any way.

June 6, 2023​

Track user actions and ensure compliance with audit logs​

You can now export audit logs from the Astro UI to view all actions taken in your Organization over a given time period. See Export audit logs for setup steps.

Additional improvements​

• You can now configure Hybrid GCP clusters with additional Memory Optimized and Compute Optimized Cloud SQL instance types. See Supported Cloud SQL instance types.

May 30, 2023​

Manage permissions for groups of users with Teams​

Configure Teams from the Astro UI to manage the permissions for many users across Workspaces from a single page. Teams are a group of users in an Organization that you grant the same Workspace permissions, without needing to define them individually.

See Make a Team for setup steps.

Bug fixes​

• In Astro Hosted, an irrelevant AWS external ID info page has been removed from the Astro UI.
• Fixed an issue where DAG-only deploys could be unreliable due to the deploy process not requesting enough resources in the cluster.

May 23, 2023​

Introducing Astro Hosted and Hybrid​

Astro Hosted is a new way to run Airflow on Astronomer's cloud. On Astro Hosted, Airflow environments are managed and hosted entirely by Astronomer, enabling you to shift your focus from infrastructure to data.

For more information about how Astro Hosted works, see the Architecture overview.

If you're already an Astro user and your Deployments run in your company's own cloud, you're using Astro Hybrid. This version of Astro was formerly known as Astro - Bring Your Own Cloud.

To see whether you're an Astro Hybrid user, open your Organization in the Astro UI and go to Settings > General. Your version of Astro is listed under Product Type.

See Documentation refactor for Astro Hybrid to learn how the documentation has changed for current Astro Hybrid users.

Configure default Kubernetes Pods on Astro Hosted​

One of the biggest risks of running the Kubernetes executor or KubernetesPodOperator is that your tasks can accidentally request more resources than expected, which can drive up costs. To limit this risk, you can now configure default and maximum Pod resources from the Astro UI. If a task tries to request Pod resources that are more than your configured limits, the task fails.

See Configure Kubernetes Pod resources for setup steps.

Documentation refactor for Astro Hybrid​

The following updates have been made to documentation to accommodate new Astro Hosted information:

• Astro Hybrid documentation now has a dedicated menu under "Administration" that contains all docs related to Hybrid installation and cluster management. See Astro Hybrid overview.
• All other docs now assume Astro Hosted by default. If a feature is functionally different between Hosted and Hybrid, the documentation for that feature will include a note about how that setup differs for Hybrid. Look for the blue Alternative Astro Hybrid setup notes throughout documentation.

May 16, 2023​

Automate Organization management with Organization API tokens​

You can now create Organization API tokens to automate key actions across your Organization and all of the Workspaces in it. You can customize the role and expiration date of the token to give it the minimum required permissions for the task it completes. Some common actions that you can automate with Organization API token are:

• Creating Workspaces.
• Inviting users to an Organization or Workspace.
• Creating and updating Deployments using a Deployment file.
• Exporting audit logs.
• Gathering metadata about Deployments using the Airflow REST API.
• Completing any of the actions you can complete with a Workspace API token or Deployment API key across all Deployments in your Organization.

See Manage Organization API tokens for more information.

May 2, 2023​

Receive Astro alerts on Slack or PagerDuty​

Astro alerts are a new way to be notified when your DAGs aren't running as expected. Unlike Airflow callbacks and SLAs, Astro alerts require no changes to DAG code and integrate with Slack and PagerDuty.

You can set an alert on any DAG to be notified when the DAG fails or when a task takes longer to run than expected. See Astro alerts for configuration steps.

Bug fixes​

• Fixed an issue where SSO configurations made through Astronomer support could be overridden by updating the SSO configuration through the Astro UI.

April 26, 2023​

Improved log viewing in the Astro UI​

The Deployment Logs page in the Astro UI now shows logs for your Deployment's workers, schedulers, triggerers, and webserver. Additionally, you can now view up to the last 10,000 logs emitted by your Deployment from the Astro UI.

To make it easier to parse this larger log volume, the Logs page now lets you filter by log type, date, and keyword. See View logs for more information.

April 18, 2023​

Self-service configuration for single sign-on (SSO) connections​

You can now configure SSO connections directly from the Astro UI without assistance from Astronomer support. Use the Authentication page to configure different authentication environments for your Organization by creating and managing multiple SSO connections and domains.

To review the new process for creating SSO connections, see Set up authentication and SSO. To create new managed domains to map to your SSO connections, see Manage domains.

April 11, 2023​

Additional improvements​

• The node type for running Airflow system components on GCP clusters has been reduced from n2-standard-4 to e2-standard-4.
• To optimize infrastructure costs for running the Kubernetes executor, Kubernetes executor worker Pods from different Deployments can now run on the same worker node. This occurs only when the Deployments are hosted in the same cluster and use the same worker node instance type.

April 4, 2023​

Preview Deployments​

You can now create preview Deployments from feature branches in your Git repository. Use a preview Deployment template or GitHub Actions template to configure your Astro pipelines to:

• Create the preview Deployment when you create a new branch.
• Deploy code changes to Astro when you make updates in the branch.
• Delete the preview Deployment when you delete the branch.
• Deploy your changes to your base Deployment after you merge your changes into your main branch.

Additional improvements​

• Added the ability to enforce CI/CD deploys. You can now configure your Deployment to only accept code deploys if they are triggered by a Deployment API key or Workspace token.
• When you create a new cell in the Astro Cloud IDE, the editor auto-scrolls to your new cell and selects it.

Bug fixes​

• Fixed a bug where the UI passed the wrong cluster type.
• Fixed an issue where the Deployment status shows as 'deploying' when KPOs are running.

March 28, 2023​

New GCP node instance types available​

You can now use the following node instance types for worker nodes in GCP clusters:

• e2-standard-32
• e2-highcpu-32
• n2-standard-32
• n2-standard-48
• n2-standard-64
• n2-highmem-32
• n2-highmem-48
• n2-highmem-64
• n2-highcpu-32
• n2-highcpu-48
• n2-highcpu-64

For a list of all instance types available for GCP, see Supported worker node pool instance types.

Additional improvements​

• You can now use db.m6g and db.r6g RDS instance types on AWS clusters.
• The default RDS instance type for new AWS clusters has been reduced from db.r5.large to db.m6g.large
• The default CIDR range for new AWS clusters has been reduced from /19 to /20.
• You can now submit a Request type in the Astro UI support form. When you choose a request type, the form updates to help you submit the most relevant information for your support request.
• You can no longer delete a Workspace if there are any Astro Cloud IDE projects still in the Workspace.
• Organization role permissions have changed so that only Organization Owners can create Workspaces.

Bug fixes​

• Fixed an issue where you could set a Deployment's scheduler resources to less than 5 AU.

March 21, 2023​

Automate Workspace and Deployment actions using Workspace API tokens​

Use Workspace API tokens to automate Workspace actions, such as adding users to a Workspace and creating new Deployments, or for processes that a Deployment API key can automate. You can customize the role and expiration date of the token to give it the minimum required permissions for the task it completes.

To create and use Workspace API tokens, see Workspace API tokens.

Additional improvements​

• In the Astro Cloud IDE, you can now specify the output table for a Warehouse SQL cell using both literal and Python expressions. See Create a SQL cell.

• Port 80 is no longer used for certificate management on the data plane.

• To switch Organizations in the Astro UI, you now use the Switch Organization button next to your Organization's name.

  

March 15, 2023​

Run the Kubernetes executor in Astro​

You can now configure your Deployments to use the Kubernetes executor for executing tasks. Using the Kubernetes executor, you can:

• Ensure that tasks running longer than 24 hours are not interrupted when your team deploys code.
• Run tasks with different version dependencies in the same Astro project.
• Request specific amounts of CPU and memory for individual tasks.
• Automatically down your resources when no tasks are running.

The Kubernetes executor runs each task in its own Kubernetes Pod instead of in shared Celery workers. Astronomer fully manages the infrastructure required to run the executor and automatically spins Pods up and down for each of your task runs. This executor is a good fit for teams that want fine-grained control over the execution environment for each of their tasks.

To learn whether the Kubernetes executor works for your use case, see Choose an executor. To configure the Kubernetes executor for a task or Deployment, see Configure the Kubernetes executor.

Simplified Organization management in the Astro UI​

The Astro UI has been redesigned so that Organization settings tabs are now available in the left menu. Use this new menu to switch between pages as you can for Workspace settings.

While most tabs were migrated directly to the left menu with the same name, some pages have been renamed and moved:

• Formerly located in Overview, your Workspace list is now available in Workspaces.
• Formerly located in the People tab, Organization user management settings are now in Settings > Access Management.
• Formerly located in the Settings tab, general Organization settings are now in Settings > General.

New Astro Cloud IDE integration with GitLab​

You can now configure a GitLab repository in your Astro Cloud IDE project. Configuring a GitLab repository allows you to commit your pipelines and deploy them to Astro directly from the Astro Cloud IDE. See Deploy a project from a Git repository to Astro.

Additional improvements​

• Clusters on an Astro - Hosted installation no longer retain Airflow logs which are older than 90 days.
• The Data Plane System node pool instance type on GCP clusters has been reduced from n2-standard-4 to n2-standard-2.

March 7, 2023​

Get expert advice on Astro and Airflow in office hours​

Office hours are a new way for Astro customers to meet with the Astronomer Data Engineering team. In an office hour meeting, you can ask questions, make feature requests, or get expert advice for your data pipelines.

You can now schedule a 30-minute office hour meeting in the Help menu next to your user profile in the Astro UI.

For more information, see Book office hours in the Astro UI.

Additional improvements​

• The node pool instance type used for Astro system components on GCP clusters has been reduced from n2-standard-4 to n2-standard-2.
• DAGs generated by the Astro Cloud IDE now use UTC instead of your current timezone as the default timezone for scheduling DAG runs.

Bug fixes​

• Fixed an issue in the Astro Cloud IDE where you could not update a pipeline that was configured with an invalid cyclic dependency chain.
• Fixed an issue where deploying an Astro project with a custom Docker image tag resulted in the Deployment always having the Deploying status in the Astro UI.
• Fixed an issue where worker Pods on Azure clusters were sometimes unable to scale because there was no prioritization for starting up essential scheduling Pods.

March 1, 2023​

Astro no longer requires administrator access on AWS​

Astro no longer requires administrator permissions for its dedicated AWS account. Instead, Astro now assumes a cross-account IAM role with the minimum necessary permissions for running and managing clusters. See Install Astro on AWS for more information.

IdP-initiated logins through the Okta dashboard​

If your Organization uses Okta as your Astro identity provider, you can now log in to Astro directly from your Okta Apps dashboard. If you've been authenticated by Okta, you no longer need to be authenticated by Astro when you access it through your dashboard.

Additional improvements​

• Ingress to the Kubernetes API on Google Cloud Platform (GCP) and Azure clusters is now limited to Astro control plane IPs. This change will be implemented on all clusters in the coming weeks.

Bug fixes​

• To protect the functionality of Astro monitoring services, you can no longer override the values of the following environment variables:

  • AIRFLOW__METRICS__STATSD_ON

  • AIRFLOW__METRICS__STATSD_HOST

  • AIRFLOW__METRICS__STATSD_PORT

  • AIRFLOW__METRICS__STATSD_ALLOW_LIST

  • AIRFLOW__METRICS__STATSD_STATSD_CUSTOM_CLIENT_PATH

  • AIRFLOW__METRICS__STATSD_PREFIX

    You can still set new values for these variables, but the values will be automatically overwritten in the Astro data plane. See Platform variables.

• Fixed an issue where a user could provision multiple accounts when their login email address included differently cased characters.

February 21, 2023​

New identity-first authentication model​

Astro has migrated to an identity-first authentication model. Users now authenticate to the Astro platform instead of individual Organizations, and Organizations can set permissions for how users can modify and access resources. This model prioritizes identity verification and enforces authentication policies for user email domains.

For all users logging in to Astro, this migration has the following effects:

• Instead of being redirected to separate login pages for each Organization, all Astro users log in through a universal login page.
• Users belonging to multiple Organizations no longer have to log in again when switching Organizations.
• Users no longer need to enter their email on a separate page before they log in to the Astro UI.
• If your Organization enforces single sign-on (SSO), users can now authenticate to Astro with a username and password when your email domain doesn't enforce SSO.

For Organization Owners, this migration has the following additional effects:

• You can now use an SSO bypass link to log in to Astro if your SSO connection is disrupted.
• Your Organization now has a list of owned email domains, and any users logging into Astro with one of those domains will be redirected to your configured identity provider.

To configure authentication behavior, see Configure SSO.

New Hosted regions available​

You can now create clusters in the following regions on an Astro - Hosted installation.

• AWS

  • ap-northeast-1
  • ap-southeast-2
  • eu-central-1
  • eu-west-1
  • us-east-1
  • us-west-2

• Google Cloud

  • asia-northeast1
  • australia-southeast1
  • europe-west1
  • europe-west2
  • us-central1
  • us-east4

• Microsoft Azure

  • australiaeast
  • japaneast
  • northeurope
  • westeurope
  • eastus2
  • westus2

Additional improvements​

The default CIDR ranges for new GCP clusters have been reduced. The following are the new CIDR ranges:

• Subnet CIDR: 172.20.0.0/22
• Pod CIDR: 172.21.0.0/19
• Service Address CIDR: 172.22.0.0/22
• Service VPC Peering: 172.23.0.0/20

Bug fixes​

In the Astro UI, when using Compare on the Lineage Graph page, you can now compare shorter run lengths.

February 14, 2023​

Authorize Workspaces to clusters​

You can now keep teams and projects isolated by authorizing Workspaces to specific clusters. Use this feature to better manage cloud resources by ensuring that only authorized Deployments are running on specific clusters.

New Deployment health statuses and information in the Astro UI​

The Astro UI now includes three additional Deployment health statuses that you might see when creating or pushing code to a Deployment.

• The Creating status indicates that Astro is still provisioning the resources for the Deployment.
• The Deploying status indicates that a code deploy is in progress. Hover over the status indicator to view specific information about the deploy, including whether it was an image deploy or a DAG-only deploy.
• The Unknown status indicates that Deployment status can't be determined.

Additionally, the Deployment information page in the Astro UI now includes fields for Docker Image and DAG Bundle Version that show unique timestamps and tags based on your latest code deploy. Use this information as the source of truth for which version of your code is currently running on the Deployment.

View OpenLineage facets for lineage job runs​

OpenLineage facets are JSON objects that provide additional context about a given job run. By default, a job run includes facets that show what kind of job was completed, whether the job run was successful, and who owns the job.

You can now view all available facets for a job run, including custom facets, by opening the job run's Lineage Graph and then selecting the Info tab. You can check the status of your facets, including whether they are correctly formatted, so that you can resolve potential issues in your data pipelines. See View metrics for a specific run or dataset.

Additional improvements​

• You can now create AWS clusters in ap-northeast-1 and ap-southeast-2 on an Astro - Hosted installation.
• You can now create GCP clusters in australia-southeast1 on an Astro - Hosted installation.

Security fixes​

• Fixed CVE-2023-0286.

February 7, 2023​

Additional improvements​

• The instructions on the welcome page for new Astro users who are not yet part of an Organization have been improved to make getting started easier.

Bug fixes​

• Removed nonfunctioning date filtering functionality from the Lineage UI.
• Fixed an issue where triggering an image deploy from an older version of the Astro CLI could unintentionally turn off DAG deploys on a Deployment.

January 31, 2023​

Bug fixes​

• When you select Mark Success or Clear for Deployment task actions in the Airflow UI, you are now correctly redirected to the DAG Tree view instead of the DAGs homepage.
• Fixed CVE-2022-41721.

January 24, 2023​

New Workspace Home page​

When you select a Workspace in the Astro UI, the Home page now appears first. On this page, you can:

• Check the status of your Deployments.
• Quickly access your most recently viewed Deployments and Cloud IDE projects.
• View release notes for all Astro products.

See Introducing Astro’s New Workspace Homepage for more information.

Additional improvements​

• Ingress to the Airflow UI and API on Astro clusters is now limited to control plane IPs. This change will be implemented on all clusters in the coming weeks.
• You can now request custom tags for your AWS clusters by submitting a support request to Astronomer support. You can view your cluster tags in the Astro UI by selecting Clusters, selecting a cluster, and then clicking the Details tab.
• You can now create new clusters in France Central for Bring Your Own Cloud installations of Astro on Azure.
• Improved the speed of DAGs appearing in the Airflow after completing a DAG-only deploy.

Bug fixes​

• Fixed CVE-2022-48195.

January 18, 2023​

Bug fixes​

• Fixed an issue with Google Cloud Platform (GCP) clusters where the metadata database for a Deployment could persist after the Deployment was deleted.

January 10, 2023​

New Astro Cloud IDE cell types​

To simplify the creation of new tasks, the following new cell types are now available in the Astro Cloud IDE:

• SQL: Run a SQL query against an existing database connection and save the query results in an XCom file for use by other cells. Use this cell type to run smaller queries and store the results in Airflow for quick access by other cells.
• Warehouse SQL: Run a SQL query against an existing database connection and store the query results in your data warehouse. Use this cell type for data operations that require more storage and reliability.
• Markdown: Add inline Markdown comments to your generated DAG code. Use this cell type to document code decisions and to make it easier for team members to collaborate on shared pipelines.

For more information about a specific cell type, see Run SQL in the Astro Cloud IDE, Run Python in the Cloud IDE, and Add documentation to an Astro Cloud IDE pipeline.

Additional improvements​

• To reduce the time it takes for Airflow to parse new DAG files, the default value for AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL has been reduced from 5 minutes to 30 seconds for all Deployments regardless of Runtime version. For most users, this means that you will see new DAGs appear in the Airflow UI faster.
• In the Astro UI, a banner now appears if there is an incident reported on the Astro status page.

Bug fixes​

• Sorting the Organization Role column in the People tab of the Astro UI now works as expected.
• Fixed an issue where lineage groups would occasionally not collapse as expected in the Lineage Graph view.

December 20, 2022​

Additional improvements​

• You can now configure OneLogin and Ping Identity as identity providers on Astro.

• Workspace Members can now view Workspace settings in the Astro UI.

• Node groups that are collapsed in the lineage graph in the Astro UI now show only the total number of connected Jobs and Datasets, instead of listing each job and dataset. This makes the lineage graph easier to navigate.

  

Bug fixes​

• Fixed an issue where the lineage UI did not show dataset metrics when a dataset had no column-level metrics.
• Fixed an issue where some instances of a dataset's name were inconsistent in the lineage UI.

December 13, 2022​

Improvements to the Cloud IDE​

The Cloud IDE includes several new features which improve DAG authoring and testing:

• There is a new Commit button in the Astro UI that is separate from the Configuring GitHub menu.
• The default CI/CD pipeline included in the Cloud IDE project supports DAG-only deploys. Deploying DAG changes to Astro using the CI/CD pipeline is now significantly faster.
• The Configure GitHub menu in the Astro UI now includes a Clone Repo settings menu. Enabling this option makes other files in your GitHub repository, such as helper functions in the include folder of your project, accessible when you run DAGs in the Cloud IDE.
• You can now explicitly mark upstream dependencies for a task cell from the cell's configuration menu.

For more information about configuring GitHub and deploying code with the Cloud IDE, see Deploy a project from the Cloud IDE to Astro.

Support for n2 worker types on GCP​

You can now configure worker queues with the following n2 worker types on Google Cloud Platform (GCP) clusters:

• n2-standard-4
• n2-standard-8
• n2-standard-16
• n2-highmem-4
• n2-highmem-8
• n2-highmem-16
• n2-highcpu-4
• n2-highcpu-8
• n2-highcpu-16

For more information about these worker types, see N2 machine series. For a list of all worker types available on GCP, see Worker node size resource reference.

Additional improvements​

• In the Clusters tab of the Astro UI, you can now click a cluster entry to see details about the cluster configuration, including which Worker Types are enabled for the cluster.
• The Deployment details page in the Astro UI now includes an ID pane. A Deployment ID is required when you deploy code using a CI/CD process.
• The OpenLineage URL for your Organization is now available on the Settings page in the Astro UI. An OpenLineage URL is required to integrate data lineage from some external systems.
• Workspaces are now sorted alphabetically in the Astro UI.
• In Astro CLI version 1.8.0 or later, running astro deploy with an empty or missing dags folder does not erase or override existing DAGs. Instead, the directory is excluded from the build and push process to Astro. This lets you manage your DAGs and project files in separate repositories when using DAG-only deploys.

Bug fixes​

• Fixed an issue where Astro temporarily stored DAGs for DAG-only deploys in a new directory named /usr/local/airflow/dags/current, which could cause import errors in user code.
• Fixed an issue where task runs triggered in the Cloud IDE did not have access to project environment variables.
• Fixed an issue where Deployment metrics for memory usage were not always accurate.

November 15, 2022​

Additional improvements​

• In the Astro UI, the People page now shows the IDs of users belonging to your Organization.
• In the Astro UI, the Deployments page now shows the user or API key that most recently updated each Deployment and when they updated it.

Bug fixes​

• Availability zone (AZ) rebalancing has been disabled for worker node pools on AWS clusters. This change should result in fewer zombie tasks and less volatility across workers. AZ rebalancing is enabled for other system components on Astro.
• The Updated at field for a transferred Deployment now displays the correct time.
• astro deploy --dags now handles deferrable tasks correctly.

November 8, 2022​

Deploy only DAGs with astro deploy -—dags​

Using Astro CLI 1.7, you can run astro deploy -—dags to push only the dags directory of your Astro project to a Deployment on Astro. This is an additional option to astro deploy that makes for a faster development experience and gives you more flexibility in how you configure CI/CD processes.

For more information, see Astro CLI 1.7 or Deploy DAGs only. For example CI/CD workflows with this feature enabled, see CI/CD.

Improved data lineage interface​

The Lineage tab has new features and is better integrated into the Astro UI.

Specifically, the tab includes the following improvements:

• The process for comparing runs uses a simpler interface and provides more information about the runs you're comparing. See Compare lineage graphs from previous runs.
• Names for UI elements have been updated to more clearly represent Airflow resources. For example, jobs is now runs, and the Explore tab is now Runs.
• Lineage graphs include new colors and animations to show the flow of data as it moves between runs and datasets.

Transfer a Deployment​

You can now transfer a Deployment from one Workspace to another in your Organization. This feature is helpful if you need to change the group of users that have access to a Deployment, or if you create a Deployment in the wrong Workspace.

See Transfer a Deployment to another Workspace.

Additional improvements​

• The Kubernetes API is no longer exposed to the public internet on AWS data planes. The allowlist is limited to control plane IPs. New clusters will be created with this configuration, while all existing clusters will be updated by end of next week.

November 1, 2022​

Introducing the Astro Cloud IDE, a new Airflow development experience​

Astronomer is excited to introduce the Astro Cloud IDE, which is a notebook-inspired development environment for writing, running, and deploying data pipelines. Now you can develop an entire Airflow project, including DAGs, dependencies, and connections entirely within the Astro UI.

The Astro Cloud IDE was created with the following objectives:

• Configuring Airflow shouldn't be a barrier to running Airflow.
• Passing data between tasks should be seamless regardless of what language is used to write the task.
• Data pipelines should be quick to deploy and easy to test with CI/CD.

Most importantly, the Astro Cloud IDE was developed to make it easier for new Airflow users to get started and to provide experienced users with a robust development environment.

To create your first project in the Astro Cloud IDE, see the Cloud IDE quickstart. To deploy your project to Astro, see Deploy your Cloud IDE project to Astro.

Additional improvements​

• In the Astro UI, cluster selection menus are now alphabetized.

Bug fixes​

• Fixed an issue where the KubernetesPodOperator was not aware of available ephemeral storage in m5d and m6id worker nodes. This issue resulted in Pods being evicted to free up storage even when there was enough available storage for tasks.
• Fixed an issue in the Astro UI where you could select a worker type before selecting a cluster when creating a Deployment.
• Fixed an issue where Deployments on Runtime 5.0.10 and earlier showed a nonfunctional Configuration tab in the Airflow UI.
• Fixed CVE-2022-32149.

October 25, 2022​

Additional improvements​

• In the Astro UI, you can now view a cluster's external IP addresses in the Clusters tab.

Bug fixes​

• Fixed an issue where some Deployments were running tasks after being deleted.

October 18, 2022​

Additional improvements​

• In the Astro UI, Access has been moved from the left menu to a tab on the Workspace Settings page.
• In the Astro UI, Workspace Settings in the left menu is now available to all Workspace members.

New Azure regions​

You can now create an Astro cluster on Azure in the following regions:

• japaneast
• southafricanorth
• southcentralus

October 11, 2022​

Additional improvements​

• New worker node pools on Azure and Google Cloud Platform (GCP) clusters can now scale to zero. When you set your minimum worker count to 0, you don't incur costs for enabling a new worker type for your cluster until it's used in a Deployment.

Bug fixes​

• Fixed an issue where worker queues with a minimum worker count of zero would appear with a minimum worker count of one in the Astro UI.

October 4, 2022​

New permissions boundary for managed AWS Accounts​

The operational roles that Astronomer assumes on dedicated customer AWS accounts now have new permissions boundaries that limit the roles to a subset of their permissions. The remote management role is now limited to the following actions across all contexts:

• autoscaling:*
• cloudformation:*
• cloudwatch:*
• ec2:*
• ecr:*
• eks:*
• elasticloadbalancing:*
• iam:*OpenID*
• kms:DescribeKey
• lambda:*
• logs:*
• route53:AssociateVPCWithHostedZone
• s3:*
• secretsmanager:*
• servicequotas:*
• ssm:*
• tag:*

These permissions might change in the future to enable new Astro features or to refine permissions for specific contexts.

Additional improvements​

• Users with the required permissions can now access a Configuration tab in the Admin menu of the Airflow UI. This page no longer shows sensitive values in plain-text and can be used to verify all configurations running on your Deployment.
• In the Astro UI, the maximum time for Deployment metrics has been extended from 24 hours to 7 days.
• The Deployment metrics overview now shows metrics for the default worker queue instead of an aggregate of all worker queues. Improved worker queue metrics coming soon.

Bug fixes​

• Added the global environment variable AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET=stdout so that a scheduler's logs don't overcrowd its local storage
• Removed misleading maximum CPU and memory lines from Deployment metric graphs

September 28, 2022​

Additional improvements​

• All worker queue configurations in the Worker Queues tab of the Astro UI now have tooltips.
• The Worker CPU and Worker Memory metrics in the Analytics tab of the Astro UI now show metrics only for the default worker queue instead of an average across queues. Improved worker queue metrics coming soon.

Bug fixes​

• Values in the Organization Settings page no longer overlap with other UI elements.
• Organization Owners can now push code to a Deployment even if they aren't explicit members of the Deployment's Workspace.

September 21, 2022​

A simpler Deployment page​

All of a Deployment's configurations, including analytics, API keys, environment variables, and resource configurations, are now organized as tabs within the Deployment's page in the Astro UI.

This new UI moves the Analytics and Logs from the left sidebar to the main Deployment page so that you no longer have to filter those views separately by Deployment. The left sidebar now exclusively contains Workspace-level menus.

New Account Dashboard​

You can now access your Account Dashboard to manage your user account settings and find links to helpful resources. Access this page by going to account.astronomer.io in your browser or by clicking Profile > Manage your Astro Account in the Astro UI. You must be authenticated to Astro.

Additional improvements​

• You can now use the m6id worker node type series for Deployments on AWS clusters. This worker type is general purpose and includes significant storage as well as up to 15% better performance compared to m5d nodes. For more information, see Worker instance types.
• New worker node pools on Amazon Web Services (AWS) clusters can now scale to zero. This means that enabling a new worker type for your cluster does not cost you until it's used in a Deployment.

Bug fixes​

• Fixed an issue where the Astro UI Deployment metrics showed a maximum worker CPU and memory that was inconsistent with your configured worker queues.

September 14, 2022​

Additional improvements​

• When you create a new worker queue, the default worker type in your cluster is now pre-selected in the Worker Type list.
• You can now configure multiple instances of the same identity provider (IdP). See Configure an identity provider.
• You can now expand and collapse the Workspace menu in the Astro UI.

Bug fixes​

• Fixed an issue where you could not open the Airflow UI from a Deployment.

August 31, 2022​

Export Deployment metrics to Datadog​

You can now export over 40 Airflow metrics related to the state of your Astro Deployment to Datadog by adding a Datadog API key to the Deployment. Metrics include task successes, DAG processing time, frequency of import errors, and more.

For organizations already using the observability service, this integration allows your team to standardize on tooling and gain a more granular view of Deployment metrics in a single place. Once the integration is configured, Astro automatically exports all available metrics to Datadog. For a complete list of supported metrics, see Data Collected.

To learn more, see Export Airflow metrics to Datadog.

Additional improvements​

• The Astro UI now automatically ensures that worker queue names are valid as you type in real time.
• The number of times that a user can enter the wrong credentials for Astro before being locked out has been reduced from 10 to 6.
• You can now configure worker queues to have a minimum Worker count of 0 workers. Note that depending on your cloud provider and Deployment configurations, some Deployments still might not be able to scale to 0 workers.

Bug fixes​

• The timestamp shown in the Updated field of the Deployment view in the Astro UI is now properly updated when you create or modify environment variables.
• Fixed an issue where logging in to the Airflow UI with unrecognized credentials could freeze you on an error page.

August 24, 2022​

Additional improvements​

• When you configure worker queues in the Astro UI, the total CPU and memory capacity of each worker instance type is now shown instead of the nominal available resources.
• Improved error handling for creating new worker queues when soft-deleted worker queues might still exist on the data plane.

Bug fixes​

• Fixed an issue where running astro deploy with a Deployment API key could revert changes to a worker queue's size that were previously set in the Astro UI.
• Fixed an issue where the Lineage tab in the Astro UI showed all job durations as having a length of 0.

August 18, 2022​

Create multiple worker queues​

Worker queues are a new way to configure your Deployment to best fit the needs of your tasks. A worker queue is a set of configurations that apply to a group of workers in your Deployment. Within a worker queue, you can configure worker type and size as well as autoscaling behavior. By configuring multiple worker queues for different types of tasks, you can better optimize for the performance, reliability, and throughput of your Deployment.

In the Astro UI, you can now create multiple worker queues. Once you create a worker queue, you can assign a task to that worker queue by adding a simple queue='<worker-queue-name>' argument in your DAG code.

This feature enables the ability to:

• Use more than one worker type within a single Deployment and cluster. Previously, a single cluster on Astro supported only one worker type.
• Isolate long-running tasks from short-running tasks to avoid errors related to competing resource requests.
• Fine-tune autoscaling behavior for different groups of tasks within a single Deployment.

For example, if you have a task that requires significantly more CPU than memory, you can assign it to a queue that's configured with workers that are optimized for compute usage.

To learn more about configuring worker queues, see Configure Deployment resources.

New worker sizing​

This Astro release introduces a new, simple way to allocate resources to the workers in your Deployment. Instead of choosing a varying combination of CPU and memory, you can now select a worker type in the Astro UI as long as it's enabled in your cluster. For example, m5.2xlarge or c6i.8xlarge on AWS. Once you select a worker type, Astronomer will create the biggest worker that that worker type can support to ensure that your tasks have enough resources to execute successfully.

Astro's worker sizing enables a few benefits:

• You can no longer configure a worker that is too large or otherwise not supported by your underlying cluster. Previously, misconfiguring worker size often resulted in task failures.
• A more efficient use of infrastructure. Astronomer has found that a lower number of larger workers is more efficient than a higher number of smaller workers.
• A higher level of reliability. This worker sizing model results in less volatility and a lower frequency of cluster autoscaling events, which lowers the frequency of errors such as zombie tasks and missing task logs.
• The legacy AU unit is no longer applicable in the context of the worker. You only have to think about CPU, memory, and worker type.

Worker sizing on Astro is now defined in the context of worker queues. For more information about worker sizing, see Configure Deployment resources. For a list of supported worker types, see the AWS, GCP, and Azure resource references.

New Maximum Tasks per Worker setting​

A new Maximum Tasks per Worker configuration is now available in the Deployment view of the Astro UI. Maximum tasks per worker determines the maximum number of tasks that a single worker can process at a time and is the basis of worker autoscaling behavior. It is equivalent to worker concurrency in Apache Airflow.

Previously, maximum tasks per worker was permanently set to 16 and was not configurable on Astro. Now, you can set maximum tasks per worker anywhere between 1 and 64 based on the needs of your tasks. It can be set per worker queue on a Deployment.

To learn more, see Worker autoscaling logic.

New Worker Count (Min-Max) setting​

A new Worker Count (Min-Max) configuration is now available in the Deployment view of the Astro UI. This value defines the minimum and maximum number of workers that can run at a time.

Use this setting to fine-tune worker autoscaling behavior in your Deployment. By default, the minimum number of workers is 1 and the maximum is 10.

Support for multiple Organizations​

A single user account can now belong to multiple Organizations. A user with multiple Organizations can switch to another Organization by clicking on their current Organization's name in the Astro UI and then clicking Switch Organization.

Note that switching Organizations with the Astro CLI is not yet supported. For more information, see Switch Organizations.

New Azure region (Australia East)​

You can now create an Astro cluster on Azure in Australia East (New South Wales).

New Google Cloud Platform regions​

You can now create an Astro cluster on GCP in the following regions:

• australia-southeast2 (Melbourne)
• asia-east1 (Taiwan)
• asia-south2 (Delhi)
• asia-southeast2 - (Jakarta)
• europe-north1 (Finland)
• europe-southwest1 (Madrid)
• europe-west8 (Milan)
• europe-west9 (Paris)
• northamerica-northeast2 (Toronto)
• southamerica-west1 (Santiago)
• us-east5 (Columbus)
• us-south1 (Dallas)

Bug fixes​

• Fixed an issue where the Astro UI's Resource Settings page wasn't showing units for CPU and Memory values.

August 10, 2022​

Updated user permissions for Organization and Workspace roles​

The following user roles have new and modified permissions:

• Organization Owners now have Workspace Admin permissions for all Workspaces in their Organization. This role can now access Organization Workspaces, Deployments, and usage data.
• Organization Billing Admins can now view usage for all Workspaces in their Organization regardless of their Workspace permissions.
• Workspace Editors can now delete any Deployment in their Workspace.

Automatic access for new users authenticating with an identity provider​

If your organization has implemented an identity provider (IdP), any new user who authenticates to Astro through your IdP is now automatically assigned the Organization Member role. This means that users authenticating through your IdP do not need to be invited by email before joining your Organization.

Additional improvements​

• Added a security measure that ensures Workspace roles can only be assigned to users who have an Organization role in the Organization in which the Workspace is hosted. This ensures that a user who does not belong to your Organization cannot be assigned a Workspace role within it.

August 2, 2022​

Support for Astro on Azure Kubernetes Service (AKS)​

Astro now officially supports Astro clusters on AKS. This includes support for an initial set of AKS regions.

For more information about the installation process and supported configurations, see Install Astro on Azure and Resource Reference Azure.

Bug fixes​

• Pending invites no longer appear for active users in the Astro UI.

July 27, 2022​

New Deployment optimizations for high availability (HA)​

This release introduces two changes that ensure a higher level of reliability for Deployments on Astro:

• PgBouncer, a microservice that increases resilience by pooling database connections, is now considered highly available on Astro. Every Deployment must now have 2 PgBouncer Pods instead of 1, each assigned to a different node within the cluster. This change protects against pod-level connection issues resulting in zombie tasks, which was previously seen during cluster downscaling events. PgBouncer is fully managed by Astronomer and is not configurable.

• The Airflow scheduler is now configured with an anti-affinity policy to limit the possibility of all schedulers for a single Deployment being impacted by an incident within a single node on an Astro cluster. For users who set Scheduler Count in the Astro UI to 2, this means that those 2 scheduler Pods cannot be assigned to the same node and instead require a minimum of 2 nodes total. To avoid significant increases in cost, 3 or 4 schedulers can share the same 2 nodes and will not necessarily result in a higher node count minimum.

For more information on Deployment configurations, see Deployment settings.

Additional improvements​

• Added tooltips for Deployment overview metrics in the Astro UI.

July 21, 2022​

Additional improvements​

• You can now access an Organization's AWS external ID from the Settings tab of the Astro UI.
• Organizations now need only a single AWS external ID for all clusters. Previously, each cluster required a unique external ID, which added complexity to the installation and cluster creation process.
• You can now remove a user from an Organization from the Astro UI.
• Organization Billing Admins can now view task usage for all Workspaces regardless of their Workspace permissions.

July 14, 2022​

Additional improvements​

• The Astro UI Clusters page now includes the cluster ID value.
• Organization Owners and Organization Billing Admins can now update the Organization name in the Astro UI Settings page.
• The Astro UI Analytics page can now show data for the last 30 minutes.

Bug fixes​

• When you change Worker Resources for a Deployment in the Astro UI, any errors related to your worker size request are now based on the correct node instance type that your cluster is running.
• When you select a Workspace and click Go back in a browser, the page now reloads as expected.
• A Page not found error message no longer appears when you select a Deployment in the Usage page of the Astro UI.
• The Deployment Analytics page now displays the correct date.
• Deprecated versions of Astro Runtime now appear correctly in the Deployment page of the Astro UI. Previously, versions were appended with -deprecated.

June 30, 2022​

New Google Cloud Platform regions​

You can now create an Astro cluster on GCP in the following regions:

• asia-northeast1 (Tokyo)
• asia-northeast2 (Osaka)
• asia-northeast3 (Seoul)
• asia-south1 (Mumbai)
• europe-central2 (Warsaw)
• europe-west6 (Zurich)
• northamerica-northeast1 (Montreal)
• us-west3 (Salt Lake City)

Additional improvements​

• You can now search for Organization members by name, email address, and role in the People tab of the Organization view in the Astro UI. You can also search for members in the Access tab of the Workspace view.

Bug fixes​

• Fixed an issue where you could not use the KubernetesPodOperator to execute tasks in a Kubernetes cluster outside of your Astro cluster. See KubernetesPodOperator.

June 23, 2022​

New Google Cloud Platform regions​

You can now create an Astro cluster on GCP in the following regions:

• asia-southeast1 (Singapore)
• australia-southeast1 (Sydney)
• europe-west1 (Belgium)
• europe-west2 (England)
• europe-west3 (Frankfurt)
• southamerica-east1 (São Paulo)
• us-west2 (Los Angeles)
• us-west4 (Nevada)

June 16, 2022​

Submit Support Requests in the Astro UI​

Support requests can now be created and submitted in the Astro UI. You no longer need to open an account on the Astronomer support portal to reach the Astronomer team. To streamline the request process, the Submit Support Request form auto-populates your currently selected Workspace and Deployment in the Astro UI.

Parallelism Now Autoscales with a Deployment's Worker Count​

To better scale concurrent task runs, Astro now dynamically calculates parallelism, which is an Airflow configuration that determines the maximum number of tasks that can run concurrently within a single Deployment.

A Deployment's parallelism is now equal to the current number of workers multiplied by the worker_concurrency value. This change ensures that your task runs won't be limited by a static parallelism limit as workers autoscale in your Deployment. See Worker Autoscaling Logic for more information.

Note that you can still use a static parallelism value by setting AIRFLOW__CORE__PARALLELISM as an environment variable.

Bug Fixes​

• Fixed a rare issue where some user emails would be associated with the wrong username.
• Fixed an issue where you could not properly sort entries in the People tab by name.

June 9, 2022​

Update Deployment configurations with the Astro CLI​

You can now programmatically update the configurations for your Astro Deployments using Deployment API keys and the Astro CLI. Updating a Deployment with an API key doesn't require manual user authentication, meaning that you can now add Deployment configuration steps to automated processes such as CI/CD pipelines.

Specifically, you can now run the following commands with Deployment API keys:

• astro deployment list
• astro deployment update
• astro deployment variable create
• astro deployment variable list
• astro deployment variable update

Bug fixes​

• Fixed an issue where a Deployment's logs wouldn't load in the Astro UI if it was the only Deployment in the Workspace

June 2, 2022​

Support for the us-east4 GCP region​

You can now create an Astro cluster on GCP in the us-east4 region, which is located in northern Virginia, USA.

May 26, 2022​

New Datasets page in the Astro UI​

You can now use the new Datasets page in the Lineage tab to view a table of datasets that your DAGs have read or written to. This information can help you quickly identify dataset dependencies and data pipeline access requirements.

Click the name of a dataset to show its lineage graph. For more information, see Data lineage on Astro.

Bug fixes​

• Fixed an issue where the Astro Runtime field of the Astro UI listed the running version as Unknown for Deployments using an unsupported version of Astro Runtime

May 5, 2022​

Data lineage Is now available on Astro​

We are excited to introduce data lineage to Astro. You now have access to a new Lineage view in the Astro UI that visualizes data movement across datasets in your Organization based on integrations with Airflow, Apache Spark, dbt, Great Expectations, and more.

Built around the OpenLineage open source standard, the data lineage graphs and metadata in the Astro UI can help you better understand your ecosystem and diagnose issues that may otherwise be difficult to identify.

For example, if an Airflow task failed because the schema of a database changed, you might go to the Lineage page on Astro to determine which job caused that change and which downstream tasks failed because of it.

To learn more about data lineage and how you can configure it on Astro, see:

• Integrate Airflow and OpenLineage
• Enable data lineage for External Services
• Data lineage on Astro
• OpenLineage Compatibility Matrix

Support for Astro on Google Cloud Platform (GCP)​

Astro now officially supports Astro clusters on Google Cloud Platform (GCP). This includes support for an initial set of GCP regions as well as Workload Identity for secure connection to other GCP data services in your ecosystem.

For more information about the installation process and supported configurations, see Install Astro on GCP and Resource Reference GCP.

Support for Organization-Level user invites​

You can now invite users to an Astro Organization without having to first invite them to a specific Workspace. Users invited to an Organization will receive an activation email which brings them directly to the Organization view of the Astro UI.

Additional improvements​

• Improved the templated emails sent out for user invites with clear instructions for how to get started on Astro
• Improved error messaging behavior on the DAGs and Usage pages of the Astro UI
• New user accounts must now be verified via email before they can access Astro

April 28, 2022​

New AWS node instance types available​

To widen our support for various use cases and levels of scale, we've expanded the types of AWS node instances that are supported on Astro. You can now create clusters with:

• General Purpose M6i instances
• Compute Optimized C6i instances
• Memory Optimized R6i instances

To modify an existing Astro cluster to use any of these instance types, see Modify a Cluster.

Additional improvements​

• Improve the error message that renders in the Astro UI if you try to create a worker that is too large for the Deployment's node instance type to support. This error message now specifies a clear call to action

April 21, 2022​

Feedback in Astro UI on worker size limits​

The Astro UI now renders an error if you try to modify the Worker Resources to a combination of CPU and memory that is not supported by the node instance type of the cluster that the Deployment is hosted on. This validation ensures that the worker size you request is supported by the infrastructure available in your Astro cluster, and minimizes silent task failures that might have occurred due to invalid resource requests.

If your Astro cluster is configured with the m5d.8xlarge node type, for example, the Astro UI will show an error if you try to set Worker Resources to 350 AU. This is because the maximum worker size an m5d.8xlarge node can support is 307 AU.

April 14, 2022​

Additional improvements​

• The data plane now connects to various AWS services via AWS PrivateLink. This ensures that traffic to AWS services is kept private and does not traverse the NAT and Internet gateways, reducing the risk of exposing your resources to the internet.

Bug fixes​

• Fixed an issue where you could not add a new user to a Workspace if the user had an email address that contained uppercase characters

March 31, 2022​

New analytics page in Astro UI to monitor Deployments​

The Astro UI now includes a dedicated Analytics page that contains various Deployment-level metrics. These metrics are collected in real time and can provide insight into how your data pipelines are performing over time:

For more information about accessing the Analytics page and the available metrics, see Deployment Analytics.

Lineage backend upgrade scheduled for all Organizations​

As part of Astronomer's acquisition of Datakin, data lineage features are coming soon to Astro. The first step in enabling these features is to implement lineage backends for existing Astro customers.

Starting on March 31st and continuing over the next couple of weeks, all Astro Deployments on Runtime 4.2.0+ will be upgraded to emit lineage events. As a result of this change, you might start seeing lineage-related scheduler logs such as the following:

[2022-03-30, 12:17:39 UTC] {great_expectations_extractor.py:17} INFO - Did not find great_expectations_provider library or failed to import it
[2022-03-24, 23:40:01 UTC] {client.py:74} INFO - Constructing openlineage client to send events to https://api.astro-astronomer.datakin.com

A few additional notes about this upgrade:

• You can ignore any lineage logs that indicate an error or failed process, such as the first line in the example logs above. These logs will more accurately reflect the state of your lineage functionality once lineage features are launched on Astro.
• Deployments on Runtime 4.2.0+ will be updated to emit data lineage events only after you push code. Until you do so, this change will not be applied.
• Because Astronomer is upgrading each customer individually over time, the exact date that you will start seeing these logs will vary.
• When you push code to a Deployment on Runtime 4.2.0+ and trigger this update, all other Deployments on Runtime 4.2.0+ in the same Workspace will also restart in order to receive the lineage backend update. If you plan to push code to any Deployment affected by this change, then we recommend doing so at a time where you can tolerate some Airflow components restarting. For more information about expected behavior, see What Happens During a Code Deploy.

For more information about what to expect when lineage tools go live, read Astronomer's OpenLineage and Airflow guide.

New AWS regions available​

You can now create new Clusters in:

• af-south-1 (Cape Town)
• ap-east-1 (Hong Kong)
• ap-northeast-3 (Osaka)
• me-south-1 (Bahrain)

Additional improvements​

• The Astro UI now includes a button that links to Astronomer support and status pages:

  

March 25, 2022​

Maximum node count is now configurable per Cluster​

As of this release, Maximum Node Count is now a configurable setting for new and existing clusters. On Astro, maximum node count represents the total number of EC2 nodes that your cluster can support at any given time. For an Astro cluster on AWS, EC2 nodes are the primary unit of infrastructure required to run a Deployment and its components, including workers and the Airflow scheduler. New clusters have a maximum node count of 20 by default, but the setting can be modified to any value from 2 to 100 at any time.

Previously, maximum node count was a fixed, global setting that applied to all customers on Astro and could not be configured per cluster. Now, your organization can modify this setting as your workloads evolve and more Deployments are created. Once the limit is reached, your cluster will not be able to auto-scale and worker pods may fail to schedule.

To update this setting for an existing cluster, reach out to Astronomer support and provide the name of your cluster and the desired maximum node count.

Additional improvements​

• In Resource Settings, the maximum allowed value for Worker Resources has been increased to 400 AU.

March 17, 2022​

Export task usage as a CSV file​

In the Astro UI, you can now export your task usage data from the Usage tab as a CSV file to perform more complex data analysis related to your Airflow usage and costs. For example, you can use the file as the basis for a pivot table that shows total task usage by Workspace.

To export your task usage data as a CSV file, click the Export button in the Usage tab:

Bug fixes​

• Fixed an issue where saving new environment variables in the Astro UI would occasionally fail

March 10, 2022​

Running Docker image tag in Airflow UI​

The Docker image that is running on the Airflow webserver of your Deployment is now shown as a tag in the footer of the Airflow UI. Depending on how your team deploys to Astro, this tag is either a unique identifier generated by a CI tool or a timestamp generated by the Astro CLI on astro deploy. Both represent a unique version of your Astro project.

When you push code to a Deployment on Astro via the Astro CLI or CI/CD, reference this tag in the Airflow UI to verify that your changes were successfully applied.

To upgrade a Deployment to the latest Runtime version, see Upgrade Runtime.

March 3, 2022​

Additional improvements​

• The threshold in order for bars in the Worker CPU and Worker Memory charts to appear red has been reduced from 95% to 90%. This is to make sure that you get an earlier warning if your workers are close to hitting their resource limits.

Bug fixes​

• Fixed an issue where malformed URLs prevented users from accessing the Airflow UI of some Deployments on Astro
• Fixed an issue where Astro Runtime 4.0.11 wasn't a selectable version in the Astro Runtime menu of the Deployment creation view in the Astro UI

February 24, 2022​

Bug fixes​

• Removed the Teams tab from the Astro UI. This view was not yet functional but coming back soon
• Fixed an issue where the number of users per Workspace displayed in the Organization view of the Astro UI was incorrect
• Fixed an issue where if a secret environment value was updated in the Astro UI and no other values were modified, the change was not applied to the Deployment

February 17, 2022​

Introducing Astro and a new look​

This week's release introduces a reimagined Astronomer brand that embraces Astro as a rename of Astronomer Cloud. The rebrand includes a new Astronomer logo, color palette, and font.

The new Astronomer brand is now reflected both in the Astro UI as well as in the main Astronomer website and documentation.

In addition to visual changes, we've renamed the following high-level Astro components:

• Astronomer Cloud CLI is now Astro CLI
• Astronomer UI is now Cloud UI
• Astro Runtime is now Astro Runtime

We hope you find this exciting. We're thrilled.

New Organization roles for users​

The following Organization-level roles are now supported on Astro:

• Organization Member: This role can view Organization details and membership. This includes everything in the People, Clusters, and Settings page of the Astro UI. Organization members can create new Workspaces and invite new users to an Organization.
• Organization Billing Admin: This role has all of the Organization Member's permissions, plus the ability to manage Organization-level settings and billing. Organization Billing Admins can access the Usage tab of the Astro UI and view all Workspaces across the Organization.
• Organization Owner: This role has all of the Organization Billing Admin's permissions, plus the ability to manage and modify anything within the entire Organization. This includes Deployments, Workspaces, Clusters, and users. Organization Owners have Workspace Admin permissions to all Workspaces within the Organization.

Organization roles can be updated by an Organization Owner in the People tab of the Astro UI. For more information about these roles, see User permissions.

Create new Workspaces from the Astro UI​

All users can now create a new Workspace directly from the Overview tab of the Astro UI:

When you create a new Workspace, you will automatically become a Workspace Admin within it and can create Deployments. For more information about managing Workspaces, see Manage Workspaces.

Bug fixes​

• Fixed an issue where authentication tokens to Astro weren't properly applied when accessing the Airflow UI for a Deployment. This would result in an authenticated user seeing Error: Cannot find this astro cloud user in the Airflow UI.
• Fixed an issue where long environment variable values would spill out of the Value column and onto the Updated column in the Environment Variables view of a Deployment in the Astro UI.

February 11, 2022​

Monitor DAG runs across all Deployments in a Workspace​

You can view key metrics about recent DAG runs through the new DAGs page in the Astro UI. Use this page to view DAG runs at a glance, including successes and failures, across all Deployments in a given Workspace. You can also drill down to a specific DAG and see metrics about its recent runs.

For more information about the DAGs page, see Deployment metrics.

Additional improvements​

• All resource settings in the Deployment view of the Astronomer UI now show exact CPU and Memory usage to the right of every slider, previously shown only in Astronomer Units (AUs). This makes it easy to know exactly how many resources you allocate to each component.
• A banner now appears in the Astronomer UI if a Deployment is running a version of Astro Runtime that is no longer maintained. To make the most of features and bug fixes, we encourage users to upgrade to recent versions as much as possible.
• Added more ways to sort pages that utilize card views, such as the Deployments page
• Added user account avatars next to usernames in several places across the Astro UI

Bug fixes​

• Removed the Environment field from the Deployment view of the Astronomer UI. This field is not currently functional and will be re-added as soon as it is.

February 3, 2022​

Support for third-party identity providers​

You can now integrate both Azure AD and Okta as identity providers (IdPs) for federated authentication on Astro. By setting up a third-party identity provider, a user in your organization will be automatically logged in to Astro if they're already logged in via your IdP. By adding new Astro users through your IdP's own user management system, Workspace Admins can automatically add new users to their Workspace without those users needing to individually sign up for Astro.

For more information about this feature read Set up an identity provider.

Support for the Astro CLI​

The Astro CLI (astro) is now generally available as the official command-line tool for Astro. It is a direct replacement of the previously released astro executable and comes with various significant improvements. We encourage all customers to upgrade.

For more information on the Astro CLI, see CLI Release Notes. For install instructions, read Install the CLI.

Multiple authentication methods for a single user account​

Astro now supports multiple authentication methods for a single user account. This means that as long as you're using a consistent email address, you now have the flexibility to authenticate with GitHub, Google, username/password, and/or an external identity provider (idP) at any time. Previously, a single user account could only be associated with one authentication method, which could not be changed after the account was created.

This also means that all Organizations now have GitHub, Google, and username/password authentication methods enabled by default for all users.

Additional improvements​

• Changed the default RDS instance type for new clusters from db.r5.xlarge to db.r5.large, which represents a monthly cost reduction of ~50% for newly provisioned clusters. Customers with existing clusters will need to request a downscale via Astronomer support

January 13, 2022​

Identity-based login flow​

Astro now utilizes an identity-based login flow for all users. When you first log in via the Astro UI, you now only need to enter the email address for your account. Astro assumes your Organization and brings you directly to your Astro Organization's login screen.

This change serves as a foundation for future SSO and authentication features. In upcoming releases, users will be able to authenticate via custom identity providers like Okta and Azure Active Directory.

Additional improvements​

• Significant improvements to the load times of various Astro UI pages and elements.
• In the Astro UI, the tooltips in the Resource Settings section of a Deployment's page now show the definition of 1 AU. This should make it easier to translate AU to CPU and Memory.
• Scheduler logs in the Astro UI no longer show DEBUG-level logs.
• To ensure that all workers have enough resources to run basic workloads, you can no longer allocate less than 10 AU to Worker Resources.

January 6, 2022​

Improvements to "Scheduler Logs" in the Astro UI​

The Scheduler Logs tab in the Astro UI has been updated to make logs easier to read, separate, and parse. Specifically:

• You can now filter logs by type (DEBUG, INFO, WARN, and ERROR).
• The page now shows logs for the past 24 hours instead of the past 30 minutes.
• The page now shows a maximum of 500 logs instead of a lower maximum.
• When looking at a Deployment's logs, you can return to the Deployment's information using the Deployment Details button.

Removal of worker termination grace period​

The Worker Termination Grace Period setting is no longer available in the Astro UI or API. Previously, users could set this to anywhere between 1 minute and 24 hours per Deployment. This was to prevent running tasks from being interrupted by a code push. Today, however, existing Celery workers don't have to terminate in order for new workers to spin up and start executing tasks. Instead, existing workers will continue to execute running tasks while a new set of workers gets spun up concurrently to start executing the most recent code.

To simplify Deployment configuration and reflect current functionality:

• The worker Termination Grace Period was removed from the Astro UI
• This value was permanently set to 24 hours for all Deployments on Astro

This does not change or affect execution behavior for new or existing Deployments. For more information, read What Happens During a Code Deploy.

Additional improvements​

• Removed Kubernetes Version column from the Clusters table. This value was previously inaccurate and is not needed. The Kubernetes version of any particular Astro cluster is set and modified exclusively by Astro as part of our managed service.

December 16, 2021​

View scheduler error logs from the Astro UI​

The new Logs tab in the Astro UI shows scheduler error and warning logs for all Deployments in your Workspace. When you select a Deployment in this menu, all error logs generated over the last 30 minutes appear in the UI.

To access logs directly for a given Deployment, click the new Logs button on the Deployment's page or in the Deployments table.

For more information on how to view logs, read View logs.

Bug fixes​

Fixed various bugs in the Astro UI to better handle nulls and unknowns in Deployment metrics

December 9, 2021​

Additional improvements​

• In the Astro UI, the Open Airflow button now shows more specific status messages when a Deployment's Airflow UI is inaccessible.

Bug fixes​

• Fixed Deployment table scrolling and alignment issues in the UI

December 6, 2021​

New "Usage" tab in the Astro UI​

Total task volume for your Organization is now available in a new Usage tab in the Astro UI. Astro is priced based on successful task runs, so this view can help you monitor both Astro cost as well as Airflow usage in aggregate and between Deployments.

New AWS regions available​

You can now create new clusters in:

• us-west-1
• ap-northeast-1
• ap-southeast-1
• ap-northeast-2
• ap-southeast-2
• ap-south-1
• us-west-1
• us-west-2

For a full list of AWS regions supported on Astro, see Resources required for Astro on AWS.

Additional improvements​

• You can now see your Deployment's Namespace in the Deployments menu and on the Deployment information screen in the Astro UI. Namespace is a required argument to run tasks with the KubernetesPodOperator. It is also required to submit an issue to Astronomer support.

  

• The Astro UI now shows a warning if you attempt to exit Environment Variable configuration without saving your changes.

• A Deployment's health status is now based on the health of both the Airflow webserver and scheduler. Previously, a Deployment's health status was only based on the health of the webserver. Now, the Astro UI will show that your Deployment is "Healthy" only if both components are running as expected.

Bug fixes​

• The Astro UI now has error handling for attempts to access a Deployment that does not exist.
• If you attempt to modify an existing secret environment variable, the Value field is now blank instead of showing hidden characters.

Data plane improvements​

• Amazon EBS volumes have been upgraded from gp2 to gp3 for improved scale and performance.
• EBS volumes and S3 buckets are now encrypted by default.
• The ability to enable public access to any Amazon S3 bucket on an Astro data plane is now blocked per a new AWS account policy. Previously, public access was disabled by default but could be overridden by a user creating a new S3 bucket with public access enabled. This AWS account policy could be overridden by AWS account owners, but Astronomer strongly recommends against doing so.

November 19, 2021​

Secret environment variables​

You can now set secret environment variables via the Astro UI. The values of secret environment variables are hidden from all users in your Workspace, making them ideal for storing sensitive information related to your Astro projects.

For more information, read Set environment variables via the Astro UI.

Additional improvements​

• You can now create new clusters in AWS sa-east-1.
• Extra whitespace at the end of any environment variable that is set via the Astro UI is now automatically removed to ensure the variable is passed correctly.

November 11, 2021​

Deployment metrics dashboard​

In the Astro UI, your Deployment pages now show high-level metrics for Deployment health and performance over the past 24 hours.

For more information on this feature, read Deployment metrics.

Bug fixes​

• Resolved a security vulnerability by setting AIRFLOW__WEBSERVER__COOKIE_SECURE=True as a global environment variable

November 5, 2021​

Bug fixes​

• Fixed an issue where a new user could not exit the Astro UI "Welcome" screen if they hadn't yet been invited to a Workspace

October 29, 2021​

Cloud UI redesign​

The Astro UI has been redesigned so that you can more intuitively manage Organizations, Workspaces, and your user profile.

To start, the homepage is now a global view. From here, you can now see all Workspaces that you have access to, as well as information and settings related to your Organization: a collection of specific users, teams, and Workspaces. Many features related to Organizations are coming soon, but the UI now better represents how Organizations are structured and what you can do with them in the future:

You can now also select specific Workspaces to work in. When you click in to a Workspace, you'll notice the left menu bar is now entirely dedicated to Workspace actions:

• The Rocket icon brings you to the Deployments menu.
• The People icon brings you to the Workspace Access menu.
• The Gear icon brings you to the Workspace Settings menu.

To return to the global menu, you can either click the Astro "A" or click the Workspace name to produce a dropdown menu with your Organization.

All user configurations can be found by clicking your user profile picture in the upper right corner of the UI. From the dropdown menu that appears, you can both configure user settings and access other Astronomer resources such as documentation and the Astronomer Registry.

Additional improvements​

• You can now create new clusters in us-east-2 and ca-central-1.
• In the Deployment detail page, Astro Runtime now shows the version of Apache Airflow that the Deployment's Astro Runtime version is based on.
• You can now create or modify an existing Astro cluster to run any size of the t2,t3, m5, or m5d AWS EC2 instances.

Bug fixes​

• Fixed an issue where a new Deployment's health status did not update unless you refreshed the Astro UI

October 28, 2021​

Bug fixes​

• Fixed an issue where you couldn't push code to Astro with a Deployment API key via a CI/CD process
• Fixed an issue where you couldn't update or delete an API key after creating it

October 25, 2021​

Additional improvements​

• When deleting a Deployment via the UI, you now have to type the name of the Deployment in order to confirm its deletion.

Bug fixes​

• Fixed an issue where you could not access Airflow's REST API with a Deployment API key
• Fixed an issue where calling the imageDeploy API mutation with a Deployment API key would result in an error

October 15, 2021​

Additional improvements​

• When creating a new Deployment, you can now select only the latest patch version for each major version of Astro Runtime.

• When creating a new Deployment in the Astro UI, the cluster is pre-selected if there is only one cluster available.

• The name of your Astro Deployment now appears on the main DAGs view of the Airflow UI.

• You can now see the health status for each Deployment in your Workspace on the table view of the Deployments page in the Astro UI:

  

• In the Astro UI, you can now access the Airflow UI for Deployments via the Deployments page's card view:

  

• The Astro UI now saves your color mode preference.

October 1, 2021​

Additional improvements​

• In the Astro UI, the Open Airflow button is now disabled until the Airflow UI of the Deployment is available.
• Workspace Admins can now edit user permissions and remove users within a given Workspace.

September 28, 2021​

Additional improvements​

• In the Astro UI, a new element on the Deployment information screen shows the health status of a Deployment. Currently, a Deployment is considered unhealthy if the Airflow webserver is not running and the Airflow UI is not available:

  

• The documentation home for Astro has been moved to www.astronomer.io/docs, and you no longer need a password to access the page.

Bug fixes​

• The Astro UI now correctly renders a Deployment's running version of Astro Runtime.

September 17, 2021​

Support for Deployment API keys​

Astro now officially supports Deployment API keys, which you can use to automate code pushes to Astro and integrate your environment with a CI/CD tool such as GitHub Actions. For more information on creating and managing Deployment API keys, see Deployment API keys. For more information on using Deployment API keys to programmatically deploy code, see CI/CD. Support for making requests to Airflow's REST API using API keys is coming soon.

September 3, 2021​

Bug fixes​

• Added new protections to prevent S3 remote logging connections from breaking
• Fixed an issue where environment variables with extra spaces could break a Deployment
• Fixed an issue where Deployments would occasionally persist after being deleted via the UI
• In the UI, the Organization tab in Settings is now hidden from non-admin users
• In the UI, the table view of Deployments no longer shows patch information in a Deployment's Version value

August 27, 2021​

Additional improvements​

• You can now remain authenticated to Astro across multiple active browser tabs. For example, if your session expires and you re-authenticate to Astro on one tab, all other tabs running Astro will be automatically updated without refreshing.
• If you try to access a given page on Astro while unauthenticated and reach the login screen, logging in now brings you to the original page you requested.

Bug fixes​

• Fixed an issue where an incorrect total number of team members would appear in the People tab

August 20, 2021​

Support for the Airflow REST API​

You can now programmatically trigger DAGs and update your Deployments on Astro by making requests to Airflow's REST API. Currently this feature works only with temporary tokens, which are available at cloud.astronomer.io/token. Support for Deployment API keys is coming soon. For more information on using this feature, read Airflow API.

Additional improvements​

• Set AIRFLOW_HOME = 'usr/local/airflow' as a permanent global environment variable
• In the Astro UI, long environment variable keys and values now wrap to fit the screen
• Added links for the Astronomer Registry and certification courses to the left-hand navbar
• Moved the Teams and People tabs into the Settings page of the UI
• Added Cluster information to the metadata section of a Deployment's information page in the UI
• Renamed various UI elements to better represent their functionality
• Increased the maximum Worker Termination Grace Period from 600 minutes (10 hours) to 1440 minutes (24 hours)

Bug fixes​

• The left-hand navbar in the UI is no longer cut off when minimized on smaller screens
• Fixed an issue where you could not delete a Workspace via the UI
• Fixed an issue where expired tokens would occasionally appear on cloud.astronomer.io/token
• Fixed an issue where the UI would initially load an inaccurate number of team members on the Access page
• Fixed alphabetical sorting by name in the People tab in the UI
• Removed placeholder columns from various tables in the UI

August 6, 2021​

Additional improvements​

• Informational tooltips are now available on the New Deployment page.

Bug fixes​

• Fixed an issue where adding a user to a Workspace and then deleting the user from Astro made it impossible to create new Deployments in that Workspace
• Improved error handling in the Airflow UI in cases where a user does not exist or does not have permission to view a Deployment

July 30, 2021​

Improvements​

• Increased the limit of Worker Resources from 30 AU to 175 AU (17.5 CPU, 65.625 GiB RAM). If your tasks require this many resources, reach out to us to make sure that your cluster is sized appropriately
• Collapsed the People and Teams tabs on the left-hand navigation bar into a single Access tab
• Added a Cluster field to the Deployments tab in the Astro UI. Now, you can reference which cluster each of your Deployments is in
• Replaced our white "A" favicon to one that supports color mode
• Informational tooltips are now available in Deployment Configuration

Bug fixes​

• Fixed an issue where a deleted user could not sign up to Astro again
• Removed Deployment-level user roles from the Astro UI. Support for them coming soon
• Fixed an issue where a newly created Deployment wouldn't show up on the list of Deployments in the Workspace