With the launch of Airflow 2.9.0, the Airflow Community kicked off April with high spirits. This month we’re thrilled to announce:

• The release of Airflow 2.9.0

• The closing of our Call for Presentations for Airflow Summit

• And the launch of our Lisbon Chapter!

As always, please reach out with items for the next issue.

Thanks for reading!

Tickets are on sale for Airflow Summit 2024! Book yours here to join Airflow leaders and practitioners for 3 days of amazing talks, after parties, and networking in beautiful San Francisco.

The next Airflow Monthly Virtual Town Hall will be held on Wednesday, May 8th. During the session, we will deep dive into a demo on the new heartbeat recovery message, and hear how CustomInk utilizes Airflow in an interview! Please register here if you haven’t already.

Airflow 2.9.0 is officially here, and includes new features for data-aware scheduling and a lot of UI-related improvements.

Feature Highlights

• Logical operators and conditional expressions for DAG scheduling

• Combining Dataset and Time-Based Schedules

• Dataset Event REST API endpoints

• Dataset UI Enhancements

• Custom names for Dynamic Task Mapping

• Object Storage as XCom Backend

• Display names for DAGs and Tasks

• Task log grouping

• UI Modernization

Thanks to everyone who contributed to this release, including Amogh Desai, Andrey Anshin, Brent Bovenzi, Daniel Standish, Ephraim Anierobi, Hussein Awala, Jarek Potiuk, Jed Cunningham, Jens Scheffler, Tzu-ping Chung, Vincent Beck, Wei Lee, and over 120 others!

Learn more about the 2.9.0 release here

Airflow 2.9.0

Airflow 2.9.0 was released on April 8th and includes several exciting new features. See above for details.

• PyPI | Release Notes | Docs | Sources | Docker | Constraints

Providers

New versions of 49 Airflow Providers packages were released.

For all source releases, PyPI packages and docs, see: 

• Sources | PyPI | Docs

Ecosystem

Recent releases: 

• OpenLineage 1.13.1

• 5/8 Webinar: Airflow Monthly Virtual Town Hall

• 5/11 Meetup in Seoul: Korea Apache Airflow® Meetup at Elice-Lab

• 5/15 Webinar: Reliable Data Orchestration for AI Applications

• 5/15 Meetup in NYC: NYC Airflow Rooftop Happy Hour ft. PMC Member Jarek Potiuk!

• 5/15 Meetup in San Francisco: Bay Area Airflow Meetup at Amazon!

• 5/16: Meetup in Chicago: Chicago Airflow Meetup at mHUB

• 6/6 Conference in San Francisco: Snowflake Dev Day

• 6/3-5 Conference in Bratislava: Community Over Code

• 6/10-13 Conference in San Francisco: Data and AI Summit

• 9/10-12 Conference in San Francisco: Airflow Summit 2024

• 4/3 Webinar: Airflow Monthly Town Hall

• 4/9-11 Conference in Las Vegas: Google Cloud Next Startup Showcase

• 4/11 Webinar: What’s new in Airflow 2.9

• Exploring: Data-Aware Orchestration with Apache Airflow® Datasets by Kenny Peeters on Airflow Medium

• Orchestrating the Training of Online ML Models with Airflow at X3M by Mauricio Anderson on Airflow Medium

• Introducing Apache Airflow® 2.9 by Kenten Danas on The Astronomer Blog

• A Beginner's Guide to Orchestrating Pipelines With Apache Airflow® and Teradata by Satish Chinthanippu on the Teradata Blog

• Data Aware Scheduling: A Case Study of Custom Ink by Tom Stark and Michael Peteuil on Airflow Medium

• What’s new in Airflow 2.9 webinar recap

• Apache Airflow® 2.9.0: Dataset and UI Improvements by Jed Cunningham on The Airflow Blog

• Utilizing DockerOperator in Airflow to Run Containerized Applications in Data Engineer Projects by Rio Dwi Putra Perkasa on Airflow Medium

• Teradata and Apache Airflow® Integration on Youtube

• Optimizing Integration of Microsoft Graph API and Power BI with the MSGraphAsyncOperator in Airflow by David Blain on Airflow Medium

• Optimizing ML/AI Workflows with Essential Airflow Features webinar recap

Votes: 

• [RESULT][VOTE] Airflow Providers - release of March 25, 2024

• [RESULT][VOTE] AIP-62 Getting Lineage from Hook Instrumentation

• Release Airflow 2.9.0 from 2.9.0rc1

• Release Airflow 2.9.0 from 2.9.0rc2

• Release Airflow 2.9.0 from 2.9.0rc3

• [RESULT][VOTE] Release Airflow 2.9.0 from 2.9.0rc3

• Airflow Providers prepared on April 10, 2024

• [RESULT][VOTE] Airflow Providers - release of April 10, 2024

• Airflow Providers prepared on April 13, 2024

• [RESULT][VOTE] Airflow Providers - release of April 13, 2024

• Team id as dag_id prefix: (was [VOTE] AIP-67)

• Airflow Providers prepared on April 16, 2024

• [RESULT][VOTE] Airflow Providers - release of April 16, 2024

• AIP-67 Multi-team deployment of Airflow components

• April 2024 PR of the Month

• Release Apache Airflow® Python Client 2.9.0 from 2.9.0rc1

• Release Airflow 2.9.1 from 2.9.1rc1

   

Discussed:

• Redis licencing changes impact

• Asynchronous SQLAlchemy

• Consider disabling self-hosted runners for commiter PRs

• DRAFT AIP-68 Extended Plugin Interface + AIP-69 Remote Executor

• Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

   

Announced:

• Apache Airflow® Providers prepared on March 25, 2024 are released

• Participate in the ASF 25th Anniversary Campaign

• Community over Code EU 2024: Start planning your trip!

• Deadlines inside! Apply for a scholarship

• Apache Airflow® Providers prepared on April 10, 2024 are released

• Apache Airflow® Providers prepared on April 13, 2024 are released

• Help on Connexion 3 migration needed

• Apache Airflow® Providers prepared on April 16, 2024 are released

• To Understand the requirements regarding TRAINING & TRAINING CONTENT in your ASF project

Proposed:

• Keep >= LATEST MINOR for all providers using common.sql provider

• AIP-44: Airflow Internal API

• AIP-49: Open Telemetry Support

• AIP-51 Removing Executor Coupling from Core Airflow

• AIP-57: Refactor SLA feature

• AIP-58: Airflow ObjectStore (AS)

• AIP-59: Performance test framework

• AIP-61 Hybrid Execution

• AIP-62 Getting Lineage from Hook Instrumentation

• AIP-63: DAG Versioning

• AIP-64: Keep TaskInstance try history

• AIP-65: Improve DAG history in UI

• AIP-66: Execution of specific DAG versions

• AIP-67 Multi-tenant deployment of Airflow components

• See the Wiki for more of the project’s approved AIPs.

Did you know that cows have best friends? Like humans, cows can form deep bonds with other cows and develop friendships which can last their whole lives. Research has shown they can exhibit signs of stress when they're not near their friend.

A good reminder to give your best friend a call! 

Source: Beano

Please subscribe to this newsletter to ensure you always receive the latest issue:  http://eepurl.com/hXUA3r.

Know of an upcoming event or publication the community should know about? Let us know at viraj@astronomer.io or briana.okyere@astronomer.io.