In October, the Airflow community has been heads down building new releases, hosting events around the globe, and as always, helping one another grow in their Airflow journey. This month, we’re excited to share that we:

• Released Airflow 2.7.2

• Grew our Slack Community by 700+ members

• Hosted events in San Francisco, Washington, D.C., and Austin, TX

As always, please reach out with items for the next issue.

Thanks for reading!

Airflow 2.7.2

Airflow 2.7.2 was released on October 12th and contains fixes to security vulnerabilities.

• PyPI | Release Notes | Docs | Sources | Docker | Constraints

Providers

New versions of 18 Airflow Providers packages were released.

For all source releases, PyPI packages and docs, see: 

• Sources | PyPI | Docs

Ecosystem

Initial releases: 

• Astronomer Cosmos 1.2.1

• OpenLineage 1.4.1

• 11/2 Webinar: Run LLMOps in Production with Airflow

• 11/9 Webinar: Get Started Writing Pipelines for Any Use Case

• 11/16 Meetup in Cincinnati: ML Ops in Airflow @ Astronomer HQ

• 11/22 Meetup in London: Airflow Meetup @ Monzo

• Nov. 27th-30th Conference in Las Vegas, NV: AWS re:Invent

• 12/7 Meetup in Prague: First Airflow Prague Meetup with Kiwi.com & Astronomer

• 10/4 Meetup in Washington, D.C.: Airflow Happy Hour at City Tap

• 10/5 Webinar: Power your LLMOps with Airflow’s Weaviate Provider

• 10/19 Webinar: Airflow at Salesforce: Building a fully managed workflow orchestration system

• 10/25 Workshop in SF: BUILD.local Workshop with Snowflake

• Oct. 25th-26th Conference in Austin, TX: Generative AI World Summit

• 10/26 Webinar: The Laurel Algorithm: MLOps, AI, and Airflow for Perfect Timekeeping

@bolkedebruin

#34729 | AIP-58: Add Airflow ObjectStore

This PR of the Month received an almost unanimous vote for implementing Airflow FileSystem, which among other things, resolved many of the inconsistencies within the AWS provider.

• Customizable feature using Airflow Plugins by Giseung Beak & Ferran Fernández Garrido on Airflow Medium.

• Interviews with the Airflow 2.7 Contributors on Youtube.

• Ask Astro: An open source LLM Application with Apache Airflow® by Michael Gregory on the Astronomer Blog.

• Trigger Azure DevOps Pipeline using Apache Airflow® by Harshit Mohan on Airflow Medium. 

• Databricks vs. Airflow From a Management Perspective, Part 2 by George Yates on the Astronomer Blog.

• OpenLineage’s Big Week by Michael Robinson on Airflow Medium.

• Using Astronomer’s new Cosmos to deploy dbt pipelines onto Snowflake by Adrian Lee and George Yates on the Astronomer Blog.

• Apache Airflow® in Azure Kubernetes Services by Selva Kumar Arumugapandian and his team on Airflow Medium.

• Data Engineering End-to-End Project — Part 1 — Airflow, Kafka, Cassandra, MongoDB, Docker, EmailOperator, SlackWebhookOperator by Dogukan Ulu on Airflow Medium.

• Data Engineering End-to-End Project — Part 2 — Airflow, Kafka, Cassandra, MongoDB, Docker, EmailOperator, SlackWebhookOperator by Dogukan Ulu on Airflow Medium.

• The Best Way to Get Lineage from Airflow by Michael Robinson on Airflow Medium.

• Operationalizing Data Ingest for Retrieval Augmented Generation with LLMs, Part 3 by Michael Gregory on the Astronomer Blog.

• Creating a DAG in Apache Airflow® for Beginners: A Comprehensive Guide by Pooja Chaudhari on Airflow Medium.

• Votes: 

  • [RESULT][VOTE] Release Apache Airflow® Helm Chart 1.11.0 based on 1.11.0rc2

  • New Provider for Open Search

  • Release Airflow 2.7.2 from 2.7.2rc1

  • [RESULT][VOTE] Release Airflow 2.7.2 from 2.7.2rc1

  • Release Airflow Python Client 2.7.2 from 2.7.2rc1

  • [RESULT][VOTE] Release Airflow Python Client 2.7.2 from 2.7.2rc1

  • Airflow Providers prepared on October 13, 2023

  • [RESULT][VOTE] Airflow Providers - release of October 13, 2023

  • Airflow Providers prepared on October 18, 2023

  • [RESULT][VOTE] Airflow Providers - release of October 18, 2023

  • AIP-58 Airflow ObjectStore

  • Add PgVector & Weaviate Providers to enable first-class LLMOps

  • [RESULT] [VOTE] AIP-58 Airflow ObjectStore

  • Add providers for Pinecone, OpenAI & Cohere to enable first-class LLMOps

  • Add Multiple PR's of the Month to the Airflow Newsletter

  • [RESULT] [VOTE] Add Multiple PR's of the Month to the Airflow Newsletter

  • October 2023 PR of the Month

  • Airflow Providers prepared on October 28, 2023

  • Remove Qubole provider

   

• Discussed:

  • Future of Pendulum in Airflow

  • AIP-58 Add Airflow FS

  • Add 5 new Providers to enable first-class LLMOps

  • Requesting Feedback on Airflow Survey

  • Changing configuration options not updating airflow webserver login options

  • AIP-49 OpenTelemetry call for action/help

  • Removing Qubole provider (and adding removal process)

   

• Announced:

  • Apache Airflow® Helm Chart version 1.11.0 Released

  • Apache Airflow® 2.7.2 Released

  • FlaskCon 2023 Speaking Invitation

  • Apache Airflow® Python Client 2.7.2 Released

  • Apache Airflow® Providers prepared on October 13, 2023 are released

  • Apache Airflow® Providers prepared on October 18, 2023 are released

  • Call for Presentations now open: Community over Code EU 2024

   

• Proposed:

  • Keep Mssql support

  • Limiting (or erroring out) Airflow for Python 3.12 until our dependencies/we catch up

  • (likely) significantly shorten CI test time: splitting to db/non-db tests

  • Explain what milestones mean in our PRs/Issues - what users can expect

  • Airflow Docs Development Issues

   

• AIP-51 Removing Executor Coupling from Core Airflow

• AIP-56: Extensible user management

• See the Wiki for more of the project’s approved AIPs.

   

Did you know that the word technology was coined in 330 BC by Aristotle? In ancient times, technology was defined by Homer and Hesiod as the spoken word of manual craft or cunning skill. By 330 BC, Aristotle coined the Greek term technologia and split scientific knowledge into three parts: theoretical science, practical science, and productive science (technology). 

It’s fascinating to think how far we have come in the realm of technology, and how much we have yet to learn!

Please subscribe to this newsletter to ensure you always receive the latest issue:  http://eepurl.com/hXUA3r.

Know of an upcoming event or publication the community should know about? Let us know at viraj@astronomer.io or briana.okyere@astronomer.io.