Introducing the First Generative AI Cookbook for Data Orchestration

Generative AI's effectiveness is heavily reliant on the quality and orchestration of data. Though increasingly versatile and knowledgeable, to be useful to the business, Generative AI (GenAI) models need access to rich, proprietary datasets and real-time operational data streams to create truly differentiated applications.

As discussed in our earlier post, The Dividing Line Between Generative AI Success and Failure, Apache Airflow®, a standard for data orchestration, plays a crucial role in managing complex data and machine learning workflows, enabling teams to build GenAI apps grounded with their enterprise data. The post also highlighted a number of engineering teams already using Apache Airflow®, managed by Astronomer's Astro data platform, to release enterprise grade GenAI apps faster with higher quality and at lower cost.

One of the most common questions we get asked when discussing data orchestration for GenAI is how to get started. That is what our new GenAI Cookbook is designed to answer.

Why a cookbook?

As state-of-the-art in AI advances, the stack of technologies needed to build an enterprise grade GenAI application is complex and rapidly evolving. Understanding how and where data orchestration integrates into the stack was the primary driver behind developing the cookbook.

In the cookbook, we demonstrate how Airflow is the foundation for the reliable delivery of AI applications through six common GenAI use cases:

1. Support automation
2. E-commerce product discovery
3. Product insight from customer reviews
4. Customer churn risk analysis
5. Legal document summarization and categorization
6. Dynamic cluster provisioning for image creation

For each use case we discuss its benefits to the business along with common technical challenges before presenting a detailed reference architecture.

Each reference architecture is built on a full stack of GenAI technologies — from embedding and foundation models to vector databases, search engines, retrieval frameworks, and cloud services. Don’t worry if you don’t see your own preferred technology included in a specific reference architecture. Because the Astronomer Registry curates Airflow providers for many components of the AI stack and Airflow allows for any custom Python code to run in a task, you can easily swap out one technology or cloud platform for your preferred option.

A few tasters from the cookbook

We’ve worked to incorporate a cross section of the most common generative AI use cases we encounter in the community. To give you a taster of what to expect, we’ve extracted two examples below.

Support Automation

The first use case is an example of conversational AI using GenAI to power a user-facing chatbot for answering support questions. Rather than showcase a simple prototype, in our reference architecture the chatbot learns from interactions to continuously improve its performance.

This solution uses Airflow running on Astro to orchestrate an application that includes data retrieval, embedding, and reranking for Retrieval Augmented Generation (RAG). The pipeline periodically fine-tunes a Cohere Rerank model, and uses a feedback loop to fine-tune answer generation with Cohere Command R+ over time. Vector embeddings of proprietary information are stored in a Weaviate database. Airflow’s provider ecosystem provides specialized operators to interact with Weaviate, Amazon EKS and a hook to connect to Amazon Bedrock, alongside implementation of custom Python functions and operators to transform text data with LangChain.

Customer Churn Risk Analysis: combining GenAI with traditional ML

Classifying customer churn risk using GenAI for sentiment analysis and traditional machine learning for classification is a hybrid approach that enhances prediction accuracy and helps businesses proactively target customers at risk of churning with personalized retention strategies. The LLM interprets customer interactions in-context to extract customer sentiment as a valuable feature, which is then used in combination with features derived from customer information by a traditional ML model to classify customers based on their churn risk.

This solution shows how Airflow can orchestrate both traditional MLOps with the ingestion of user information and feature creation, as well as sentiment analysis via a language model. Customer messages from Slack channels and Hubspot are combined and aggregated, Llama 3 hosted on Amazon Bedrock conducts sentiment analysis in-context which creates an additional feature for the XGBoost classifier model run with Amazon SageMaker. Churn risk scores are saved to an Amazon Redshift database which powers a dashboard.

Next steps to getting started

The six reference architectures presented in the cookbook show Apache Airflow® providing a robust data orchestration framework to operationalize and scale Generative AI workflows. From ingesting and preprocessing training data, to deploying and monitoring models in production, to governing the full AI lifecycle, Airflow enables teams to reliably integrate, run, and govern the complex data pipelines that power AI-driven applications.

Our teams are ready to run workshops with you to discuss how Airflow and Astronomer can accelerate your GenAI initiatives, so go ahead and contact us to schedule your session.