FreightWaves scales Data Engineering with Astronomer and Apache Airflow

Challenge:

• Scattered Microservices Setup: Managing hundreds of bespoke Python scripts and cloud functions became operationally unscalable.
• Painful Development Cycle: Slow, manual iterations in Google Cloud Composer hindered agility and development speed.
• Operational Risk: Lack of workflow reliability threatened timely updates, crucial for FreightWaves customers’ decision-making.

Solution:

FreightWaves migrated to Astro, Astronomer’s fully managed Apache Airflow service, which provided:

• Rich Ecosystem of Provider Packages for integrating with diverse data sources.
• Local Development Environments for faster iterations and developer satisfaction.
• Centralized Workflow Management to simplify scaling and maintenance.

Results:

• Faster Onboarding: Streamlined integration of new data partners, accelerating time to value.
• Enhanced Developer Productivity: Faster iterations unlocked resources for innovation and experimentation.
• Improved Reliability: Eliminated downtime in daily updates, ensuring customers received uninterrupted insights.

Astronomer has truly transformed our workflow. We can iterate faster, deliver greater business impact, and onboard new data sources seamlessly.

Eric Crowley
Lead Data Engineer
FreightWaves

Overview

FreightWaves, the “Bloomberg of Freight,” provides real-time market intelligence for the logistics industry, offering metrics and data-driven insights to customers via their platform. As the logistics marketplace relies on FreightWaves for daily decision-making, delivering timely and reliable data is critical. Eric Crowley, Lead Data Engineer at FreightWaves, shared how the company transitioned from a scattered and inefficient setup to Astronomer’s Astro, powered by Apache Airflow, to transform their data engineering practices.

Challenges

Before adopting Airflow, FreightWaves relied on numerous bespoke Python scripts and cloud functions for data orchestration. Over time, this approach revealed significant challenges such as managing hundreds of pipelines across disparate setups created inefficiencies and scalability. Updating libraries or scaling required changes across numerous scripts and functions, consuming significant resources. The inability to centralize pipeline monitoring and updates also slowed down operations.

FreightWaves initially adopted Google Cloud Composer but faced additional obstacles like slow iterations and rigid infrastructure. Developers had to upload changes to a cloud bucket and wait for DAGs to parse, hampering development cycles. Modifying OS-level packages or configurations was cumbersome and resource-intensive. FreightWaves’ reliance on timely data delivery meant operational delays had direct consequences for their customers, who needed daily updates to make critical decisions.

Solution: Migration to Astronomer and Apache Airflow

FreightWaves selected Astronomer’s Astro for its robust feature set, which addressed their needs for agility, scalability, and developer satisfaction. ** ** Astro enabled engineers to iterate faster by testing workflows locally before deployment. With Astro, FreightWaves could manage and monitor all workflows from a unified interface, improving visibility and operational control. The open-source ecosystem of Airflow operators allowed FreightWaves to integrate seamlessly with new data sources, like Snowflake, without needing bespoke solutions. Astro supported FreightWaves’ rapid growth, enabling efficient onboarding of new data sources and partners.

Migrating to Astro was supported by Astronomer’s documentation and Astro CLI, which minimized disruptions and ensured a seamless transition. Developers could upgrade workflows and onboard new team members with ease, further accelerating the adoption of Airflow across the organization.

Outcomes

FreightWaves eliminated downtime and disruptions in their daily data updates. This ensured that customers continued to receive timely insights critical for their operations. The ability to iterate locally and deploy quickly increased developer productivity, allowing the team to deliver more value in less time. Astronomer’s flexible architecture enabled FreightWaves to onboard new data partners rapidly. With reusable, schema-agnostic pipelines, integrating new data sources became seamless, reducing time to market.

FreightWaves ensured uninterrupted data delivery, reinforcing customer trust. Developers had the freedom and tools to innovate, leading to the creation of high-value data products. The migration supported FreightWaves’ plans for expansion.

Astronomer’s local dev environments and documentation have been game-changers. We’re able to iterate faster, onboard new data sources effortlessly, and deliver value to our customers consistently.

Eric Crowley
Lead Data Engineer
FreightWaves

Ready to scale your data engineering operations like FreightWaves?

Start your free trial of Astro today and unlock the power of Apache Airflow with Astronomer.