Reducing Job Runtime from 2 Hours to 2 Minutes: Campspot's Migration from AWS Managed Workflows for Apache Airflow (MWAA) to Astro

Introduction

Campspot is a leader in the camping industry, specializing in campground reservation systems. Operating both B2B and B2C models, Campspot offers a comprehensive property management system for campground owners and an online booking portal for campers. With over 2700 campgrounds utilizing Campspot's services, this dual approach helps streamline operations for campground owners while providing a seamless booking experience for campers. Airflow plays a crucial role in automating various operational tasks within Campspot, enhancing the company's capability to deliver exceptional service continuously.

Transitioning to Enhanced Data Orchestration with Astro

Campspot initially adopted AWS Managed Workflows for Apache Airflow (MWAA) when they decided to implement Airflow for data orchestration. While MWAA served as the tool that introduced the team to Airflow's capabilities, Campspot encountered some limitations in terms of scalability and flexibility as their needs grew. "MWAA provided a solid starting point for us, but as we evolved, we needed more tailored configurations and isolated environments to manage our growing data operations effectively," reflected John Marriott, manager of the Data Platform team. These challenges led them to seek an alternative that could offer greater customization and support.

"Astro emerged as a better fit for our long-term goals, offering the kind of scalable environment and dedicated support we needed. The migration to Astro was streamlined and completed within a two-week sprint, significantly enhancing our data orchestration capabilities," John explained. This switch allowed Campspot to leverage Astro's advanced scheduling, monitoring, and logging capabilities more effectively, leading to improvements in operational efficiency and a reduction in job runtime. Astro’s integrated approach has since enabled Campspot to optimize their workflow deployments and manage critical data tasks more effectively, such as their nightly accounting roll-ups, which are now completed reliably and swiftly.

Astro emerged as a better fit for our long-term goals, offering the kind of scalable environment and dedicated support we needed. The migration to Astro was streamlined and completed within a two-week sprint, significantly enhancing our data orchestration capabilities.

John Marriott
Manager, Data Platform Team
Campspot

Astro’s Impact on Data Management and Developer Experience

The adoption of Astro introduced a more seamless and integrated data management experience. The platform’s intuitive interface, combined with Google Workspace integration, has facilitated easier user authentication and streamlined environment management. The Astronomer CLI tool has simplified local development, enabling the Campspot team to efficiently manage deployments. Jacob Yablonski, a data engineer at Campspot, noted the ease of deployment: "The deployment process to the environment instances is really straightforward, starting with our PR on GitHub. Just pressing the merge button initiates the deployment pipeline—only 20 lines of YAML—and it runs in GitHub Actions, deploying to the environment instance in about five to ten minutes." John appreciated the rapid updates provided by Astro, noting, 'Astro offers upgrades immediately, which helps us stay ahead with the latest features.'"

Operational Improvements and Performance Gains

Astro has brought significant performance improvements to Campspot's operations.A critical nightly roll-up job that previously took over two hours in a custom batch job setup now completes in just 2-3 minutes in Astronomer, powered by Apache Airflow, enhancing both reliability and efficiency. "Now we've got better alerting and the ability to retry if the job fails halfway through," John stated. Furthermore, the scalability of Astro has enabled Campspot to manage an increasing number of DAGs without significant cost increases.

Conclusion

The transition to Astro has been a pivotal development for Campspot, enabling the company to efficiently manage an expanding platform and deliver timely and reliable data insights to both B2B and B2C customers. "As we bring more jobs on, it's not like, 'Is this going to balloon our bill?'" John summarized, reflecting the cost efficiency and scalability that Astro has introduced. The platform's user-friendly features and robust support have transformed Campspot's approach to data management, making Astro a key component of their strategy within the competitive outdoor hospitality industry. John concluded, “We are big fans of Astro.”