Airflow in Action: How Vibrant Planet Accelerates Climate Resilience With Geospatial Analytics and ML
At the Airflow Summit, Vibrant Planet shared how they’re leveraging Apache Airflow® to mitigate one of climate change’s biggest challenges — wildfires. Building sophisticated data pipelines that process huge volumes of geospatial data, complemented with detailed aerial and satellite imagery, Vibrant Planet provides a prioritization system to inform land management resilience and restoration efforts.
Cyrus Dukart, David Sacerdote, and Jason Bridgemohansingh — members of the company’s software and data engineering team — explored how they’ve optimized memory allocation to create self-healing pipelines that minimize manual intervention and ensure workflow reliability while reducing resource usage and cost.
Making Ecosystems More Resilient
Vibrant Planet’s mission is to help communities and ecosystems adapt to climate change by enabling large-scale, data-driven scenario planning. Using aerial imagery, geospatial data, and machine learning, their platform maps wildfire risks, simulates fire spread under different forestry management regimes, and provides actionable recommendations.
These recommendations help local governments and land managers prioritize wildfire prevention efforts—such as thinning vegetation or controlled burns—to safeguard communities and natural habitats.
Building these recommendations traditionally took years. Now the process is accelerated to months thanks to Vibrant Planet’s use of advanced geospatial modeling, ML, and analytical tools.
Figure 1: Vibrant Planet feeds geospatial data along with satellite and aerial imagery into ML models to help communities and land owners prioritize land management initiatives. Image source.
How Vibrant Planet Uses Apache Airflow
To power their platform, Vibrant Planet employs Apache Airflow to orchestrate workflows across diverse data sources, predictive models, and geospatial analyses. Their tasks involve processing large datasets representing bounded geographic areas called Hydrologic Units, or HUCs, each varying in size and complexity. Airflow’s Kubernetes Executor enables them to scale processing dynamically by allocating pods with specific resource requirements for each task.
Figure 2: Vibrant Planet uses Airflow to ingest and orchestrate the processing of complex data from many different sources to generate recommendations for wildfire management. Image source.
However, a significant challenge arose with dynamic task mapping: tasks were prone to out-of-memory (OOM) failures. With hundreds of tasks running simultaneously, manual intervention to adjust memory allocation for retries became a frequent—and frustrating—bottleneck.
Engineering Solutions to Address Out-of-Memory Challenges
Recognizing manual intervention was neither scalable or sustainable Vibrant Planet engineered two approaches to address these OOM issues:
- Self-Healing Pipelines
On detecting an OOM failure, the pipeline automatically increments the memory allocation for retries, doubling resources as needed until the task succeeds. This eliminates the need for manual reconfiguration, significantly reducing operational overhead. - Smart Initial Memory Recommendations
The team also created heuristics to intelligently predict initial memory based on task history. By logging resource usage and analyzing data characteristics, they can provide accurate memory estimates, ensuring tasks start with the right resources. For unknown tasks, they use linear regression to make reasonable estimations, relying on retries to refine allocations.
These enhancements have reduced compute costs by sizing memory requests to each task instance, preventing over-provisioning while maintaining pipeline reliability. The use of a custom decorator also simplifies adoption across teams by abstracting away tuning and configuration complexity, enabling developers to focus on building pipelines.
Want to Learn More?
Watch Vibrant Planet’s full session replay Adaptive Memory Scaling for Robust Airflow Pipelines. If podcasts are more your thing, check out Vibrant Planet’s appearance on the Astronomer Data Flowcast.
Here at Astronomer, we’ve seen multiple engineering teams build custom workarounds to improve the resilience of their Airflow workflows and pipelines. Astro, the fully managed Airflow service, offers a better solution:
- Tasked-Optimized Worker Queues: Assign tasks to specific worker sets based on computational needs for optimized resource utilization.
- Dynamic Workers: Astro dynamically scales workers up and back down to zero based on the needs of your workloads.
We have also seen an increasing number of Airflow users turn to Astronomer’s Ray and Anyscale Providers to handle their most complex AI/ML workloads. The providers enable them to seamlessly integrate with Astro, enabling them to more effectively manage complex distributed computing tasks within their workflows. They benefit from more efficient scaling, real-time monitoring, and optimized resource management, ensuring their data pipelines handle growing workloads with ease.
