Bringing E-Commerce Brands and Customers Closer Together with Black Crow AI

Facilitating Intelligent Customer Acquisition

Black Crow AI revolutionizes the customer journey by using AI to automate decision making for E-commerce businesses and their digital marketing programs. Leveraging data and optimizing campaigns across Shopify, Email, SMS, social media, and other marketing channels, Black Crow AI is tasked with the unique challenge of helping its customers to target the right buyers with the right messages, at the right time so that they are most inclined to make a purchase.

“We help online brands solve their headache problems” said Will Norman, Sr. Staff Engineer at Black Crow AI. “Whether it’s helping to create a consistent view of activity as users move across different browsers, or predicting when a person is most likely to make a repeat purchase – and then proactively sending them an alert; we use data to facilitate meaningful interactions with their customers that lead to sales.”

We’re excited about the ability to make changes and test them beforehand, and feel confident that when we decide to make adjustments, we have both a trusted support partner to ensure that we’re applying best practices – but specific tooling that ensures that we’re able to stay up and running as we make those changes.

Will Norman
Sr. Staff Engineer
Black Crow AI

Building Data Products in a Complex Ecosystem

Given the broad landscape of different tools and services that span a typical brand’s E-commerce stack, Airflow is a natural choice to orchestrate inputs across the end-users’ marketing platforms and advertising channels into useful data products – like recommendation engines and dashboards.

Aware of the value of Airflow early on, and the role that it plays in building and delivering insights and offerings to customers, the Black Crow team engaged with MWAA as a measure to ensure a strong return on their investment into the platform.

“We’re pulling from Shopify – in addition to pulling from the platforms that our customers use for owned media like Facebook, TikTok, Google, Pinterest. You can imagine we have a lot of connectors where we’re pulling data from, that’s getting extracted on a daily basis – and Airflow is critical to power these ETL processes. And then once we have the data, then we start running our own internal processes and build models that leverage all of that data to deliver value to our customers.”

Using data from multiple platforms, Black Crow then feeds this data into proprietary models, which are used to predict metrics like customer lifetime value (LTV), propensity to purchase, as well as vector models that drive tailored product recommendations. Brands also have access to a dashboard that aggregates activity across their entire marketing stack to provide a single view of performance and campaign tracking.

Instead of just throwing money at a problem in the face of an outage, we can be confident that Astronomer will help us get to the root of the failure and make adjustments so that it doesn’t happen again.

Will Norman
Sr. Staff Engineer
Black Crow AI

Eliminating Risk to Facilitate Growth

With Airflow facilitating the orchestration of multiple data products, uptime and reliability emerged as critical KPIs across their stack. Customers rely on Black Crow’s dashboards and models to leverage quality and timely data for decision making; which is why Norman and his team decided to investigate new managed Airflow options after encountering instances of degraded service.

“We had tasks being marked as zombies and getting killed, when they were in fact still running” continued Norman. “This caused resources to be distributed in strange ways; and we found ourselves spending considerable time digging through docs, troubleshooting – and sometimes just throwing compute at the problem so that we could get back up and running.”

The team also recognized that they did not want to get caught in a cycle of reacting and responding to issues within their workflows, which prompted them to prioritize options where support would be a key element of the offering; as MWAA does not provide support for Airflow, where there was a need for troubleshooting and optimization.

“We knew we needed a few technical upgrades, like a more powerful scheduler and fine-grained task management, but the biggest selling point we saw in partnering with Astronomer was the level of access we would be getting to Airflow experts. Instead of just throwing money at a problem in the face of an outage, we can be confident that Astronomer will help us get to the root of the failure and make adjustments so that it doesn’t happen again.”

In addition to having expert guidance and support, Norman remarked that features like upgrade utilities and deployment rollbacks were attractive, given the safety net they create around upgrading to new Airflow versions, and reducing the risk of downtime in the event of bugs or compatibility issues.

Before migrating from MWAA, the team held off on upgrading to new Airflow releases out of concern that issues could slow down or even halt their pipelines. A few weeks after migrating to Astronomer, they made the jump from 2.7.3 to 2.9.3 and although they didn’t experience any problems during the upgrade, the team felt confident to finally make the switch with deployment rollbacks as a contingency plan.

The full migration process from MWAA to Astronomer, including Proof of Value (POV), lasted about two weeks. During the POV, field engineers at Astronomer worked closely with Black Crow’s engineering team to solve for unique connectivity requirements, and carry over 19 active DAGs (each with over 1,000 tasks) into the new environment. In addition, the team has recouped working hours previously lost to troubleshooting and responding to technical issues; and cites having approximately 20% more time to focus on building and optimizing their pipelines than before.

Unlocked for Deeper Adoption

No longer feeling burdened by the risk of outages, Norman is looking forward to deeper adoption of Airflow through Astro. With plans on the horizon to integrate their CI/CD tooling, and leverage Ephemeral Deploys to test out this configuration before putting it into production, Black Crow has identified multiple opportunities to extract new value and more efficiency with Airflow – including exploring how to improve their coordination with DBT.

“We’re excited about the ability to make changes and test them beforehand, and feel confident that when we decide to make adjustments, we have both a trusted support partner to ensure that we’re applying best practices – but specific tooling that ensures that we’re able to stay up and running as we make those changes.”