Modernizing Financial Services with Astronomer: Northern Trust's Journey Away from Control-M

Company Overview: A Leader in Financial Services

Northern Trust is a financial services firm offering a broad spectrum of services, including wealth management, asset management, asset servicing, and custody. Data and workflow orchestration are critical for the internal financial warehouse, which holds vital data for balance sheets, income statements, and other financial reports. This data must be accurate and timely, underscoring the importance of reliable orchestration tools.

The support from Astronomer during the transition was comprehensive, providing technical assistance and best practices for DAG development and deployment.

Kevin Suchy
Solutions Architect
Northern Trust

Operational Challenges: The Need for Modernization

Before adopting Astronomer’s Airflow-powered orchestration platform, Northern Trust used Control-M for scheduling and IBM DataStage for data loads and transformations. These legacy tools posed several challenges: "A lot of our technology is legacy, running on old technology stacks and methodologies," explained Kevin Suchy, Solutions Architect at Northern Trust. Issues included limited control and flexibility over scheduling, dependency on a separate Control-M team for job setup and monitoring, and inadequate visibility into job execution and failures. Additionally, the lack of automated testing led to reactive management of data quality issues.

The dependency on the Control-M team meant that application teams had to submit templates for job scheduling, creating a black box scenario where they had little visibility into how schedules were configured. "In the past, with Control-M, we had to fill out a template, submit it to the Control-M team, and all that was kind of a black box to the application team," Kevin noted. This setup often resulted in discovering issues only after they had occurred, leading to inefficiencies. "We were more reactive, finding out about bad data only when our data load failed," Kevin explained.

The orchestration of complex financial reporting schedules was another significant pain point. Northern Trust's financial warehouse processes involved quarterly, annual, and month-end reporting, which required sophisticated scheduling capabilities. "Our financial warehouse has a lot of different nuances because we do quarterly reporting, annual reporting, and month-end reporting," Kevin said. Control-M's limited scheduling made it challenging to accommodate the intricate financial calendar needs, especially with varying business days and holiday calendars across the globe.

Additionally, the lack of visibility and control over job execution hindered operational efficiency. Non-technical staff found it difficult to track job statuses and understand workflow progress. "Our non-technical folks didn't really have that observability to go in and see what is run for a given day and where they are at," Kevin mentioned. This lack of transparency increased operational overhead as teams had to spend more time managing and troubleshooting workflows manually.

Control-M's limitations extended to its inability to orchestrate tasks across multiple heterogeneous platforms effectively. "The challenge with Control-M is that it's pretty limited when you need to orchestrate it with multiple different types of heterogeneous platforms," Vishal Modak, Chief Architect at Northern Trust, added. This restriction made it difficult to integrate various systems and automate end-to-end workflows seamlessly. The lack of advanced orchestration capabilities and dynamic behavior modification further compounded these challenges.

Control M's limitations are particularly evident in its immature plugins and connectors, which hinder seamless orchestration. "Control M is really a scheduling system, not an orchestration system," Vishal pointed out. In contrast, Airflow excels in orchestrating tasks across different environments, including on-prem systems and Snowflake. This capability has been crucial for Northern Trust, enabling more complex and customized workflows. Vishal explained that the flexibility of Airflow allows them to integrate various processes and manage dependencies dynamically, something that Control M could not handle. This advanced orchestration, combined with improved visibility and failure handling, has significantly benefited Northern Trust's operations, making their workflows more efficient and transparent.

With Astronomer, we’ve seen our end-to-end nightly execution time drop nearly 20%. Having Astronomer manage the orchestration in one central place rather than scheduling jobs through Control-M gives the application and support teams a lot more control and visibility into our running processes.

Kevin Suchy
Solutions Architect
Northern Trust

Strategic Decision: Transitioning to Astronomer

The decision to transition to Astronomer’s Airflow-powered orchestration platform was driven by several factors. Northern Trust needed a more flexible and robust solution that could provide greater insight and control over their data workflows. "We needed a solution that would give us more insight and control over our scheduling and dependencies," Kevin stated.

The evaluation process was comprehensive, involving a detailed comparison of Astronomer’s Airflow capabilities with Control-M and other orchestration tools. Key criteria included flexibility, visibility, ease of use, and support for complex workflows. Northern Trust also considered the need for a platform that could seamlessly integrate with their existing systems and scale to meet future demands.

Astronomer's ability to provide a unified view of the entire data workflow was a significant advantage. "We wanted to have all the scheduling and holiday calendars within our own purview to give us more control and insight," Kevin explained. The team was particularly impressed with Astronomer's advanced scheduling capabilities, which allowed for more customized and intricate scheduling to meet their specific needs. "The ability to write our own custom timetables that take into account various global holiday calendars and business days was a huge benefit," Kevin noted.

Another critical factor in the decision was Astronomer's support for automated testing and proactive data quality checks. With Astronomer, Northern Trust could integrate data quality tests directly into their pipelines, allowing for early detection and resolution of data issues. "Astronomer helped enable us to build data quality tests into our pipelines using the Great Expectations operator, which we couldn't do with our legacy architecture," Kevin said.

The decision was also influenced by Astronomer's community and support model. As active contributors to the Apache Airflow project, Astronomer provided deep expertise and timely support, ensuring a smooth transition and ongoing optimization. "The support from Astronomer during the transition was comprehensive, providing technical assistance and best practices for DAG development and deployment," Kevin affirmed.

Implementation Process: A Seamless Migration to Astronomer Software

The migration to Astronomer began in 2021, with major implementation phases in 2022 and 2023, and the new system went live in January 2024. Approximately 200 jobs were migrated to Astronomer. "Our core processes have been migrated to being scheduled and executed through Astronomer," Kevin affirmed. Astronomer provided comprehensive support throughout the transition, offering technical assistance and best practices for DAG development and deployment. The migration process involved several key steps.

The first phase involved a thorough assessment of existing workflows and dependencies. The team mapped out all the processes that needed to be migrated and identified potential challenges. This planning and assessment phase was crucial to ensure a smooth transition.Northern Trust's engineers then worked closely with Astronomer to develop and test the new DAGs. "We had a team of engineers developing the DAGs, testing them, and ensuring they met our requirements," Kevin explained. This phase also included setting up automated data quality checks using the Great Expectations operator.

To ensure a smooth transition, the team conducted parallel runs of the old and new systems. This approach allowed them to compare results and make necessary adjustments before fully switching over to the new platform. "Conducting parallel runs was essential in identifying and resolving any discrepancies," Kevin noted. Training sessions were conducted for various team members, including engineers, business analysts, and data analysts. "There is definitely a learning curve, but for baseline software engineers with an understanding of Python, the ramp-up time is quick," Kevin noted. This comprehensive training ensured that all team members were equipped to utilize the new platform effectively.

After successful parallel runs and thorough testing, the final migration was executed. The new system went live in January 2024, marking the culmination of the migration process. "The support from Astronomer during the transition was instrumental in overcoming challenges and ensuring a smooth go-live," Kevin added.

Astronomer's support was instrumental in overcoming challenges during the migration. There were challenges, particularly in managing complex dependencies and ensuring seamless integration with existing systems, but these were effectively resolved with Astronomer's support. The process of transitioning from Control-M to Astronomer involved meticulous planning, development, and testing to ensure that the new system could handle the complex workflows and dependencies of Northern Trust's financial operations.

Post-Implementation Success: Enhanced Efficiency and Control

Since adopting Astronomer, Northern Trust has experienced a notable positive impact on business operations. Kevin explained, "Our nightly processing consists of hundreds of data streams with complex orchestration and inter-source dependencies. With Astronomer, we’ve seen our end-to-end nightly execution time drop nearly 20%. Having Astronomer manage the orchestration in one central place rather than scheduling jobs through Control-M gives the application and support teams a lot more control and visibility into our running processes." This enhancement has enabled faster data processing and reporting, allowing the organization to make quicker and more informed decisions.

Operational overhead also decreased due to the platform's automated monitoring and alerting capabilities. These features significantly reduced manual interventions, allowing teams to allocate resources more efficiently and focus on higher-value tasks. "The Astronomer monitoring page is a lot more intuitive than someone going into a DataStage server," Kevin highlighted. Kevin also noted, "We have seen operational overhead significantly decrease with Astronomer. Email alerts can capture error messages and direct support members to a direct link to the failed DAG. Overall, The UI gives support teams much more traceability to pinpoint particular issues in data loads, retry tasks/DAGs as required, and seamless integration with ServiceNow for incident management." This reduction in manual effort has translated into substantial cost savings and increased operational efficiency.

Specific features of the Airflow platform, such as timetables for complex scheduling needs and integrated connectors for various data sources, have been particularly beneficial. Kevin emphasized, "Timetables have been a huge differentiator for us, especially with all the scheduling requirements we face in the financial sector. We have about five or six timetables, and those are different variations of how we schedule data loads based on varying regional calendars" The ability to create custom timetables for different scheduling needs has streamlined their orchestration processes significantly, enhancing the accuracy and reliability of scheduling their data workflows.”

Automating data workflows and proactive monitoring have also led to increased data processing and reporting efficiency. "We can now flag bad data early on and address issues proactively, which was not possible before," Kevin explained. This proactive approach to data quality has minimized disruptions and improved the accuracy of financial reporting, further demonstrating the value of Astronomer's capabilities.

The transition to Astronomer transformed how Northern Trust manages its workflows, allowing for more agile and responsive operations. "The reduction in manual interventions and enhanced workflow automation has notably cut our operational costs," Kevin noted, underscoring the impact on the company’s overall efficiency and financial performance. The streamlined deployment process through the Astro CLI has also enabled faster time-to-market for new data workflows, supporting the team's agility in responding to changing business needs. "The Astro CLI makes our lives a whole lot easier. We can package all our DAGs and deploy them with a simple command," Kevin said.

Furthermore, the transition has improved collaboration among team members. "Everyone uses it in different ways, from engineers developing DAGs to business analysts monitoring processes," Kevin explained. The platform's intuitive interface and comprehensive monitoring capabilities have empowered non-technical users to gain insights into data workflows, enhancing cross-functional collaboration and decision-making across the organization.

Looking Ahead: Future Plans and Long-Term Impact

Northern Trust plans to further utilize Airflow for additional data workflows and processes and is exploring Astronomer's SaaS offerings for greater simplification and scalability. "We see this platform supporting our business growth and innovation," Kevin affirmed. They advise other companies considering a similar transition to thoroughly evaluate their challenges and requirements, engage with technical support, and focus on training and onboarding to ensure a smooth transition.

Reflecting on the transition, Northern Trust acknowledges the significant benefits they have gained. "The transition has been highly beneficial, with significant improvements in efficiency and control," Kevin concluded. With Astronomer's support and the robust features, Northern Trust has successfully modernized its data workflows, laying a solid foundation for future growth and innovation.