Deploying advanced engineering to drive process improvement

In the fast-paced world of financial services, timely and reliable data can make all the difference in gaining a competitive edge on investment opportunities.

Working with a leading asset management firm, QuantSpark was able to optimise the collection & validation of data from a key 3rd party provider, reducing the end-to-end processing time from 5 days to 1.5 days, a 70% reduction in time.

The challenge: A data bottleneck

The asset management firm had relied on manual processes to collect, clean, and validate data from a prominent fast-moving consumer goods (FMCG) market intelligence provider.

This information was used to provide a snapshot of the market with key insights on consumer spending from credit card purchase data. However the data would typically reach portfolio managers several days after its release. Where managers rely on hundreds of data points to make quantitative assessments of their position, having ready access what they need, when they need it, can make a real difference.

The solution: A data engineering transformation

To solve this problem, QuantSpark approached the project in three key steps, each designed to exhaustively cover the nuances of the complex data process flows involved.

1. Data audit - Laying the foundation for efficiency
   The first step was a comprehensive analysis of the existing data processes, built using a no-code data management tool. By thoroughly examining the workflows, the team identified areas where optimisation was necessary. This laid the groundwork for creating more efficient data flows.

2. Data engineering - Leveraging the power of automation
   At the heart of the solution was advanced data engineering, seamlessly combining Python, AWS cloud infrastructure and automated pipelines. These pipelines allowed for the direct integration of data from the data providers to interactive dashboards, whilst implementing the solution within the clients' cloud infrastructure to meet their security requirements.

3. Quality assurance and optimisation - Providing reliability and confidence
   Given the nature of the investment decisions, it was imperative that there is absolute reliability and assurance in the numbers. The team recognised and implemented a set of rigorous quality checks using Great Expectations, with automated alerts where source data does not been the expected thresholds.

The impact: Powering towards success

1. Faster data processing with rapid time enhancements
   Within the end to end process, the technical run itself was cut down from a few hours to under 30 minutes, providing a secondary benefit to engineering teams.

2. Productivity gains through efficient data validation
   By streamlining the data validation process valuable resources could be redirected towards more strategic endeavours, delivering against the bottom line.

3. Accelerated decision-making
   The value in improving data pipelines always lies with the decisions that can be made faster as a result. Making incremental improvements to how quickly portfolio managers can access useful data will assist them to make smart decisions quickly.

Embrace the automated future of financial services

In the realm of financial services, data is the cornerstone of success.

By leveraging the power of data engineering, this leading asset management firm optimised their data processes and driving efficiency.

Through modern data engineering solutions providing faster data, clear efficiency gains, and accelerated decision-making, automation has become the new normal. Contact QuantSpark today to learn how your firm can use technology and engineering to make better decisions, faster.