Data engineering diagnostic and roadmap prioritisation

Profile

A B2B switching service for energy, telecoms, and insurance in the process of growing its internal data analytics team.

Situation

Our client needed an objective assessment of their data engineering stack in order to:

• Mitigate risks created by staff turnover

• Improve institutional knowledge

• Embed best practice into the rebuild of the data architecture

• Scope future phases of data engineering work

Our client’s data strategy was based on providing a customer experience with real-time personalisation in which they can anticipate customer needs to increase conversion rates. To enable this, our client required a clear data engineering roadmap for cloud-based data engineering deployment covering people, processes, and tech stack selection to make live data available to their team.

Action

We wrote comprehensive data engineering documentation based on data analysis and workshops with the client team to give a commercially focused understanding of the current state. This documentation covered architecture, data pipelines and table reporting views. This was used to assess readiness to deliver a strategy based on real-time data and assess key pain points where technical issues will throttle continued growth of the business.

With this established, we also provided a clear set of recommendations laid out in a roadmap with implementation plans and ROI for each deliverable. These deliverables covered role profiles to assess future training and hiring needs, process recommendations to encode and implement business logic consistently across the marketing and sales funnel, and tech stack selections based on best practice and availability of skills in the job market.

Tools and techniques used in this work

• Exploratory Data Analysis

• Data warehousing

• ETL

• AWS

• MySQL

• Tableau

• SalesForce

Impact

Through clear, standardised documentation of current data engineering systems and processes new team members could be onboarded in a seamless way and key-person risks were reduced in a highly competitive job market with high attrition rates.

A prioritised roadmap of data engineering deliverables will enable our client to transition to a data-led operating model with a customer experience driven by real-time personalisation.

Furthermore, risks to further growth based on an ever-growing technical debt have been significantly reduced.

So what?

QuantSpark’s hybrid approach of analytics engineering and strategy brought a commercial focus to a highly technical area of the business. Through our knowledge of best-in-class cloud engineering practices and developing real-time product recommendation engines, we ensured that a roadmap of technical work was aligned to enable the overall strategy of the business.