Turning Data into Decisions: Evaluating Churn Models with Live Dashboards

A leading Cyber Security SaaS company engaged QuantSpark to provide real-time analytics on the performance of their churn and downsell neural network model. In response, QuantSpark developed a suite of dashboards to comprehensively analyse the model's performance and address critical business questions.

The Business Problem

While the Private Equity backed firm had a downsell and churn model, which was developed by QuantSpark, it lacked visibility into its actual performance across various business units and customer segments.

To answer any inquiries from the C-suite regarding the model's effectiveness, the team had to conduct extensive ad-hoc analysis, which was time-consuming and inefficient.

There was a clear need for a comprehensive suite of dashboards that could provide an in-depth evaluation of the model's performance, highlighting areas of success and opportunities for improvement.

Understanding and Developing the Core User Journeys

The first step in the process was to thoroughly understand the user journeys and core business questions that the model needed to address. This involved collaborating closely with key stakeholders, including the data science team and senior management, to identify the most critical metrics and areas of focus.

Additionally, QuantSpark developed hypothesis trees and mapped out the narratives QuantSpark wanted to convey, ensuring that the dashboards and data models would effectively paint a clear picture of the model's performance and its impact on the business.

By gaining a deep understanding of the user journeys, core questions and overall story, QuantSpark were able to ensure that the dashboards and data models were tailored to provide actionable insights and drive meaningful business impact.

Data Engineering

With a clear understanding of the requirements, QuantSpark proceeded to wireframe and build the core data models necessary to answer the identified questions.

QuantSpark leveraged the powerful capabilities of DBT (data build tool) and Snowflake to engineer and organise the required data tables, ensuring they were structured in an optimal way for efficient querying and analysis.

By carefully designing and implementing these data models, QuantSpark ensured that the underlying data was accurate, up-to-date, and structured in a way that allowed for efficient querying and analysis.

Wire-framing and Dashboard Design

Finally, QuantSpark developed a suite of real-time dashboards that integrated with the core data models, providing a comprehensive view of the model's performance. These dashboards were designed to monitor the model's efficacy in identifying at-risk customers, its ability to provide clear explanations for the identified signals, risks, and the impact of the recommended actions taken by the customer success teams based on the model's insight.

The dashboard quite was tailored to provide insightful answers to the following key questions, enabling a more thorough investigation into the reasons behind the observed trends:

1. Is the model able to successfully determine which customers are more likely to downsell & churn?
   1. How is the recall/precision trending at different time intervals to the contract end date over time?
   2. Are customers classified as high-risk churning more than those predicted low-risk?
   3. How is the model performing in different regions and customer seat segments?
      
2. Which signals and frameworks are being used by the model to determine customer risk?
   1. Are we able to identify which signals contribute most towards churn & downsell risk? Are they changing over time?
   2. Do the signals sent to CSMs correspond with the signals which drive risk?
3. Are customer engagements focussed on high-risk customers and effective for preventing downsell and churn?
   1. Are we spending more time engaging with high-risk customers?
   2. Does higher engagement lead to lower downsell and churn rates?
4. Which customers were missed by the model, and which were successfully saved?
   1. How much churn & downsell comes from the low-risk predictions? i.e. came as a surprise to the model

Impact

QuantSpark’s dashboard suite provided deeper insights into the model's performance, enabling efficient responses to board and senior management inquiries that previously time-consuming ad-hoc analysis. The real-time dashboards saved valuable time and resources.

Additionally, QuantSpark offered potential calls to action for each question addressed by the dashboards. These recommendations aimed to improve both the model's performance and the operational processes surrounding its implementation, allowing the company to optimise the model's impact on reducing customer churn and downsell.