Communications Mining user guide

Creating a suitable model to be deployed into a production environment requires an investment of time that is quickly paid back by the value of ongoing analytics and efficiency savings through automation.

If you do not effectively maintain a model over time, its benefits may diminish as model performance could decline without periodic supplementary training.

This is due to concept drift, which refers to the situation where the concepts a model is trying to predict can change in unforeseen ways over time, making predictions less and less accurate.

This essentially relates to how, over time, things can change in a business and the way it communicates internally, with other businesses and with its customers. If the training data of your model is no longer representative of the way your business operates today, it will perform worse when trying to identify concepts within your communications data.

Maintaining a production model is a straightforward and low-effort process. The majority of effort required has already been put in to create the training data for your model before it is deployed.

There are two main approaches to maintaining a model, both of which ensure that your model is provided with additional helpful and representative training examples:

1. Exception training

Any model used for automation purposes should have an exception process in place that identifies which messages were exceptions that the platform could not confidently or correctly identify. For more details, check Real-time automation.

This is important as it essentially allows you to quickly find and annotate the messages the platform struggled with, which improves the model's ability to predict future similar messages.

An automation process will be set up to automatically flag messages with a user property that identifies it as an exception. You can then filter in Explore to those messages and annotate them with the correct labels, to ensure that the platform can confidently and correctly identify similar messages in future.

This should form part of a regular process that aims to consistently improve the model. The more exceptions are captured and annotated, the better a model will perform over time, minimising the number of future exceptions and maximising the efficiency savings that an automation focused model enables.

2. Using the Balance and Rebalance mode

The Balance rating of your model is a component part of its Model Rating. This is a reflection of how similar, that is, representative the taining data of your model is to the dataset as a whole.

In theory, if the most recent data being added to a dataset over time is significantly different to the older data that was used to train the model, this would cause a drop in the similarity score that determines the Balance rating of your model.

When doing exception training, it is important to check if the similarity score for the model drops. If it does, this should be addressed as it could be an indication of concept drift and will mean performance in production will ultimately fall.

The simplest way to correct a drop in the similarity score is to complete some training using Rebalance mode.

To ensure that you train the most recent data that's representative of the kind of communications being received today, you can also add a timestamp filter whilst training in Rebalance, either to the last 3 or 6 months. This ensures that your model is not solely relying on training data that is old and may not reflect any changes in your business.