- Introduction
- Setting up your account
- Balance
- Clusters
- Concept drift
- Coverage
- Datasets
- General fields
- Labels (predictions, confidence levels, label hierarchy, and label sentiment)
- Models
- Streams
- Model Rating
- Projects
- Precision
- Recall
- Annotated and unannotated messages
- Extraction Fields
- Sources
- Taxonomies
- Training
- True and false positive and negative predictions
- Validation
- Messages
- Access Control and Administration
- Manage sources and datasets
- Understanding the data structure and permissions
- Creating or deleting a data source in the GUI
- Uploading a CSV file into a source
- Preparing data for .CSV upload
- Creating a dataset
- Multilingual sources and datasets
- Enabling sentiment on a dataset
- Amending dataset settings
- Deleting a message
- Deleting a dataset
- Exporting a dataset
- Using Exchange integrations
- Model training and maintenance
- Understanding labels, general fields, and metadata
- Label hierarchy and best practices
- Comparing analytics and automation use cases
- Turning your objectives into labels
- Overview of the model training process
- Generative Annotation
- Dastaset status
- Model training and annotating best practice
- Training with label sentiment analysis enabled
- Training chat and calls data
- Understanding data requirements
- Train
- Introduction to Refine
- Precision and recall explained
- Precision and Recall
- How validation works
- Understanding and improving model performance
- Reasons for label low average precision
- Training using Check label and Missed label
- Training using Teach label (Refine)
- Training using Search (Refine)
- Understanding and increasing coverage
- Improving Balance and using Rebalance
- When to stop training your model
- Pinning and tagging a model version
- Deleting a pinned model
- Adding new labels to existing taxonomies
- Maintaining a model in production
- Model rollback
- Using general fields
- Generative extraction
- Using analytics and monitoring
- Automations and Communications Mining™
- Developer
- Exchange Integration with Azure service user
- Exchange Integration with Azure Application Authentication
- Exchange Integration with Azure Application Authentication and Graph
- Fetching data for Tableau with Python
- Elasticsearch integration
- Self-hosted Exchange integration
- UiPath® Automation Framework
- UiPath® Marketplace activities
- UiPath® official activities
- How machines learn to understand words: a guide to embeddings in NLP
- Prompt-based learning with Transformers
- Efficient Transformers II: knowledge distillation & fine-tuning
- Efficient Transformers I: attention mechanisms
- Deep hierarchical unsupervised intent modelling: getting value without training data
- Fixing annotating bias with Communications Mining™
- Active learning: better ML models in less time
- It's all in the numbers - assessing model performance with metrics
- Why model validation is important
- Comparing Communications Mining™ and Google AutoML for conversational data intelligence
- Licensing
- FAQs and more
Communications Mining user guide
Maintaining a model in production
Creating a suitable model to be deployed into a production environment requires an investment of time that is quickly paid back by the value of ongoing analytics and efficiency savings through automation.
If you do not effectively maintain a model over time, its benefits may diminish as model performance could decline without periodic supplementary training.
This is due to concept drift, which refers to the situation where the concepts a model is trying to predict can change in unforeseen ways over time, making predictions less and less accurate.
This essentially relates to how, over time, things can change in a business and the way it communicates internally, with other businesses and with its customers. If the training data of your model is no longer representative of the way your business operates today, it will perform worse when trying to identify concepts within your communications data.
Maintaining a production model is a straightforward and low-effort process. The majority of effort required has already been put in to create the training data for your model before it is deployed.
There are two main approaches to maintaining a model, both of which ensure that your model is provided with additional helpful and representative training examples:
- Exception training
- Using Rebalance mode
1. Exception training
Any model used for automation purposes should have an exception process in place that identifies which messages were exceptions that the platform could not confidently or correctly identify. For more details, check Real-time automation.
This is important as it essentially allows you to quickly find and annotate the messages the platform struggled with, which improves the model's ability to predict future similar messages.
An automation process will be set up to automatically flag messages with a user property that identifies it as an exception. You can then filter in Explore to those messages and annotate them with the correct labels, to ensure that the platform can confidently and correctly identify similar messages in future.
This should form part of a regular process that aims to consistently improve the model. The more exceptions are captured and annotated, the better a model will perform over time, minimising the number of future exceptions and maximising the efficiency savings that an automation focused model enables.
2. Using the Balance and Rebalance mode
The Balance rating of your model is a component part of its Model Rating. This is a reflection of how similar, that is, representative the taining data of your model is to the dataset as a whole.
In theory, if the most recent data being added to a dataset over time is significantly different to the older data that was used to train the model, this would cause a drop in the similarity score that determines the Balance rating of your model.
When doing exception training, it is important to check if the similarity score for the model drops. If it does, this should be addressed as it could be an indication of concept drift and will mean performance in production will ultimately fall.
The simplest way to correct a drop in the similarity score is to complete some training using Rebalance mode.
To ensure that you train the most recent data that's representative of the kind of communications being received today, you can also add a timestamp filter whilst training in Rebalance, either to the last 3 or 6 months. This ensures that your model is not solely relying on training data that is old and may not reflect any changes in your business.
