Communications Mining user guide

General fields can be of the following types:

Trainable general fields

Trainable general fields are those that will update live in the platform based on training that users provided. For more details on training general fields, checkReviewing and applying general fields.

If you enable training on a pre-trained general field that is based on a set of standard or custom-defined rules, you can refine the understanding of the platform of that general field within the parameters of those rules. Essentially, further training on these will reduce the scope of what the platform can consider that general field, but not increase it.

This is because many of these general fields, like dates, such as tomorrow, and monetary quantities, such as £20, need to be normalized into a structured data format for downstream systems. Also for general fields, such as ISINs or CUSIPs, these must have a set format, so the platform should not be taught to predict anything that does not conform to their defined formats.

When any trainable general fields are assigned, the platform looks at both the text of the general field, as well as the context of the general field within the rest of the communication, that is, what happens before and after the general field value in the same paragraph, and the one surrounding it. It learns to better predict the general field based on the values themselves, as well as how the value appears within the context of the communication.

Non-trainable general fields

If a pre-trained general field is not set as trainable, you can still accept or reject the general field predictions you identify in your dataset. These are updated and refined offline using this in-platform user feedback.

It is helpful for you to accept or reject these general fields when reviewing messages.

To learn more on how to enable general fields on a dataset, check the Enabling, disabling, updating, and creating general fields page.

When configuring general field types, you can select from one of the following pre-built options, through the template option when selecting the data type for the field type:

To enable general fields on a new dataset that you want to create, select them during the setup process.

To understand more about how to create a new dataset, check Create a new dataset.

If you want to enable, update or disable general fields for an existing dataset, you can do so from the settings tab on the top navigation bar, and then selecting the Labels and extraction fields tab.

To enable existing general fields, select inside the General Fields box, and select the general fields you want to enable from the drop down menu. Once you're happy with your selections, select Update General Fields (as shown below).

These general fields will have their settings pre-selected for you. You can then update them, including making them trainable, as shown below.

To update an enabled general field, select the general field in the general field box as shown in the previous images and the Edit general field modal will appear as shown in the following image.

Here you can update the base general field, the title of the general field and the API name (these concepts are described in detail below), as well as making the general field 'trainable'.

If you have previously reviewed general fields for a general field kind that was not set to 'trainable', this information is still stored.

To remove any selected general fields, simply select the 'X' icon next to the general field name, and then select Update General Fields.

If you make a mistake while updating the general fields, select 'Reset' before you select Update General fields and your changes will not be applied.

The previous sections covered how to enable and update existing pre-trained general fields for both new and existing datasets. In each instance, for either a new or existing dataset, you can also create new general fields.

Newly created general fields can be based on an existing pre-trained general field or can be trained from scratch like a new label.

This will bring up the Add a new general field modal as shown below.

Here you can set the field types, title, and API name, as well as select whether the general field is trainable or not. These can be updated later as previously shown.

When you've filled in each of the fields (explained below), simply select 'Create'.

API name

=======================================================================

To enable general fields on a new dataset that you want to create, select them during the setup process.

To understand more about how to create a new dataset, check Creating a new dataset.

There are now two ways to apply general field filters, and you can use them in combination with each other to create the right type of query.

The default state is the one where no filter is applied and all messages are shown, unless another filter is applied.

To update the general field filter, use the buttons explained in the following table, which also change colour when selected:

If you want to filter to messages that have any annotated general fields or predicted to contain a general field, use the buttons at the top as shown in the previous table. If you want to filter to messages with specific annotated or predicted general fields, hover over the general field in question and the same two buttons will appear to the right.

If you want to filter to either an assigned or predicted general field, select the name of the general field, and it shows messages with either one of them.

To remove your selection, select the button again, and to remove multiple selections, select All. You can also select Clear All from the the filter bar, but this will clear every filter you have selected, not just general field filters.

The taxonomy of general fields functions as a normal filter bar, and allows you to select multiple general fields at once with a single select for each.

Selecting multiple general fields from the list creates an ANY OF type query.

If you selected General field A, General field B, and General field C in the General field bar, this creates a Show me messages with General field A, General field B, or General field C predicted query.

When filtering to specific general fields, you can make multiple selections. For instance, you could filter to see messages that have an address line general field assigned OR a city general field predicted as shown in the following image.

The second filter option is the + Add General field filter button.

This enables a dropdown general field bar that allows you to select more complex filters, such as excluding certain general fields from consideration.

From this dropdown, you can select multiple general fields to include or exclude by selecting the name of the general field (for assigned and predicted), or the individual buttons (including minus for where this general field is neither assigned nor predicted).

The result looks like in this example, which returns messages predicted to have the Invoice ID general field, but not the Prod ID general field assigned or predicted:

You can select + Add General field Filter, multiple times to add additional layers to your query. Two separate general field filters create an AND type query, whilst multiple general fields selected in the same general field filter create an ANY OF type query.

In the example below, multiple general field filters have been applied individually. This creates a filter that will return messages predicted to have any of the three general fields in the first filter, but that also have the Policy Number general field predicted, and do not have the UK Postcode general field predicted or assigned.

A helpful tip is that by selecting the & sign in an individual filter containing multiple general fields, you can automatically split them out into individual filters. This would change the query from ANY OF, that is, any of these general fields predicted to AND, that is, all of these general fields predicted.

You can combine filters from both the general field bar, and individually added general field filters. Filters applied in the general field bar are treated as an AND query with any individually applied general field filters.

For example, in the image below, this combined query would return any messages that had either ORDER ID or PROD ID predicted.

Combine general field filter using general field bar and individually added general field filters.

What these filters also mean, is that you can now apply general field filters and sort by a specific general field for a training mode.

Predicted general fields appear as colour-highlighted text, such as in the first line of the message depicted in the following image, with a different colour appearing for each different general field type. Once you confirm a general field, either by manually applying it or accepting a prediction, the general field will appear as highlighted text with a bold, darker outline as shown in the following image.

If a paragraph has had general fields assigned, dismissed, or applied, it will appear highlighted in grey, as shown in the body of the message from the following image.

When reviewing trainable general fields, that the platform will learn from both the general field values that you assign, as well as the context of where they appear within the communications, that is, the other language used around the values themselves.

The platform will consider the context of the language in the same paragraph as the general field value, as well as the single paragraphs, denoted by a new separated line, directly before and after the paragraph that the general field sits in.

When the platform predicts which general fields apply to a communication, it assigns each prediction a confidence score (%) to show how confident it is that the general field applies to the highlighted span of text. You can view a general field’s confidence score by hovering over the general field.

This confidence score is also made available via the API so that it can inform automated actions taken downstream.

Once general fields are enabled, the platform will automatically start predicting them within the messages throughout your dataset. For more details, check Reviewing and applying general fields. You can then accept the predictions that are correct or reject them where they are incorrect. Each of these actions sends training signals that will be used to improve the understanding of the platform of that general field.

For the pre-trained general fields that are trained offline, such as monetary quantity, URL, and so on, it is more important from an improvement perspective for you to reject or correct wrong predictions than it is for you to accept correct predictions.

For the general fields that train live in the platform, it is equally important to accept correct predictions as well as reject incorrect predictions. You do not, however, need to keep accepting many correct examples of each unique general field for these kinds if you aren't finding incorrectly predicted ones. For example, Example Bank Ltd. is a unique organization general field.

To review a general field prediction, hover the mouse over the prediction and the general field review modal will appear, as shown in the example from the following image. To accept it, select Confirm, to reject it, select Dismiss.

You can train general fields and labels independently of each other. Reviewing labels for a message does not mean you have to review the general fields in that same message. However, it is good practice to do both at the same time, as the most efficient use of your time while model training.

To understand how well the platform can predict each general field enabled for a dataset, particularly the trainable ones, check Validation for general fields.

To apply a general field to some text where the platform may not have predicted it, users simply need to highlight the section of test like you would if you were going to copy it.

A dropdown menu will appear, as shown in the following image, containing all of the general fields that you have enabled for your dataset. Select the correct one to apply it, or press the corresponding keyboard shortcut.

The default keyboard shortcut for each general field is the letter it starts with. If more than one general field starts with the same letter, one will be assigned at random to the other.

Once a general field has been applied, it will be highlighted in colour with a bold outline as shown in the following image. Each general field type will have its own specific colour.

Do not split words

Make sure you do not split words as the highlighted general field should cover the entire word, or several, in question, not just part of it. Check the following images for an example of an incorrect and a correct application.

Do not partially annotate paragraphs

When annotating, if a user assigns one label to a message, they should apply all the labels that could apply to that message, otherwise you teach the model that those other labels should not apply. For general fields, the same is true, except general fields are reviewed or applied at the paragraph level, rather than the whole message.

Paragraphs in a message are separated by new lines. The subject line of an email message is considered its own single paragraph.

Make sure to review or apply all of the general fields within a paragraph across all general field kinds if you review or apply one of them. Applying, accepting or rejecting general fields in a paragraph means that the paragraph is treated as reviewed by the platform from a general field perspective. Therefore, make sure to accept or reject all of the predictions in that paragraph.

The following example shows the different paragraphs that have been reviewed within the email message.

The message depicted in the following image shows the same example where the user has not accepted or rejected all of the general field predictions in a single paragraph. This is incorrect, as the model will falsely treat the monetary quantity general field as an incorrect prediction.

The platform displays validation statistics, warnings and recommended actions for enabled general fields in the Validation page, much like it does for every label in your taxonomy.

To see these, navigate to the Validation page and select the General fields tab at the top, as shown in the image below.

The process in which the platform validates its ability to correctly predict general fields is very similar to how it does it for labels.

Messages are split (80:20) into a training set and a test set (determined randomly by the message ID of each message) when they are first added to the dataset. Any general fields that have been assigned (predictions that were accepted or corrected) will fall into the training set or the test set, based on whichever set the message that they're in was assigned to originally.

As there can sometimes be a very large number of general fields in one message and no guarantee whether a message is in the training set or the test set, you may see a large disparity between the number of general fields in each set.

There may also be instances where all of the assigned general fields fall into the train set. As at least one example is required in the test set to calculate the validation scores, this general field would require more assigned examples until some were present in the test set.

The individual precision and recall statistics for each general field with sufficient training data are calculated in a very similar way to that of labels:

Precision = No. of matching general fields / No. of predicted general fields

Recall = No. of matching general fields / No. of actual general fields

A 'matching general field' is where the platform has predicted the general field exactly (i.e. no partial matches)

The F1 Score is simply the harmonic mean of both precision and recall.

It's worth noting that the precision and recall stats shown in this page are most useful for the general fields that are trainable live in the platform, shown in the second column above, as all of the general fields reviewed for these general field kinds will directly impact the platform's ability to predict those general fields.

Hence accepting correct general fields and correcting or rejecting wrong general fields should be done wherever possible.

For general fields that are pre-trained via template field types, in order for the validation statistics to provide an accurate reflection of performance, users would need to ensure they accept a considerable amount of correct predictions, as well as correcting wrong ones.

If they were only to correct wrong predictions, the train and test sets would be artificially full of only the instances where the platform has struggled to predict a general field, and not those where it is better able to predict them. As correcting wrong predictions for these general fields does not lead to a real-time update of these general fields (they are updated periodically offline), the validation statistics may not change for some time and could be artificially low.

Accepting lots of the correct predictions may not always be convenient, as these general fields are predicted correctly far more often than not. But if the majority of the predictions are correct for these general fields, it's likely that you may not need worry about their precision and recall stats in the Validation page.

The summary stats (average precision, average recall and average F1 score) are simply averages of each of the individual general field scores.

Like with labels, only general fields that have sufficient training data are included in the average scores. Those that do not have sufficient training data to be included have a warning icon next to their name.

The General fieldsValidation page shows the average general field performance statistics, as well as a chart showing the average F1 score of each general field versus their training set size. The chart also flags general fields that have amber or red performance warnings.

The general field performance statistics shown are:

The general field performance chart shown in the Metrics tab of the Validation page gives an immediate visual indication of how each individual general field is performing. For more details, check the previous section.

For a general field to appear on this chart, it must have at least 20 pinned examples present in the training set used by the platform during validation. To ensure that this happens, users should make sure they provide a minimum of 25 (often more) pinned examples per general field from 25 different messages.

Each general field will be plotted as one of three colours, based on the model's understanding of how the general field is performing. Below, we explain what these mean:

General field performance indicators

Users can select individual general fields from the general field filter bar (or by selecting the general field's plot on the All general fields chart) in order to see the general field's performance statistics.

The specific general field view will also show any performance warnings and recommended next best action suggestions to help improve its performance.

The general field view will show the average F1 score for the general field, as well as its precision and recall.

Like training labels, training general fields is the process by which a user teaches the platform which general fields apply on a given message using various training modes.

Like with labels, the Teach, Check, and Missed modes are available to help train and improve the performance of general fields and can be accessed either 1) on the Explore page using the training dropdown, or 2) by following the recommended actions on the General fields tab of the Validation page.

If a specific general field has a performance warning, the platform recommends the next best action that it thinks will help address that warning, listed in order of priority. This will be shown when you select a specific general field from the taxonomy or the All general field chart.

The next best actions suggestions act as links that you can select to take you direct to the training view that the platform suggests in order to improve the general field's performance. The suggestions are intelligently ordered with the highest priority action to improve the general field listed first.

This is the most important tool to help you understand the performance of your general fields, and should regularly be used as a guide when trying to improve general field performance.

The following table summarises when the platform recommends each general field training mode:

Using Teach General field boosts general field performance, because the model is being given new information on messages it is unsure about, as opposed to ones that it already has highly confident predictions for.

The platform recommends Teach General Fields when:

The following image contains an example of training a general field in Teach General Fields mode:

Using check general field helps identify inconsistencies in the reviewed set, while improving the model's understanding of the general field, by ensuring that the model has correct and consistent examples to make predictions. This will improve the recall of a general field.

The platform recommends Check General Fields when:

For more details on calculations for general field validation, check Validation for general field.

Using missed general field helps find examples in the reviewed set that should have the selected general field but do not. It will also help identify partially annotated messages which can be detrimental to the model's ability to predict a general field. This will improve the precision of a general field and ensure the model has correct and consistent examples to make predictions from.

The platform recommends Missed General Field when:

For more details on calculations for general field validation, check the Validation for general fields page.

Use custom Regex general fields to extract and format spans of text that have a known repetitive structure, such as IDs or reference numbers.

This is a useful option for simple, structured general fields with little variation. In case of general fields with significant variation and where the context has a big influence on predictions, a machine-learning based general field is the right choice. You can use combinations of the two in any dataset within Communications Mining™.

A broader Regex (i.e., set of rules to define the general field) can also be used as the base of a custom general field. This combines the rules with contextual, machine learning based refinement through training within Communications Mining to create sophisticated custom general fields. This provides the most optimal performance as well as the necessary restrictions on values extracted for automation.

A Custom Regex general field is made up of a field type with the Regex data type, which in turn has one or more custom Regex Templates. Each template expresses one way to extract (and format) the general field.

Combined together, these templates offer a flexible and powerful way to cover multiple representations of the same general field type.

A template is made of the following:

For instance, if your customer IDs is either the ID word, followed by 7 digits, or an alphanumeric string of 9 characters. The following image shows what your two templates would look like:

The Custom Regex Template can be tested on text to ensure that it behaves as expected. Any general field that would be extracted with the Template will be shown in a list, with its value, as well as the position of the start and end characters.

The regex is the pattern used to extract general fields in the text. Check the syntax documentation.

Named capture groups can be used to identify a specific section of the extracted string for subsequent formatting. The names of the capture groups should be unique across all templates, and should only contain lowercase letters or digits.

Formatting can be provided to post-process the extracted general field.

By default, no formatting is applied and the string returned by the platform will be the string extracted by the regex. However, if needed, more complex transformations can be defined, using the following rules.

Some functions can also be used in the formatting to transform the extracted string. The names of the functions and their signatures are inspired by Excel.

Converts all characters in the extracted span to uppercase:

Converts all characters in the extracted span to lowercase:

Capitalises the extracted span:

Pads the extracted span up to a given size with a given character.

Function arguments:

Replaces characters with other characters.

Function arguments:

Returns the first n characters from the span.

Function arguments:

Returns the last n characters from the span.

Function arguments:

Returns n characters after the specified position from the span.

Function arguments: