Intro to NLP Monitoring

Whether it's text classification, information extraction, or question answering, use Aporia to monitor your Natural Language Processing models in production.

This guide will walk you through the core concepts of NLP model monitoring, including drift detection and model performance. 🚀

Throughout the guide, we will use a simple sentiment analysis model based on 🤗 HuggingFace:

>>> from transformers import pipeline

>>> classifier = pipeline("sentiment-analysis")

This downloads a default pre-trained model and tokenizer for Sentiment Analysis. Now you can use the classifier on your target text:

>>> classifier("I love cookies and Aporia")
[{'label': 'POSITIVE', 'score': 0.9997883439064026}]

Extract Embeddings

To effectively detect drift in NLP models, we use embeddings.

But... what are embeddings?

Textual data is complex, high-dimensional, and free-form. Embeddings represent text as low-dimensional vectors.

Various language models, such as Word2Vec and transformer-based models like BERT, are used obtain embeddings for NLP models. In case of BERT, embeddings are usually vectors of size 768.

To get embeddings for our HuggingFace model, we'll need to do two things:

1. Pass output_hidden_states=True to our model params.

2. When we call pipeline(...) it does a lot of things for us - preprocessing, inference, and post processing. We need to break all of this down into each step, so we can extract the embeddings.

In other words:

classifier = pipeline(
    task="sentiment-analysis",
    model_kwargs={"output_hidden_states": True}
    )

# Preprocessing
model_input = classifier.preprocess("I love cookies and Aporia")

# Inference
model_output = classifier.forward(model_input)

# Postprocessing
classifier.postprocess(model_output)
  # ==> {'label': 'POSITIVE', 'score': 0.9998340606689453} 

And finally, to extract embeddings for this prediction:

embeddings = torch.mean(model_output.hidden_states[-1], dim=1).squeeze()

Storing your Predictions

The next step would be to store your predictions in a data store, including the embeddings themselves. For more information on storing your predictions, please check out the Storing Your Predictions section.

For example, you could use a Parquet file on S3 or a Postgres table that looks like this:

id	raw_text (text)	embeddings (embedding)	prediction (boolean)	score (numeric)	timestamp (datetime)
1	I love cookies and Aporia	[0.77, 0.87, 0.94, ...]	True	0.98	2021-11-20 13:41:00
2	This restaurant was really bad	[0.97, 0.82, 0.13, ...]	False	0.88	2021-11-20 13:45:00
3	Hummus is the tastiest thing ever	[0.14, 0.55, 0.66, ...]	True	0.92	2021-11-20 13:49:00

• Note that in the prediction column True is the Positive sentiment, and the false is the Negative.

To integrate this type of model follow our Quickstart.

Check out the data sources section for more information about how to connect from different data sources.

Schema mapping

There are 2 unique types in Aporia to help you integrate your NLP model - text, and embedding.

The text should be used with your raw_text column. Note that by default, in the UI every string column will be automatically marked as categorical, but you'll have the option to change it to text for NLP use cases.

The embedding as the name suggested, should be used with your embedding column. Note that by default, in the UI every array column will be automatically marked as array, but you'll have the option to change it to embedding for NLP use cases.

Next steps

• Create a custom dashboard for your model in Aporia - Drag & drop widgets to show different performance metrics, top drifted features, etc.

• Visualize NLP drift using Aporia's Embeddings Projector - Use the Embedding Projector widget within the investigation room, to view drift between different datasets in production, using UMAP for dimension reduction.

• Set up monitors to get notified for ML issues - Including data integrity issues, model performance degradation, and model drift.