Intro to NLP Monitoring
Whether it's text classification, information extraction, or question answering, use Aporia to monitor your Natural Language Processing models in production.
This guide will walk you through the core concepts of NLP model monitoring. Before soon, you'll be able to detect drift and measure model performance for your NLP models 🚀
>>> from transformers import pipeline
>>> classifier = pipeline("sentiment-analysis")
This downloads a default pretrained model and tokenizer for Sentiment Analysis. Now you can use the
classifier on your target text:>>> classifier("I love cookies and Aporia")
[{'label': 'POSITIVE', 'score': 0.9997883439064026}]
To effectively detect drift in NLP models, we use embeddings.
But... what are embeddings?
Textual data is complex, high-dimensional, and free-form. Embeddings represent text as low-dimensional vectors.
To get embeddings for our HuggingFace model, we'll need to do two things:
- 1.Pass
output_hidden_states=Trueto our model params. - 2.When we call
pipeline(...)it does a lot of things for us - preprocessing, inference, and postprocessing. We'll need to break all this, so we can interfere in the middle and get embeddings 😉
In other words:
classifier = pipeline(
task="sentiment-analysis",
model_kwargs={"output_hidden_states": True}
)
# Preprocessing
model_input = classifier.preprocess("I love cookies and Aporia")
# Inference
model_output = classifier.forward(model_input)
# Postprocessing
classifier.postprocess(model_output)
# ==> {'label': 'POSITIVE', 'score': 0.9998340606689453}
And finally, to extract embeddings for this prediction:
embeddings = torch.mean(model_output.hidden_states[-1], dim=1).squeeze()
The next step would be to store your predictions in a data store, including the embeddings themselves. For more information on storing your predictions, please check out the Storing Your Predictions section.
For example, you could use a Parquet file on S3 or a Postgres table that looks like this:
id | raw_text | embeddings | prediction | score | timestamp |
|---|---|---|---|---|---|
1 | I love cookies and Aporia | [0.77, 0.87, 0.94, ...] | POSITIVE | 0.98 | 2021-11-20 13:41:00 |
2 | This restaurant was really bad | [0.97, 0.82, 0.13, ...] | NEGATIVE | 0.88 | 2021-11-20 13:45:00 |
3 | Hummus is the tastiest thing ever | [0.14, 0.55, 0.66, ...] | POSITIVE | 0.92 | 2021-11-20 13:49:00 |
Now let’s add some monitoring to this model 🚀 To monitor this model in Aporia, the first step is to create a model version:
apr_model = aporia.create_model_version(
model_id="<MODEL_ID>",
model_version="v1",
model_type="multiclass"
raw_inputs={
"raw_text": "text",
},
features={
"embeddings": {"type": "tensor", "dimensions": [768]}
},
predictions={
"prediction": "string",
"score": "numeric"
},
)
Next, we can log predictions directly to Aporia:
classifier = pipeline(
task="sentiment-analysis",
model_kwargs={"output_hidden_states": True}
)
def predict(raw_text: str):
# Run model pipeline
model_input = classifier.preprocess(raw_text)
model_output = classifier.forward(model_input)
result = classifier.postprocess(model_output)
# Extract embeddings
embeddings = torch.mean(model_output.hidden_states[-1], dim=1).squeeze().tolist()
# Log prediction to Aporia
apr_model.log_prediction(
id=str(uuid.uuid4()),
raw_inputs={
"raw_text": raw_text
},
features={
"embeddings": embeddings
},
predictions={
"prediction": result["prediction"],
"score": result["score"]
}
)
return result
apr_model.connect_serving(
data_source=<DATA_SOURCE>,
id_column="id",
timestamp_column="timestamp"
)
Your model should now be integrated to Aporia! 🎉
- Create a custom dashboard for your model in Aporia - Drag & drop widgets to show different performance metrics, top drifted features, etc.
- Visualize NLP drift using Aporia's Embeddings Projector - Use the UMap tool to visually see drift between different datasets in production.
- Set up alerts to get notified for ML issues - Including data integrity issues, model performance degradation, and model drift.
Last modified 7mo ago