Aporia Documentation
Get StartedBook a DemoπŸš€ Cool StuffBlog
V1
V1
  • Welcome to Aporia!
  • πŸ€—Introduction
    • Quickstart
    • Support
  • πŸ’‘Core Concepts
    • Why Monitor ML Models?
    • Understanding Data Drift
    • Analyzing Performance
    • Tracking Data Segments
    • Models & Versions
    • Explainability
  • 🏠Storing your Predictions
    • Overview
    • Real-time Models (Postgres)
    • Real-time Models (Kafka)
    • Batch Models
    • Kubeflow / KServe
    • Logging to Aporia directly
  • πŸš€Model Types
    • Regression
    • Binary Classification
    • Multiclass Classification
    • Multi-Label Classification
    • Ranking
  • πŸ“œNLP
    • Intro to NLP Monitoring
    • Example: Text Classification
    • Example: Token Classification
    • Example: Question Answering
  • πŸͺData Sources
    • Overview
    • Amazon S3
    • Athena
    • BigQuery
    • Delta Lake
    • Glue Data Catalog
    • PostgreSQL
    • Redshift
    • Snowflake
  • ⚑Monitors
    • Overview
    • Data Drift
    • Metric Change
    • Missing Values
    • Model Activity
    • Model Staleness
    • New Values
    • Performance Degradation
    • Prediction Drift
    • Value Range
    • Custom Metric
  • πŸ“‘Integrations
    • Slack
    • JIRA
    • New Relic
    • Single Sign On (SAML)
    • Webhook
    • Bodywork
  • πŸ”‘API Reference
    • Custom Metric Definition Language
    • REST API
    • SDK Reference
    • Metrics Glossary
Powered by GitBook
On this page
  • Extract Embeddings
  • Storing your Predictions
  • Integrate to Aporia
  • Next steps
  1. NLP

Intro to NLP Monitoring

Whether it's text classification, information extraction, or question answering, use Aporia to monitor your Natural Language Processing models in production.

PreviousRankingNextExample: Text Classification

Last updated 2 years ago

This guide will walk you through the core concepts of NLP model monitoring. Before soon, you'll be able to detect drift and measure model performance for your NLP models πŸš€

Throughout the guide, we will use a simple sentiment analysis model based on πŸ€— :

>>> from transformers import pipeline

>>> classifier = pipeline("sentiment-analysis")

This downloads a default pretrained model and tokenizer for Sentiment Analysis. Now you can use the classifier on your target text:

>>> classifier("I love cookies and Aporia")
[{'label': 'POSITIVE', 'score': 0.9997883439064026}]

Extract Embeddings

To effectively detect drift in NLP models, we use embeddings.

But... what are embeddings?

Textual data is complex, high-dimensional, and free-form. Embeddings represent text as low-dimensional vectors.

Various language models, such as and transformer-based models like , are used obtain embeddings for NLP models. In case of BERT, embeddings are usually vectors of size 768.

To get embeddings for our HuggingFace model, we'll need to do two things:

  1. Pass output_hidden_states=True to our model params.

  2. When we call pipeline(...) it does a lot of things for us - preprocessing, inference, and postprocessing. We'll need to break all this, so we can interfere in the middle and get embeddings

In other words:

classifier = pipeline(
    task="sentiment-analysis",
    model_kwargs={"output_hidden_states": True}
)

# Preprocessing
model_input = classifier.preprocess("I love cookies and Aporia")

# Inference
model_output = classifier.forward(model_input)

# Postprocessing
classifier.postprocess(model_output)
  # ==> {'label': 'POSITIVE', 'score': 0.9998340606689453} 

And finally, to extract embeddings for this prediction:

embeddings = torch.mean(model_output.hidden_states[-1], dim=1).squeeze()

Storing your Predictions

For example, you could use a Parquet file on S3 or a Postgres table that looks like this:

id
raw_text
embeddings
prediction
score
timestamp

1

I love cookies and Aporia

[0.77, 0.87, 0.94, ...]

POSITIVE

0.98

2021-11-20 13:41:00

2

This restaurant was really bad

[0.97, 0.82, 0.13, ...]

NEGATIVE

0.88

2021-11-20 13:45:00

3

Hummus is the tastiest thing ever

[0.14, 0.55, 0.66, ...]

POSITIVE

0.92

2021-11-20 13:49:00

Integrate to Aporia

Now let’s add some monitoring to this model πŸš€ To monitor this model in Aporia, the first step is to create a model version:

apr_model = aporia.create_model_version(
  model_id="<MODEL_ID>",
  model_version="v1",
  model_type="multiclass"
  raw_inputs={
    "raw_text": "text",
  },
  features={
     "embeddings": {"type": "tensor", "dimensions": [768]}
  },
  predictions={
    "prediction": "string",
    "score": "numeric"
  },
)

Next, we can log predictions directly to Aporia:

classifier = pipeline(
    task="sentiment-analysis",
    model_kwargs={"output_hidden_states": True}
)


def predict(raw_text: str):
    # Run model pipeline
    model_input = classifier.preprocess(raw_text)
    model_output = classifier.forward(model_input)
    result = classifier.postprocess(model_output)
    
    # Extract embeddings
    embeddings = torch.mean(model_output.hidden_states[-1], dim=1).squeeze().tolist()
    
    # Log prediction to Aporia
    apr_model.log_prediction(
        id=str(uuid.uuid4()),
        raw_inputs={
            "raw_text": raw_text
        },
        features={
            "embeddings": embeddings
        },
        predictions={
            "prediction": result["prediction"],
            "score": result["score"]
        }
    )
    
    return result
apr_model.connect_serving(
    data_source=<DATA_SOURCE>,
    
    id_column="id",
    timestamp_column="timestamp"
)

Your model should now be integrated to Aporia! πŸŽ‰

Next steps

  • Create a custom dashboard for your model in Aporia - Drag & drop widgets to show different performance metrics, top drifted features, etc.

  • Visualize NLP drift using Aporia's Embeddings Projector - Use the UMap tool to visually see drift between different datasets in production.

  • Set up alerts to get notified for ML issues - Including data integrity issues, model performance degradation, and model drift.

The next step would be to store your predictions in a data store, including the embeddings themselves. For more information on storing your predictions, please check out the section.

Alternatively, connect Aporia to a data source. For more information, see

πŸ“œ
HuggingFace
Word2Vec
BERT
πŸ˜‰
Storing Your Predictions
Data Sources - Overview: