Overview
Monitoring your Machine Learning models begins with storing their inputs and outputs in production.
Oftentimes, this data is used not just for model monitoring, but also for retraining, auditing, and other purposes; therefore, it is crucial that you have complete control over it.
Aporia monitors your models by connecting directly to your data, in your format. This section discusses the fundamentals of storing model predictions.
Depending on your existing enterprise data lake infrastructure, performance requirements, and cloud costs constraints, storing your predictions can be done in a variety of data stores.
Here are some common options:
- Parquet files on S3 / GCS / ABS
When storing your predictions, it's highly recommended to adopt a standardized directory structure (or SQL table structure) across all of your organization's models.
With a standardized structure, you'll be able to get all models onboarded to the monitoring system automatically.
Here is a very basic example:
s3://myorg-models/
├── my-model/
├── v1/
│ ├── train.parquet
│ ├── test.parquet
│ ├── serving.parquet
│ ├── artifact.onnx
├── v2/
│ ├── train.parquet
│ ├── test.parquet
│ └── serving.parquet
│ └── artifact.onnx
Even though this section focuses on the storage of predictions, you should also consider saving the training and test sets of your models. They can serve as a monitoring baseline.
Recommendations:
- One row per prediction.
- One column per feature, prediction or raw input.
- Use a prefix for column names to identify their group (e.g
features.,raw_inputs.,predictions.,actuals., etc.) - For serving, add ID and prediction timestamp columns.
Example:
+-----+----------------------+-------------------+---------------+----------------+-------------------+-------------------------+--------------+----------------------+------------------------+
| id | timestamp | predictions.score | actuals.score | raw_inputs.age | raw_inputs.gender | features.my_embeddings | features.age | features.gender_male | features.gender_female |
+-----+----------------------+-------------------+---------------+----------------+-------------------+-------------------------+--------------+----------------------+------------------------+
| 1 | 2022-10-19T14:21:08Z | 0.58 | 0.59 | 64 | male | [0.58, 0.19, 0.38, ...] | 64 | 1 | 0 |
| 2 | 2022-10-19T14:21:08Z | 0.64 | 0.66 | 62 | woman | [0.48, 0.20, 0.42, ...] | 62 | 0 | 1 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
+-----+----------------------+-------------------+---------------+----------------+-------------------+-------------------------+--------------+----------------------+------------------------+