Skip to content

Model Activity

Why monitor model activity?

In many cases, the number of predictions of a model is within a predictable range. Identifying deviations from the range can indicate on underlying problems, anomalous events, or an ongoing trend that is worth noting.

Causes of change in the number of predictions include:

  • Natural increase in model invocations
  • Serving environment fault
  • Malicious attempt to analyse model behaviour

Detection Methods

For this monitor type, you can select the following detection methods:

Model Activity Detection Methods

  • Absolute Values - The prediction count is lower or higher than a specific value.
  • Anomaly Detection - Detects anomalies in the prediction count of the inspected data, compared to the prediction count in a time period before the data was collected.
  • Change In Percentage - Detects change in the ratio between the prediction count of the inspected data and the prediction count in a time period before the data was collected.

Configuration

Model Activity Configuration

Note that the monitor configuration may vary between the detection method you choose.

Creating this monitor using the REST API

POST https://app.aporia.com/v1beta/monitors
{
    "name": "Hourly Predictions > 100",
    "type": "model_activity",
    "scheduling": "*/5 * * * *",
    "configuration":  {
        "configuration": {
            "focal": {
                "source": "SERVING",
                "timePeriod": "1h"
            },
            "metric": {
                "type": "count",
                "field": "_id"
            },
            "actions": [
                {
                    "type": "ALERT",
                    "schema": "v1",
                    "severity": "MEDIUM",
                    "alertType": "model_activity_threshold",
                    "description": "An anomaly in the number of total predictions within the defined limits was detected.<br />The anomaly was observed in the <b>{model}</b> model, in version <b>{model_version}</b> for the <b>last {focal_time_period} ({focal_times})</b> <b>{focal_segment}</b>.<br /><br />Based on defined limits, the count was expected to be above <b>{min_threshold}</b>, but <b>{focal_value}</b> was received.<br />",
                    "notification": [
                        {
                            "type": "EMAIL",
                            "emails": [
                                "dev@aporia.com"
                            ]
                        }
                    ],
                    "visualization": "value_over_time"
                }
            ],
            "logicEvaluations": [
                {
                    "max": null,
                    "min": 100,
                    "name": "RANGE"
                }
            ]
        },
        "identification": {
            "models": {
                "id": "seed-0000-5wfh"
            },
            "segment": {
                "group": null
            },
            "environment": null
        }
    }
}