Skip to content

Performance Degradation

Why monitor metric change?

ML models performance often unexpectedly degrade when they are deployed in real-world domains. It is very important to track the true model performance metrics from real-world data and react in time, to avoid the consequences of poor model performance.

Causes of model's performance degradation include:

  • Input data changes (various reasons)
  • Concept drift

Detection Methods

For this monitor type, you can select the following detection methods:

Performance Degradation Detection Methods

  • Absolute Values - The metric value is lower or higher than a specific value.
  • Anomaly Detection - Detects anomalies in the value of the metric in the inspected data and its value in a time period before the data was collected.
  • Change In Percentage - Detects change in the ratio between the metric value of the inspected data and its value in a time period before the data was collected.
  • Compared To Segment - Detects change in the ratio between the metric value of the inspected data and to its value in a different data segment.
  • Compared To Training - Detects change in the ratio between the metric value of the inspected data and its value calculated from the reported training set.

Configuration

Start from choosing the predictions you'd like to monitor. You can select as many as you want :-)

Performance Degradation Configuration

Next, choose the metric you'd like to monitor from the following options:

  • Missing Count
  • Average
  • Minimum
  • Maximum
  • Sum
  • Variance
  • Standard Deviation
  • Mean Squared Error
  • Root Mean Squared Error
  • Mean Absolute Error
  • Precision
  • Recall
  • F1
  • Accuracy
  • True Positive Count
  • True Negative Count
  • False Positive Count
  • False Negative Count

Note that the monitor configuration may vary between the detection method you choose.

Creating this monitor using the REST API

POST https://app.aporia.com/v1beta/monitors
{
    "name": "Significant change in prediction average value",
    "type": "performance_degradation",
    "scheduling": "0 */4 * * *",
    "configuration": {
        "configuration": {
            "focal": {
                "source": "SERVING",
                "timePeriod": "1d"
            },
            "metric": {
                "type": "avg"
            },
            "actions": [
                {
                    "type": "ALERT",
                    "schema": "v1",
                    "severity": "HIGH",
                    "alertType": "metric_change",
                    "description": "An anomaly in the value of the <b>'{metric}'</b> of <b>{field}</b> was detected. <br /> The anomaly was observed in the <b>{model}</b> model, in version <b>{model_version}</b> for the <b>last {focal_time_period} ({focal_times})</b> <b>{focal_segment}</b>. <br /><br /> Based on metric history average and on defined threshold, the value was expected to be below <b>{max_threshold}</b>, but <b>{focal_value}</b> was received.",
                    "notification": [
                        {
                            "type": "EMAIL",
                            "emails": [
                                "dev@aporia.com"
                            ]
                        }
                    ],
                    "visualization": "value_over_time"
                }
            ],
            "baseline": {
                "source": "SERVING",
                "skipPeriod": "1d",
                "timePeriod": "1w",
                "aggregationPeriod": "1d"
            },
            "logicEvaluations": [
                {
                    "max": 1.3,
                    "name": "RATIO"
                }
            ]
        },
        "identification": {
            "models": {
                "id": "seed-0000-fhpy"
            },
            "segment": {
                "group": null
            },
            "environment": null,
            "predictions": [
                "numeric_percentage"
            ]
        }
    }
}