Skip to content

Prediction Drift

Why monitor prediction drifts?

Prediction drift allows you to monitor a change in the distribution of the predicted label or value.

For example, a larger proportion of credit-worthy applications when your product was launched in a more affluent area. Your model still holds, but your business may be unprepared for this scenario.

Detection Methods

For this monitor type, you can select the following detection methods:

Prediction Drift Detection Methods

  • Anomaly Detection - The distribution of the inspected data is compared to the distribution of a time period before the data was collected.
  • Compared To Segment - The distribution of the inspected data is compared to the distribution of a different data segment.
  • Compared To Training - The distribution of the inspected data is compared to the distribution of the reported training data.

Configuration

Start from choosing the predictions you'd like to monitor. You can select as many as you want :-)

The monitor will compare the distributions of these predictions between the inspection period to the baseline you chose. An alert is raised if the monitor finds a drift between these distributions.

Prediction Drift Configuration

Note that the monitor configuration may vary between the detection method you choose.

You can work with the monitor preview and play with these thresholds to make sure you have a healthy amount of alerts. Note that the thresholds are different for numeric predictions vs. categorical predictions.

How are drifts calculated?

For numeric predictions, Aporia detects drifts based on the Jensen–Shannon divergence metric. For categorical predictions, we use Hellinger distance.

If you need to use other metrics, please contact us.

Creating this monitor using the REST API

POST https://app.aporia.com/v1beta/monitors
{
    "name": "Weekly prediction drift",
    "type": "prediction_drift",
    "scheduling": "0 */12 * * *",
    "configuration": {
        "configuration": {
            "focal": {
                "source": "SERVING",
                "timePeriod": "1w",
                "alignBinsWithBaseline": true
            },
            "metric": {
                "type": "histogram"
            },
            "actions": [
                {
                    "type": "ALERT",
                    "schema": "v1",
                    "severity": "HIGH",
                    "alertType": "prediction_drift_anomaly",
                    "description": "A prediction drift was detected in prediction <b>'{field}'</b>. A drift score of <b>{drift_score}</b> was detected. <br /> The drift was observed in the <b>{model}</b> model, in version <b>{model_version}</b> for the <b>last {focal_time_period} ({focal_times})</b> <b>{focal_segment}</b> compared to the <b>last {baseline_time_period} ({baseline_times})</b>. <br /><br /> Prediction drift indicates a significant change in model's behavior. In some cases, it is a strong indicator for concept drift.<br /><br /> Prediction drift might occur because: <ul><li>Natural changes in data</li><li>Data store / provider schema changes</li><li>Data store / provider issues</li><li>Data processing issues</li></ul>",
                    "notification": [
                        {
                            "type": "EMAIL",
                            "emails": [
                                "dev@aporia.com"
                            ]
                        }
                    ],
                    "visualization": "distribution_compare_chart"
                }
            ],
            "baseline": {
                "source": "SERVING",
                "skipPeriod": "1w",
                "timePeriod": "3w"
            },
            "logicEvaluations": [
                {
                    "name": "APORIA_DRIFT_SCORE",
                    "thresholds": {
                        "numeric": 0.279
                    }
                }
            ]
        },
        "identification": {
            "models": {
                "id": "seed-0000-fhpy"
            },
            "segment": {
                "group": null
            },
            "environment": null,
            "predictions": [
                "numeric_percentage"
            ]
        }
    }
}