Metric Change

Why monitor metric change?

Monitoring and measuring changes in features / raw inputs metrics allows for early detection of basic problems or changes in the model's input data.

For example - we can monitor and detect a deviation of more than 20% from the average of the feature 'age' from the average the monitor was trained with.

Detection Methods

For this monitor type, you can select the following detection methods:

• Absolute Values - The metric value is lower or higher than a specific value.
• Anomaly Detection - Detects anomalies in the value of the metric in the inspected data and its value in a time period before the data was collected.
• Change In Percentage - Detects change in the ratio between the metric value of the inspected data and its value in a time period before the data was collected.
• Compared To Segment - Detects change in the ratio between the metric value of the inspected data and to its value in a different data segment.
• Compared To Training - Detects change in the ratio between the metric value of the inspected data and its value calculated from the reported training set.

Configuration

Start from choosing the features / raw inputs you'd like to monitor. You can select as many as you want :-)

Next, choose the metric you'd like to monitor from the following options:

• Missing Count
• Average
• Minimum
• Maximum
• Sum
• Variance
• Standard Deviation

Note that the monitor configuration may vary between the detection method you choose.

Creating this monitor using the REST API

POST https://app.aporia.com/v1beta/monitors
{
    "name": "Significant change in Annual_Premium variance",
    "type": "metric_change",
    "scheduling": "0 */4 * * *",
    "configuration": {
        "configuration": {
            "focal": {
                "source": "SERVING",
                "timePeriod": "1d"
            },
            "metric": {
                "type": "variance"
            },
            "actions": [
                {
                    "type": "ALERT",
                    "schema": "v1",
                    "severity": "MEDIUM",
                    "alertType": "metric_change",
                    "description": "An anomaly in the value of the <b>'{metric}'</b> of <b>{field}</b> was detected. <br /> The anomaly was observed in the <b>{model}</b> model, in version <b>{model_version}</b> for the <b>last {focal_time_period} ({focal_times})</b> <b>{focal_segment}</b>. <br /><br /> Based on metric history average and on defined threshold, the value was expected to be below <b>{max_threshold}</b>, but <b>{focal_value}</b> was received.",
                    "notification": [
                        {
                            "type": "EMAIL",
                            "emails": [
                                "dev@aporia.com"
                            ]
                        }
                    ],
                    "visualization": "value_over_time"
                }
            ],
            "baseline": {
                "source": "SERVING",
                "skipPeriod": "1d",
                "timePeriod": "3w",
                "aggregationPeriod": "1d"
            },
            "logicEvaluations": [
                {
                    "max": 1.3,
                    "name": "RATIO"
                }
            ]
        },
        "identification": {
            "models": {
                "id": "seed-0000-fhpy"
            },
            "segment": {
                "group": null
            },
            "features": [
                "numeric_Annual_Premium"
            ],
            "environment": null
        }
    }
}