Metric Change
Why monitor metric change?
Monitoring and measuring changes in features / raw inputs metrics allows for early detection of basic problems or changes in the model's input data.
For example - we can monitor and detect a deviation of more than 20% from the average of the feature 'age' from the average the monitor was trained with.
Detection Methods
For this monitor type, you can select the following detection methods:
- Absolute Values - The metric value is lower or higher than a specific value.
- Anomaly Detection - Detects anomalies in the value of the metric in the inspected data and its value in a time period before the data was collected.
- Change In Percentage - Detects change in the ratio between the metric value of the inspected data and its value in a time period before the data was collected.
- Compared To Segment - Detects change in the ratio between the metric value of the inspected data and to its value in a different data segment.
- Compared To Training - Detects change in the ratio between the metric value of the inspected data and its value calculated from the reported training set.
Configuration
Start from choosing the features / raw inputs you'd like to monitor. You can select as many as you want :-)
Next, choose the metric you'd like to monitor from the following options:
- Missing Count
- Average
- Minimum
- Maximum
- Sum
- Variance
- Standard Deviation
Note that the monitor configuration may vary between the detection method you choose.
Creating this monitor using the REST API
POST https://app.aporia.com/v1beta/monitors
{
"name": "Significant change in Annual_Premium variance",
"type": "metric_change",
"scheduling": "0 */4 * * *",
"configuration": {
"configuration": {
"focal": {
"source": "SERVING",
"timePeriod": "1d"
},
"metric": {
"type": "variance"
},
"actions": [
{
"type": "ALERT",
"schema": "v1",
"severity": "MEDIUM",
"alertType": "metric_change",
"description": "An anomaly in the value of the <b>'{metric}'</b> of <b>{field}</b> was detected. <br /> The anomaly was observed in the <b>{model}</b> model, in version <b>{model_version}</b> for the <b>last {focal_time_period} ({focal_times})</b> <b>{focal_segment}</b>. <br /><br /> Based on metric history average and on defined threshold, the value was expected to be below <b>{max_threshold}</b>, but <b>{focal_value}</b> was received.",
"notification": [
{
"type": "EMAIL",
"emails": [
"dev@aporia.com"
]
}
],
"visualization": "value_over_time"
}
],
"baseline": {
"source": "SERVING",
"skipPeriod": "1d",
"timePeriod": "3w",
"aggregationPeriod": "1d"
},
"logicEvaluations": [
{
"max": 1.3,
"name": "RATIO"
}
]
},
"identification": {
"models": {
"id": "seed-0000-fhpy"
},
"segment": {
"group": null
},
"features": [
"numeric_Annual_Premium"
],
"environment": null
}
}
}