Links

Monitors

This guide will show you how to automatically add monitors to your models from code using the Python SDK. For more information on the various types of monitors in Aporia, please refer to the documentation on monitors.

Defining monitors

To define a new monitor, create an aporia.Monitor(...) object.

Step 1: Choose monitor type and detection method

When designing a new monitor, the first decision you have to make is:
  • What's the monitor type you'd like to create?
    • Examples: Data drift, Missing values, Performance degradation, etc.
    • This step is similar to the following step in the UI:
  • What's the detection method you'd like to use?
    • Examples: Anomaly Detection over Time, Change in Percentage, Absolute values, Compared to Training, Compared to Segment, etc.
    • This step is similar to the following step in the UI:
To begin, start by creating a monitor object, with your chosen monitor_type and detection_method, and add it to your model.
See Detection Methods Overview and Supported Monitor Types / Detection Methods for the an overview on different monitor types and their supported detection methods.
import aporia.as_code as aporia
stack = aporia.Stack(...)
data_drift = aporia.Monitor(
"Data Drift - Last week compared to Training",
monitor_type=aporia.MonitorType.DATA_DRIFT,
detection_method=aporia.DetectionMethod.COMPARED_TO_TRAINING,
...
)
my_model = aporia.Model(
"My Model",
type=aporia.ModelType.RANKING,
icon=aporia.ModelIcon.RECOMMENDATIONS,
...,
monitors=[data_drift],
)
stack.add(my_model)

Step 2: Choose focal and baseline datasets

The next step is to choose the dataset in which your monitor will be evaluated - this is called the focal dataset. In most detection methods, you'll also need to provide a baseline dataset.
For example, if you want to create a data drift monitor to compare the distribution of a feature from the last week to the training set, then focal will be "last week in serving", and baseline will be "training set".
data_drift = aporia.Monitor(
"Data Drift - Last week compared to Training",
monitor_type=aporia.MonitorType.DATA_DRIFT,
detection_method=aporia.DetectionMethod.COMPARED_TO_TRAINING,
focal=aporia.FocalConfiguration(
# Last week in serving
timePeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.WEEKS)
),
baseline=aporia.BaselineConfiguration(
# Training dataset
source=aporia.SourceType.TRAINING
),
...
)
Baseline is required for any monitor that has a "Compared to" field like in the example below, or in any detection method that is not DetectionMethod.ABSOLUTE:
Here's an example for focal / baseline in an anomaly detection over time monitor:
activity_anomaly_detection = aporia.Monitor(
"Activity Anomaly Detection",
monitor_type=aporia.MonitorType.MODEL_ACTIVITY,
detection_method=aporia.DetectionMethod.ANOMALY,
focal=aporia.FocalConfiguration(
# Last day
timePeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
),
baseline=aporia.BaselineConfiguration(
# Last 2 weeks *before* the last day
source=aporia.SourceType.SERVING,
timePeriod=aporia.TimePeriod(count=2, type=aporia.PeriodType.WEEKS),
skipPeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
),
...
)

Step 3: Configure monitor

Next, it is time to configure some important parameters for the monitor. For example:
activity_anomaly_detection = aporia.Monitor(
"Activity Anomaly Detection",
monitor_type=aporia.MonitorType.MODEL_ACTIVITY,
detection_method=aporia.DetectionMethod.ANOMALY,
focal=aporia.FocalConfiguration(
# Last day
timePeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
),
baseline=aporia.BaselineConfiguration(
# Last 2 weeks *before* the last day
source=aporia.SourceType.SERVING,
timePeriod=aporia.TimePeriod(count=2, type=aporia.PeriodType.WEEKS),
skipPeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
),
sensitivity=0.3,
...
)
The following table describes the required parameters for each monitor type
Monitor
Detection Method
Required Parameters
Model Activity
Anomaly Detection
  • sensitivity (0-1)
  • baseline
Model Activity
Change in Percentage
  • percentage (0-100)
  • baseline
Model Activity
Absolute values
  • min (optional)
  • max (optional)
Data Drift
Anomaly Detection
  • thresholds (aporia.ThresholdConfiguration)
  • features (list[str])
  • raw_inputs (list[str])
  • baseline
Data Drift
Compared to Segment
  • thresholds (aporia.ThresholdConfiguration)
  • features (list[str])
  • raw_inputs (list[str])
  • baseline (On segment)
Data Drift
Compared to Training
  • thresholds (aporia.ThresholdConfiguration)
  • features (list[str])
  • raw_inputs (list[str])
  • baseline (On Training)
Prediction Drift
Anomaly Detection
  • thresholds (aporia.ThresholdConfiguration)
  • predictions (list[str])
  • baseline
Prediction Drift
Compared to Segment
  • thresholds (aporia.ThresholdConfiguration)
  • predictions (list[str])
  • baseline (On segment)
Prediction Drift
Compared to Training
  • thresholds (aporia.ThresholdConfiguration)
  • predictions (list[str])
  • baseline (On Training)
Missing Values
Anomaly Detection
  • sensitivity (0-1)
  • min (0-100) (Optional)
  • raw_inputs (list[str])
  • features (list[str])
  • predictions (list[str])
  • baseline
  • testOnlyIncrease (Optional)
Missing Values
Change in Percentage
  • percentage
  • min (0-100) (Optional)
  • raw_inputs (list[str])
  • features (list[str])
  • predictions (list[str])
  • baseline
Missing Values
Absolute values
  • min (0-100) (Optional)
  • max (0-100) (Optional)
  • raw_inputs (list[str])
  • features (list[str])
  • predictions (list[str])
Missing Values
Compared to Segment
  • percentage
  • min
  • raw_inputs (list[str])
  • features (list[str])
  • predictions (list[str])
  • baseline (On segment)
Model Staleness
Absolute
  • staleness_period (aporia.TimePeriod)
New Values
Percentage
  • new_values_count_threshold (Optional)
  • new_values_ratio_threshold (Optional)
  • baseline
New Values
Compared to Segment
  • new_values_count_threshold (Optional)
  • new_values_ratio_threshold (Optional)
  • baseline (on Segment)
New Values
Compared to Training
  • new_values_count_threshold (Optional)
  • new_values_ratio_threshold (Optional)
  • baseline (on Training)
Values Range
Percentage
  • distance
  • baseline
Values Range
Absolute
  • min (Optional)
  • max (Optiona)
Values Range
Compared to Segment
  • distance
  • baseline (on Segment)
Values Range
Compared to Training
  • distance
  • baseline (on Training)
Performance Degradation
Anomaly Detection
  • metric
  • sensitivity
  • baseline
  • metric-specific parameters
Performance Degradation
Absolute
  • metric
  • min (Optional)
  • max (Optional)
  • metric-specific parameters
Performance Degradation
Percentage
  • metric
  • percentage
  • baseline
  • metric-specific parameters
Performance Degradation
Compared to Segment
  • metric
  • percentage
  • baseline (Compared to Segment)
  • metric-specific parameters
Performance Degradation
Compared to Training
  • metric
  • percentage
  • baseline (Compared to Training)
  • metric-specific parameters
Metric Change
Anomaly Detection
  • metric
  • sensitivity
  • baseline
  • metric-specific parameters
Metric Change
Absolute
  • metric
  • min (Optional)
  • max (Optional)
  • metric-specific parameters
Metric Change
Percentage
  • metric
  • percentage
  • baseline
  • metric-specific parameters
Metric Change
Compared to Segment
  • metric
  • percentage
  • baseline (Compared to Segment)
  • metric-specific parameters
Metric Change
Compared to Training
  • metric
  • percentage
  • baseline (Compared to Training)
  • metric-specific parameters
Custom Metric
Anomaly Detection
  • custom_metric/custom_metric_id
  • sensitivity
  • baseline
Custom Metric
Absolute
  • custom_metric/custom_metric_id
  • min (Optional)
  • max (Optional)
  • baseline
Custom Metric
Percentage
  • custom_metric/custom_metric_id
  • percentage
  • baseline

Metric-specific parameters

  • k: Used for ranking metrics, such as NDCG, MRR, MAP etc.
  • prediction_threshold: Used for binary confusion matrix metrics, such as accuracy, tp_count, recall etc. Used with a numeric prediction (0-1) and a boolean actual.
  • prediction_class: The class for which to calculate per-class metrics, such as accuracy-per-class.
  • average_method: Used for precision/recall/f1_score on multiclass predictions. Values are of aporia.AverageMethod enum (MICRO/MACRO/WEIGHTED)

Step 4: Configure monitor action (e.g alert)

Finally, any monitor requires a severity parameter to describe the severity of the alerts generated by this monitor (low / medium / high).
You can also add an emails parameter for receiving the alert in mail, or the messaging parameter for integration with Webhooks, Datadog, Slack, Teams, etc.
activity_anomaly_detection = aporia.Monitor(
"Activity Anomaly Detection",
monitor_type=aporia.MonitorType.MODEL_ACTIVITY,
detection_method=aporia.DetectionMethod.ANOMALY,
focal=aporia.FocalConfiguration(
# Last day
timePeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
),
baseline=aporia.BaselineConfiguration(
# Last 2 weeks *before* the last day
source=aporia.SourceType.SERVING,
timePeriod=aporia.TimePeriod(count=2, type=aporia.PeriodType.WEEKS),
skipPeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
),
sensitivity=0.3,
severity=aporia.Severity.MEDIUM,
emails=[<EMAIL_LIST>],
messaging={"WEBHOOK": [WEBHOOK_INTEGRATION_ID], "SLACK": [SLACK_INTEGRATION_ID]}
)

Detection Methods Overview

Detection Method
Enum value
Description
Exmaple
Anomaly Detection over Time
DetectionMethod.ANOMALY
This will train an anomaly detection model to raise an alert if there's an anomaly in metric value with respect to a certain baseline
Missing value ratio of last week compared to week before that
Change in Percentage
DetectionMethod.PERCENTAGE
Detect change in percentage in metric value
Standard deviation changed by >20%
Absolute values
DetectionMethod.ABSOLUTE
Raise an alert when metric value is larger or smaller than a certain value
Accuracy is lower than 0.9
Compared to Segment
DetectionMethod.COMPARED_TO_SEGMENT
Detect changes in metric value between two data segments
Data drift between gender=male to gender=female
Compared to Training
DetectionMethod.COMPARED_TO_TRAINING
Data change in metric value compared to the training set
Prediction drift of last month in serving compared to training

Supported Monitor Types / Detection Methods

The following table describes the various monitor types and their supported detection methods:
Monitor Type
Enum value
Supported detection methods
Model Activity
MonitorType.MODEL_ACTIVITY
  • DetectionMethod.ANOMALY
  • DetectionMethod.PERCENTAGE
  • DetectionMethod.ABSOLUTE
Data Drift
MonitorType.DATA_DRIFT
  • DetectionMethod.ANOMALY
  • DetectionMethod.COMPARED_TO_SEGMENT
  • DetectionMethod.COMPARED_TO_TRAINING
Prediction Drift
MonitorType.PREDICTION_DRIFT
  • DetectionMethod.ANOMALY
  • DetectionMethod.COMPARED_TO_SEGMENT
  • DetectionMethod.COMPARED_TO_TRAINING
Missing Values
MonitorType.MISSING_VALUES
  • DetectionMethod.ANOMALY
  • DetectionMethod.PERCENTAGE
  • DetectionMethod.ABSOLUTE
  • DetectionMethod.COMPARED_TO_SEGMENT
Performance Degradation
MonitorType.PERFORMANCE_DEGRADATION
  • DetectionMethod.ANOMALY
  • DetectionMethod.PERCENTAGE
  • DetectionMethod.ABSOLUTE
  • DetectionMethod.COMPARED_TO_SEGMENT
  • DetectionMethod.COMPARED_TO_TRAINING
Metric Change
MonitorType.METRIC_CHANGE
  • DetectionMethod.ANOMALY
  • DetectionMethod.PERCENTAGE
  • DetectionMethod.ABSOLUTE
  • DetectionMethod.COMPARED_TO_SEGMENT
  • DetectionMethod.COMPARED_TO_TRAINING
Custom Metric
MonitorType.CUSTOM_METRIC_MONITOR
  • DetectionMethod.ANOMALY
  • DetectionMethod.PERCENTAGE
  • DetectionMethod.ABSOLUTE
Model Staleness
MonitorType.MODEL_STALENESS
  • DetectionMethod.ABSOLUTE
Value Range
MonitorType.VALUE_RANGE
  • DetectionMethod.PERCENTAGE
  • DetectionMethod.ABSOLUTE
  • DetectionMethod.COMPARED_TO_SEGMENT
  • DetectionMethod.COMPARED_TO_TRAINING
New Values
MonitorType.NEW_VALUES
  • DetectionMethod.PERCENTAGE
  • DetectionMethod.COMPARED_TO_SEGMENT
  • DetectionMethod.COMPARED_TO_TRAINING