Monitors

This guide will show you how to automatically add monitors to your models from code using the Python SDK. For more information on the various types of monitors in Aporia, please refer to the documentation on monitors.​
Defining monitors
To define a new monitor, create an aporia.Monitor(...) object.
Step 1: Choose monitor type and detection method
When designing a new monitor, the first decision you have to make is:
What's the monitor type you'd like to create?
Examples: Data drift, Missing values, Performance degradation, etc.
This step is similar to the following step in the UI:
What's the detection method you'd like to use?
Examples: Anomaly Detection over Time, Change in Percentage, Absolute values, Compared to Training, Compared to Segment, etc.
This step is similar to the following step in the UI:
To begin, start by creating a monitor object, with your chosen monitor_type and detection_method, and add it to your model. 
See Detection Methods Overview and Supported Monitor Types / Detection Methods for the an overview on different monitor types and their supported detection methods.
import aporia.as_code as aporia
​
stack = aporia.Stack(...)
​
data_drift = aporia.Monitor(
    "Data Drift - Last week compared to Training",
    monitor_type=aporia.MonitorType.DATA_DRIFT,
    detection_method=aporia.DetectionMethod.COMPARED_TO_TRAINING,
    ...
)
​
my_model = aporia.Model(
    "My Model",
    type=aporia.ModelType.RANKING,
    icon=aporia.ModelIcon.RECOMMENDATIONS,
    ...,
    
    monitors=[data_drift],
)
​
stack.add(my_model)
Step 2: Choose focal and baseline datasets
The next step is to choose the dataset in which your monitor will be evaluated - this is called the focal dataset. In most detection methods, you'll also need to provide a baseline dataset.
For example, if you want to create a data drift monitor to compare the distribution of a feature from the last week to the training set, then focal will be "last week in serving", and baseline will be "training set".
data_drift = aporia.Monitor(
    "Data Drift - Last week compared to Training",
    monitor_type=aporia.MonitorType.DATA_DRIFT,
    detection_method=aporia.DetectionMethod.COMPARED_TO_TRAINING,
    focal=aporia.FocalConfiguration(
        # Last week in serving
        timePeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.WEEKS)
    ),
    baseline=aporia.BaselineConfiguration(
         # Training dataset
        source=aporia.SourceType.TRAINING
    ),
    ...
)
Baseline is required for any monitor that has a "Compared to" field like in the example below, or in any detection method that is not DetectionMethod.ABSOLUTE:
Here's an example for focal / baseline in an anomaly detection over time monitor:
activity_anomaly_detection = aporia.Monitor(
    "Activity Anomaly Detection",
    monitor_type=aporia.MonitorType.MODEL_ACTIVITY,
    detection_method=aporia.DetectionMethod.ANOMALY,
    focal=aporia.FocalConfiguration(
        # Last day
        timePeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
    ),
    baseline=aporia.BaselineConfiguration(
        # Last 2 weeks *before* the last day
        source=aporia.SourceType.SERVING,
        timePeriod=aporia.TimePeriod(count=2, type=aporia.PeriodType.WEEKS),
        skipPeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
    ),
    ...
)
Step 3: Configure monitor
Next, it is time to configure some important parameters for the monitor. For example:
activity_anomaly_detection = aporia.Monitor(
    "Activity Anomaly Detection",
    monitor_type=aporia.MonitorType.MODEL_ACTIVITY,
    detection_method=aporia.DetectionMethod.ANOMALY,
    focal=aporia.FocalConfiguration(
        # Last day
        timePeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
    ),
    baseline=aporia.BaselineConfiguration(
        # Last 2 weeks *before* the last day
        source=aporia.SourceType.SERVING,
        timePeriod=aporia.TimePeriod(count=2, type=aporia.PeriodType.WEEKS),
        skipPeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
    ),
    sensitivity=0.3,
    ...
)
 The following table describes the required parameters for each monitor type
Monitor
Detection Method
Required Parameters
Model Activity
Anomaly Detection
sensitivity (0-1)
baseline
Model Activity
Change in Percentage
percentage (0-100)
baseline
Model Activity
Absolute values
min (optional)
max (optional)
Data Drift
Anomaly Detection
thresholds (aporia.ThresholdConfiguration)
features (list[str])
raw_inputs (list[str])
baseline
Data Drift
Compared to Segment
thresholds (aporia.ThresholdConfiguration)
features (list[str])
raw_inputs (list[str])
baseline (On segment)
Data Drift
Compared to Training
thresholds (aporia.ThresholdConfiguration)
features (list[str])
raw_inputs (list[str])
baseline (On Training)
Prediction Drift
Anomaly Detection
thresholds (aporia.ThresholdConfiguration)
predictions (list[str]) 
baseline
Prediction Drift
Compared to Segment
thresholds (aporia.ThresholdConfiguration)
predictions (list[str]) 
baseline (On segment)
Prediction Drift
Compared to Training
thresholds (aporia.ThresholdConfiguration)
predictions (list[str]) 
baseline (On Training)
Missing Values
Anomaly Detection
sensitivity (0-1)
min (0-100) (Optional)
raw_inputs (list[str])
features (list[str])
predictions (list[str])
baseline
testOnlyIncrease (Optional)
Missing Values
Change in Percentage
percentage
min (0-100) (Optional)
raw_inputs (list[str])
features (list[str])
predictions (list[str])
baseline
Missing Values
Absolute values
min (0-100) (Optional)
max (0-100) (Optional)
raw_inputs (list[str])
features (list[str])
predictions (list[str])
Missing Values
Compared to Segment
percentage
min
raw_inputs (list[str])
features (list[str])
predictions (list[str])
baseline (On segment)
Model Staleness
Absolute
staleness_period (aporia.TimePeriod)
New Values
Percentage
new_values_count_threshold (Optional)
new_values_ratio_threshold (Optional)
baseline
New Values
Compared to Segment
new_values_count_threshold (Optional)
new_values_ratio_threshold (Optional)
baseline (on Segment)
New Values
Compared to Training
new_values_count_threshold (Optional)
new_values_ratio_threshold (Optional)
baseline (on Training)
Values Range
Percentage
distance
baseline
Values Range
Absolute
min (Optional)
max (Optiona)
Values Range
Compared to Segment
distance
baseline (on Segment)
Values Range
Compared to Training
distance
baseline (on Training)
Performance Degradation
Anomaly Detection
metric
sensitivity
baseline
metric-specific parameters
Performance Degradation
Absolute
metric
min (Optional)
max (Optional)
metric-specific parameters
Performance Degradation
Percentage
metric
percentage
baseline
metric-specific parameters
Performance Degradation
Compared to Segment
metric
percentage
baseline (Compared to Segment)
metric-specific parameters
Performance Degradation
Compared to Training
metric
percentage
baseline (Compared to Training)
metric-specific parameters
Metric Change
Anomaly Detection
metric
sensitivity
baseline
metric-specific parameters
Metric Change
Absolute
metric
min (Optional)
max (Optional)
metric-specific parameters
Metric Change
Percentage
metric
percentage
baseline
metric-specific parameters
Metric Change
Compared to Segment
metric
percentage
baseline (Compared to Segment)
metric-specific parameters
Metric Change
Compared to Training
metric
percentage
baseline (Compared to Training)
metric-specific parameters
Custom Metric
Anomaly Detection
custom_metric/custom_metric_id
sensitivity
baseline
Custom Metric
Absolute
custom_metric/custom_metric_id
min (Optional)
max (Optional)
baseline
Custom Metric
Percentage
custom_metric/custom_metric_id
percentage
baseline
Metric-specific parameters
k: Used for ranking metrics, such as NDCG, MRR, MAP etc.
prediction_threshold: Used for binary confusion matrix metrics, such as accuracy, tp_count, recall etc. Used with a numeric prediction (0-1) and a boolean actual.
prediction_class: The class for which to calculate per-class metrics, such as accuracy-per-class.
average_method: Used for precision/recall/f1_score on multiclass predictions. Values are of aporia.AverageMethod enum (MICRO/MACRO/WEIGHTED)
Step 4: Configure monitor action (e.g alert)
Finally, any monitor requires a severity parameter to describe the severity of the alerts generated by this monitor (low / medium / high).
You can also add an emails parameter for receiving the alert in mail, or the messaging parameter for integration with Webhooks, Datadog, Slack, Teams, etc.
activity_anomaly_detection = aporia.Monitor(
    "Activity Anomaly Detection",
    monitor_type=aporia.MonitorType.MODEL_ACTIVITY,
    detection_method=aporia.DetectionMethod.ANOMALY,
    focal=aporia.FocalConfiguration(
        # Last day
        timePeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
    ),
    baseline=aporia.BaselineConfiguration(
        # Last 2 weeks *before* the last day
        source=aporia.SourceType.SERVING,
        timePeriod=aporia.TimePeriod(count=2, type=aporia.PeriodType.WEEKS),
        skipPeriod=aporia.TimePeriod(count=1, type=aporia.PeriodType.DAYS)
    ),
    sensitivity=0.3,
    severity=aporia.Severity.MEDIUM,
    emails=[<EMAIL_LIST>],
    messaging={"WEBHOOK": [WEBHOOK_INTEGRATION_ID], "SLACK": [SLACK_INTEGRATION_ID]}
)
Detection Methods Overview
Detection Method
Enum value
Description
Exmaple
Anomaly Detection over Time
DetectionMethod.ANOMALY
This will train an anomaly detection model to raise an alert if there's an anomaly in metric value with respect to a certain baseline
Missing value ratio of last week compared to week before that
Change in Percentage
DetectionMethod.PERCENTAGE
Detect change in percentage in metric value
Standard deviation changed by >20%
Absolute values
DetectionMethod.ABSOLUTE
Raise an alert when metric value is larger or smaller than a certain value
Accuracy is lower than 0.9
Compared to Segment
DetectionMethod.COMPARED_TO_SEGMENT
Detect changes in metric value between two data segments
Data drift between gender=male to gender=female
Compared to Training
DetectionMethod.COMPARED_TO_TRAINING
Data change in metric value compared to the training set
Prediction drift of last month in serving compared to training
Supported Monitor Types / Detection Methods
The following table describes the various monitor types and their supported detection methods:
Monitor Type
Enum value
Supported detection methods
Model Activity
MonitorType.MODEL_ACTIVITY
DetectionMethod.ANOMALY
DetectionMethod.PERCENTAGE
DetectionMethod.ABSOLUTE
Data Drift
MonitorType.DATA_DRIFT
DetectionMethod.ANOMALY
DetectionMethod.COMPARED_TO_SEGMENT
DetectionMethod.COMPARED_TO_TRAINING
Prediction Drift
MonitorType.PREDICTION_DRIFT
DetectionMethod.ANOMALY
DetectionMethod.COMPARED_TO_SEGMENT
DetectionMethod.COMPARED_TO_TRAINING
Missing Values
MonitorType.MISSING_VALUES
DetectionMethod.ANOMALY
DetectionMethod.PERCENTAGE
DetectionMethod.ABSOLUTE
DetectionMethod.COMPARED_TO_SEGMENT
Performance Degradation
MonitorType.PERFORMANCE_DEGRADATION
DetectionMethod.ANOMALY
DetectionMethod.PERCENTAGE
DetectionMethod.ABSOLUTE
DetectionMethod.COMPARED_TO_SEGMENT
DetectionMethod.COMPARED_TO_TRAINING
Metric Change
MonitorType.METRIC_CHANGE
DetectionMethod.ANOMALY
DetectionMethod.PERCENTAGE
DetectionMethod.ABSOLUTE
DetectionMethod.COMPARED_TO_SEGMENT
DetectionMethod.COMPARED_TO_TRAINING
Custom Metric
MonitorType.CUSTOM_METRIC_MONITOR
DetectionMethod.ANOMALY
DetectionMethod.PERCENTAGE
DetectionMethod.ABSOLUTE
Model Staleness
MonitorType.MODEL_STALENESS
DetectionMethod.ABSOLUTE
Value Range
MonitorType.VALUE_RANGE
DetectionMethod.PERCENTAGE
DetectionMethod.ABSOLUTE
DetectionMethod.COMPARED_TO_SEGMENT
DetectionMethod.COMPARED_TO_TRAINING
New Values
MonitorType.NEW_VALUES
DetectionMethod.PERCENTAGE
DetectionMethod.COMPARED_TO_SEGMENT
DetectionMethod.COMPARED_TO_TRAINING
​

Monitor	Detection Method	Required Parameters
Model Activity	Anomaly Detection	`sensitivity` (0-1) `baseline`
Model Activity	Change in Percentage	`percentage` (0-100) `baseline`
Model Activity	Absolute values	`min` (optional) `max` (optional)
Data Drift	Anomaly Detection	`thresholds (aporia.ThresholdConfiguration)` `features (list[str])` `raw_inputs (list[str])` `baseline`
Data Drift	Compared to Segment	`thresholds (aporia.ThresholdConfiguration)` `features (list[str])` `raw_inputs (list[str])` `baseline` (On segment)
Data Drift	Compared to Training	`thresholds (aporia.ThresholdConfiguration)` `features (list[str])` `raw_inputs (list[str])` `baseline` (On Training)
Prediction Drift	Anomaly Detection	`thresholds (aporia.ThresholdConfiguration)` `predictions (list[str])` `baseline`
Prediction Drift	Compared to Segment	`thresholds (aporia.ThresholdConfiguration)` `predictions (list[str])` `baseline` (On segment)
Prediction Drift	Compared to Training	`thresholds (aporia.ThresholdConfiguration)` `predictions (list[str])` `baseline` (On Training)
Missing Values	Anomaly Detection	`sensitivity` (0-1) `min` (0-100) (Optional) `raw_inputs (list[str])` `features (list[str])` `predictions (list[str])` `baseline` `testOnlyIncrease` (Optional)
Missing Values	Change in Percentage	`percentage` `min` (0-100) (Optional) `raw_inputs (list[str])` `features (list[str])` `predictions (list[str])` `baseline`
Missing Values	Absolute values	`min` (0-100) (Optional) `max` (0-100) (Optional) `raw_inputs (list[str])` `features (list[str])` `predictions (list[str])`
Missing Values	Compared to Segment	`percentage` `min` `raw_inputs (list[str])` `features (list[str])` `predictions (list[str])` `baseline` (On segment)
Model Staleness	Absolute	`staleness_period (aporia.TimePeriod)`
New Values	Percentage	`new_values_count_threshold` (Optional) `new_values_ratio_threshold` (Optional) `baseline`
New Values	Compared to Segment	`new_values_count_threshold` (Optional) `new_values_ratio_threshold` (Optional) `baseline` (on Segment)
New Values	Compared to Training	`new_values_count_threshold` (Optional) `new_values_ratio_threshold` (Optional) `baseline` (on Training)
Values Range	Percentage	`distance` `baseline`
Values Range	Absolute	`min` (Optional) `max` (Optiona)
Values Range	Compared to Segment	`distance` `baseline` (on Segment)
Values Range	Compared to Training	`distance` `baseline` (on Training)
Performance Degradation	Anomaly Detection	`metric` `sensitivity` `baseline` metric-specific parameters
Performance Degradation	Absolute	`metric` `min` (Optional) `max` (Optional) metric-specific parameters
Performance Degradation	Percentage	`metric` `percentage` `baseline` metric-specific parameters
Performance Degradation	Compared to Segment	`metric` `percentage` `baseline` (Compared to Segment) metric-specific parameters
Performance Degradation	Compared to Training	`metric` `percentage` `baseline` (Compared to Training) metric-specific parameters
Metric Change	Anomaly Detection	`metric` `sensitivity` `baseline` metric-specific parameters
Metric Change	Absolute	`metric` `min` (Optional) `max` (Optional) metric-specific parameters
Metric Change	Percentage	`metric` `percentage` `baseline` metric-specific parameters
Metric Change	Compared to Segment	`metric` `percentage` `baseline` (Compared to Segment) metric-specific parameters
Metric Change	Compared to Training	`metric` `percentage` `baseline` (Compared to Training) metric-specific parameters
Custom Metric	Anomaly Detection	`custom_metric`/`custom_metric_id` `sensitivity` `baseline`
Custom Metric	Absolute	`custom_metric`/`custom_metric_id` `min` (Optional) `max` (Optional) `baseline`
Custom Metric	Percentage	`custom_metric`/`custom_metric_id` `percentage` `baseline`

Detection Method	Enum value	Description	Exmaple
Anomaly Detection over Time	`DetectionMethod.ANOMALY`	This will train an anomaly detection model to raise an alert if there's an anomaly in metric value with respect to a certain baseline	Missing value ratio of last week compared to week before that
Change in Percentage	`DetectionMethod.PERCENTAGE`	Detect change in percentage in metric value	Standard deviation changed by >20%
Absolute values	`DetectionMethod.ABSOLUTE`	Raise an alert when metric value is larger or smaller than a certain value	Accuracy is lower than 0.9
Compared to Segment	`DetectionMethod.COMPARED_TO_SEGMENT`	Detect changes in metric value between two data segments	Data drift between `gender=male` to `gender=female`
Compared to Training	`DetectionMethod.COMPARED_TO_TRAINING`	Data change in metric value compared to the training set	Prediction drift of last month in serving compared to training

Monitor Type	Enum value	Supported detection methods
Model Activity	`MonitorType.MODEL_ACTIVITY`	`DetectionMethod.ANOMALY` `DetectionMethod.PERCENTAGE` `DetectionMethod.ABSOLUTE`
Data Drift	`MonitorType.DATA_DRIFT`	`DetectionMethod.ANOMALY` `DetectionMethod.COMPARED_TO_SEGMENT` `DetectionMethod.COMPARED_TO_TRAINING`
Prediction Drift	`MonitorType.PREDICTION_DRIFT`	`DetectionMethod.ANOMALY` `DetectionMethod.COMPARED_TO_SEGMENT` `DetectionMethod.COMPARED_TO_TRAINING`
Missing Values	`MonitorType.MISSING_VALUES`	`DetectionMethod.ANOMALY` `DetectionMethod.PERCENTAGE` `DetectionMethod.ABSOLUTE` `DetectionMethod.COMPARED_TO_SEGMENT`
Performance Degradation	`MonitorType.PERFORMANCE_DEGRADATION`	`DetectionMethod.ANOMALY` `DetectionMethod.PERCENTAGE` `DetectionMethod.ABSOLUTE` `DetectionMethod.COMPARED_TO_SEGMENT` `DetectionMethod.COMPARED_TO_TRAINING`
Metric Change	`MonitorType.METRIC_CHANGE`	`DetectionMethod.ANOMALY` `DetectionMethod.PERCENTAGE` `DetectionMethod.ABSOLUTE` `DetectionMethod.COMPARED_TO_SEGMENT` `DetectionMethod.COMPARED_TO_TRAINING`
Custom Metric	`MonitorType.CUSTOM_METRIC_MONITOR`	`DetectionMethod.ANOMALY` `DetectionMethod.PERCENTAGE` `DetectionMethod.ABSOLUTE`
Model Staleness	`MonitorType.MODEL_STALENESS`	`DetectionMethod.ABSOLUTE`
Value Range	`MonitorType.VALUE_RANGE`	`DetectionMethod.PERCENTAGE` `DetectionMethod.ABSOLUTE` `DetectionMethod.COMPARED_TO_SEGMENT` `DetectionMethod.COMPARED_TO_TRAINING`
New Values	`MonitorType.NEW_VALUES`	`DetectionMethod.PERCENTAGE` `DetectionMethod.COMPARED_TO_SEGMENT` `DetectionMethod.COMPARED_TO_TRAINING`

Last modified 2mo ago

Monitors

Defining monitors

Step 1: Choose monitor type and detection method

Step 2: Choose focal and baseline datasets

Step 3: Configure monitor

Metric-specific parameters

Step 4: Configure monitor action (e.g alert)

Detection Methods Overview

Supported Monitor Types / Detection Methods

​