Skip to content

Custom Metric

In case the monitoring metrics provided by Aporia are insufficient for your use-case, you can define a metric based on custom logic.

Detection Methods

For this monitor type, you can select the following detection methods:

Custom Metric Detection Methods

  • Absolute Values - The metric value is lower or higher than a specific value.
  • Anomaly Detection - Detects anomalies in the value of the metric in the inspected data and its value in a time period before the data was collected.
  • Change In Percentage - Detects change in the ratio between the metric value of the inspected data and its value in a time period before the data was collected.

Configuration

Choose a custom metric that you defined previously, or create a new one

Custom Metric Configuration

Custom Metric Definition Language

Custom metrics are defined using a syntax that is similar to Python code.

Custom Metric Definition

A custom metric is a single expression, which can be any of the following:

  • A constant, numeric value (e.g. 5)
  • A function call - the currently available functions are sum and prediction_count
  • The sum function must receive a single parameter, which must be a valid field expression
    • The field expression must specify a field with a numeric value
    • The function returns a sum aggregation on the specified field
  • The prediction_count function can receive up to one parameter, which must be a valid segment filter
    • If no parameters were passed, the function returns the total number of predictions reported by the model
    • If a segment filter was passed, the function returns the number of predictions that fall within the data segment described by the filter.
  • A binary operation (+, -, *, /, **), in which the operands are constants or function calls (e.g. sum(features.Annual_Premium) / 100)

Field Expressions

A field expression describes a single field (feature, raw_input, prediction value or actual value), and allows operations on the values of that field across all reported predictions, or within a specific data segment

Field expressions have the following format: <field_category>.<field_name>[<segment_filter>]:

  • field_category is one of features, predictions, raw_inputs and actuals.
  • The available field categories and fields are based on the schema you defined when creating a model version.
  • The actuals category uses the fields defined for the predictions category
  • The segment filter is optional

Data Segment Filters

A data segment filter is an expression that restricts the results of an action to a specific data segment.

Segment filters are boolean expressions - each boolean condition in a segment filter is a comparison between a field and a constant value, and conditions can be combined using and, or.

The following comparisons are possible:

  • All fields can be checked for missing values using is None and is not None - for example: features.Annual_Premium is not None
  • Categorical, Boolean and String features can only be compared to constants using the == and != operators
  • Numeric fields can be compared to constants using the <, <=, >, >=, == and != operators
  • Datetime Fields are currently not supported

Examples

sum(features.Annual_Premium[features.Driving_License == True]) / prediction_count()
prediction_count(raw_inputs.Age <= 35 and raw_inputs.Region_Code == 28) * 3
prediction_count(features.Age > 27) / (sum(features.Annual_Premium) + sum(features.Vintage))

Creating this monitor using the REST API

POST https://app.aporia.com/v1beta/monitors
{
    "name": "Monitor custom metric",
    "type": "custom_metric",
    "scheduling": "0 0 */2 * *",
    "configuration": {
      "configuration": {
          "focal": {
              "source": "SERVING",
              "timePeriod": "1M"
          },
          "metric": {
              "type": "238e8488-a6a0-4276-9707-ab6b6cf56990"
          },
          "actions": [
              {
                  "type": "ALERT",
                  "schema": "v1",
                  "severity": "LOW",
                  "alertType": "metric_change",
                  "description": "An anomaly in the value of the <b>'{metric}'</b> within the defined limits was detected.<br /> The anomaly was observed in the <b>{model}</b> model, in version <b>{model_version}</b> for the <b>last {focal_time_period} ({focal_times})</b> <b>{focal_segment}</b>. <br /><br /> Based on defined limits, the metric value was expected to be below <b>{max_threshold}</b>, but <b>{focal_value}</b> was received.",
                  "notification": [
                      {
                          "type": "EMAIL",
                          "emails": [
                              "dev@aporia.com"
                          ]
                      }
                  ],
                  "visualization": "value_over_time"
              }
          ],
          "logicEvaluations": [
              {
                  "max": 10,
                  "min": null,
                  "name": "RANGE"
              }
          ]
      },
      "identification": {
          "models": {
              "id": "seed-0000-fhpy"
          },
          "segment": {
              "group": null
          },
          "environment": null
      }
  }
}