Links

Custom Metric Definition Language

In Aporia, custom metrics are defined using syntax that is similar to python's.
There are three building blocks which can be used in order to create a custom metric expression:
  • Constants - a numeric value (e.g. 2, 0.5, ..)
  • Functions - out of the builtin function collection you can find below (e.g. sum, count, ...). All those functions return a numeric value.
  • Binary operation - +, -, *, /, **. Operands can be both constants or function calls.

Builtin Functions

Before we dive into each of the supported function, there are two general concepts you should be familiar with regarding all functions - field expressions and data segment filters.

Field Expressions

A field expression can be described in the following format:
<field_category>.<field_name>[<segment filter>]
Field category is one of the following: features / raw_inputs / predictions / actuals. Note that you can only use categories which you defined in you schema while creating your model version. In addition, don't forget that predictions and actuals categories have the same field names.
The segment filter is optional, for further information about the filters read the section below.

Data Segment Filters

Data segment filters are boolean expressions, designed to reduce to a specific data segment the field on which we perform the function.
Each boolean condition in a segment filter is a comparison between a field and a constant value. For example:
[features.Driving_License == True] // will filter out records in which Driving_License != True
[raw_inputs.Age <= 35] // will only include records in which Age <= 35
Conditions can be combined using and / or and all fields can be checked for missing values using is None / is not None.
The following describe the supported combinations:
Type / Operation
==
!=
<
>
>=
<=
Boolean
True/False
True/False
✖️
✖️
✖️
✖️
Categorical
numeric constants
numeric constants
✖️
✖️
✖️
✖️
String
numeric constants
numeric constants
✖️
✖️
✖️
✖️
Numeric
numeric constants
numeric constants
numeric constants
numeric constants
numeric constants
numeric constants
The table cells indicates the type we can compare to.
Examples
// Average annual premium of those with a driving license
sum(features.Annual_Premium[features.Driving_License == True]) / prediction_count()
// Three time number of prediction of those who are under 35 years old and live in CA
prediction_count(raw_inputs.Age <= 35 and raw_inputs.Region_Code == 28) * 3
prediction_count(features.Age > 27) / (sum(features.Annual_Premium) + sum(features.Vintage))

Supported functions

accuracy
actuals_count
actuals_ratio
auc_roc
count
f1_score
fn_count
fp_count
fp_rate
logloss
mae
mape
max
mean
median
min
miss_rate
missing_count
missing_ratio
mse
ndcg
not_missing_count
precision_score
recall_score
rmse
specificity
std
sum
tn_count
tp_count
unique_count
value_count
value_percentage
variance
wape