In Aporia, custom metrics are defined using syntax that is similar to python's.

There are three building blocks which can be used in order to create a custom metric expression:

Constants - a numeric value (e.g. 2, 0.5, ..)

Functions - out of the builtin function collection you can find below (e.g. sum, count, ...). All those functions return a numeric value.

Binary operation - +, -, *, /, **. Operands can be both constants or function calls.

Builtin Functions

Before we dive into each of the supported function, there are two general concepts you should be familiar with regarding all functions - field expressions and data segment filters.

Field Expressions

A field expression can be described in the following format:

<field_category>.<field_name>[<segment filter>]

Field category is one of the following: features / raw_inputs / predictions / actuals. Note that you can only use categories which you defined in you schema while creating your model version. In addition, don't forget that predictions and actuals categories have the same field names.

The segment filter is optional, for further information about the filters read the section below.

Data Segment Filters

Data segment filters are boolean expressions, designed to reduce to a specific data segment the field on which we perform the function.

Each boolean condition in a segment filter is a comparison between a field and a constant value. For example:

[features.Driving_License == True] // will filter out records in which Driving_License != True
[raw_inputs.Age <= 35] // will only include records in which Age <= 35

Conditions can be combined using and / or and all fields can be checked for missing values using is None / is not None.

The following describe the supported combinations:

The table cells indicates the type we can compare to.

Examples

// Average annual premium of those with a driving license
sum(features.Annual_Premium[features.Driving_License == True]) / prediction_count()
// Three time number of prediction of those who are under 35 years old and live in CA
prediction_count(raw_inputs.Age <= 35 and raw_inputs.Region_Code == 28) * 3
prediction_count(features.Age > 27) / (sum(features.Annual_Premium) + sum(features.Vintage))

Supported functions

accuracy

Parameters

prediction: prediction field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

filter: the filter we want to apply on the records before calculating the metric

actuals_count

Parameters

No parameters needed, cannot apply filters on this metric.

actuals_ratio

Parameters

No parameters needed, cannot apply filters on this metric.

auc_roc

Parameters

prediction: prediction probability field

label: label field

filter: the filter we want to apply on the records before calculating the metric

count

Parameters

No parameters needed, cannot apply filters on this metric.

f1_score

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

average: the average strategy (micro / macro / weighted)

top_k: consider only top-k items.

filter: the filter we want to apply on the records before calculating the metric

fn_count

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

filter: the filter we want to apply on the records before calculating the metric

fp_count

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

filter: the filter we want to apply on the records before calculating the metric

fp_rate

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

filter: the filter we want to apply on the records before calculating the metric

logloss

Parameters

prediction: prediction field

label: label field

filter: the filter we want to apply on the records before calculating the metric

mae

Parameters

prediction: prediction field

label: label field

filter: the filter we want to apply on the records before calculating the metric

mape

Parameters

prediction: prediction field

label: label field

filter: the filter we want to apply on the records before calculating the metric

max

Parameters

field: numeric or dict. The field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict.

mean

Parameters

field: numeric or dict. The field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict.

median

Parameters

field: numeric or dict. The field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict.

min

Parameters

field: numeric or dict. The field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict.

miss_rate

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

filter: the filter we want to apply on the records before calculating the metric

missing_count

Parameters

field: the field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

missing_ratio

Parameters

field: the field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

mse

Parameters

prediction: prediction field

label: label field

filter: the filter we want to apply on the records before calculating the metric

ndcg

Parameters

prediction: prediction field

label: label field

rank: the rank position

filter: the filter we want to apply on the records before calculating the metric

not_missing_count

Parameters

field: the field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

precision_score

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

average: the average strategy (micro / macro / weighted)

top_k: consider only top-k items.

filter: the filter we want to apply on the records before calculating the metric

recall_score

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

average: the average strategy (micro / macro / weighted)

top_k: consider only top-k items.

filter: the filter we want to apply on the records before calculating the metric

rmse

Parameters

prediction: prediction field

label: label field

filter: the filter we want to apply on the records before calculating the metric

specificity

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

filter: the filter we want to apply on the records before calculating the metric

std

Parameters

field: numeric or dict. The field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict

sum

Parameters

field: numeric or dict. The field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict

tn_count

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

filter: the filter we want to apply on the records before calculating the metric

tp_count

Parameters

prediction: prediction probability field

label: label field

threshold: numeric. Probability threshold according to which we decide the if a class is positive

filter: the filter we want to apply on the records before calculating the metric

unique_count

Parameters

field: numeric or dict. The field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict.

value_count

Parameters

field: the field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

value: the value we want to count

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict.

value_percentage

Parameters

field: the field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

value: the value we want to count

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict.

variance

Parameters

field: numeric or dict. The field for which the metric will be computed. Can be of any category (feature / raw_input / prediction / actual)

filter: the filter we want to apply on the records before calculating the metric

keys: keys to filter in when field type is dict.

wape

Parameters

prediction: prediction field

label: label field

filter: the filter we want to apply on the records before calculating the metric