Comment on page

# Custom Metric Definition Language

In Aporia, custom metrics are defined using syntax that is similar to python's.

There are three building blocks which can be used in order to create a custom metric expression:

**Constants**- a numeric value (e.g.`2`

,`0.5`

, ..)**Functions**- out of the builtin function collection you can find below (e.g.`sum`

,`count`

, ...). All those functions return a numeric value.**Binary operation**-`+`

,`-`

,`*`

,`/`

,`**`

. Operands can be both constants or function calls.

Before we dive into each of the supported function, there are two general concepts you should be familiar with regarding all functions - field expressions and data segment filters.

A field expression can be described in the following format:

<field_category>.<field_name>[<segment filter>]

Field category is one of the following:

`features`

/ `raw_inputs`

/ `predictions`

/ `actuals`

. Note that you can only use categories which you defined in you schema while creating your model version. In addition, don't forget that `predictions`

and `actuals`

categories have the same field names.The segment filter is optional, for further information about the filters read the section below.

Data segment filters are boolean expressions, designed to reduce to a specific data segment the field on which we perform the function.

Each boolean condition in a segment filter is a comparison between a field and a constant value. For example:

[features.Driving_License == True] // will filter out records in which Driving_License != True

[raw_inputs.Age <= 35] // will only include records in which Age <= 35

Conditions can be combined using

`and`

/ `or`

and all fields can be checked for missing values using `is None`

/ `is not None`

.The following describe the supported combinations:

Type / Operation | == | != | < | > | >= | <= |
---|---|---|---|---|---|---|

Boolean | True/False | True/False | ✖️ | ✖️ | ✖️ | ✖️ |

Categorical | numeric constants | numeric constants | ✖️ | ✖️ | ✖️ | ✖️ |

String | numeric constants | numeric constants | ✖️ | ✖️ | ✖️ | ✖️ |

Numeric | numeric constants | numeric constants | numeric constants | numeric constants | numeric constants | numeric constants |

The table cells indicates the type we can compare to.

**Examples**

// Average annual premium of those with a driving license

sum(features.Annual_Premium[features.Driving_License == True]) / prediction_count()

// Three time number of prediction of those who are under 35 years old and live in CA

prediction_count(raw_inputs.Age <= 35 and raw_inputs.Region_Code == 28) * 3

prediction_count(features.Age > 27) / (sum(features.Annual_Premium) + sum(features.Vintage))

**Parameters**

**prediction**: prediction probability field**label**: label field**threshold**: numeric. Probability threshold according to which we decide the if a class is positive**average**: the average strategy (micro / macro / weighted)**top_k**: consider only top-k items.**filter**: the filter we want to apply on the records before calculating the metric

**Parameters**

**prediction**: prediction probability field**label**: label field**threshold**: numeric. Probability threshold according to which we decide the if a class is positive**average**: the average strategy (micro / macro / weighted)**top_k**: consider only top-k items.**filter**: the filter we want to apply on the records before calculating the metric

**Parameters**

**prediction**: prediction probability field**label**: label field**threshold**: numeric. Probability threshold according to which we decide the if a class is positive**average**: the average strategy (micro / macro / weighted)**top_k**: consider only top-k items.**filter**: the filter we want to apply on the records before calculating the metric

**Parameters**

**field**: the field for which the metric will be computed. Can be of any category (`feature`

/`raw_input`

/`prediction`

/`actual`

)**value**: the value we want to count**filter**: the filter we want to apply on the records before calculating the metric**keys**: keys to filter in when field type is dict.

**Parameters**

**field**: the field for which the metric will be computed. Can be of any category (`feature`

/`raw_input`

/`prediction`

/`actual`

)**value**: the value we want to count**filter**: the filter we want to apply on the records before calculating the metric**keys**: keys to filter in when field type is dict.

Last modified 10mo ago