Querying metrics

To query metrics from Aporia, initialize a new client and call the query_metrics API:

from datetime import datetime, timedelta
from aporia import (
    Aporia,
    MetricDataset,
    MetricParameters,
    TimeRange,
)
from aporia.sdk.datasets import DatasetType

aporia_token = os.environ["APORIA_TOKEN"]
aporia_account = os.environ["APORIA_ACCOUNT"]
aporia_workspace = os.environ["APORIA_WORKSPACE"]

aporia_client = Aporia(
    base_url="https://platform.aporia.com",  # or "https://platform-eu.aporia.com"
    token=aporia_token,
    account_name=aporia_account,
    workspace_name=aporia_workspace,
)

last_week_dataset = MetricDataset(
    dataset_type=DatasetType.SERVING,
    time_range=TimeRange(
        start=datetime.datetime.now() - datetime.timedelta(days=7),
        end=datetime.datetime.now(),
    ),
)

res = metrics.query_batch(
    model_id=model_id,
    metrics=[
        MetricParameters(
            dataset=last_week_dataset,
            name="count",
        ),
    ],
)

print(f"The model had {metrics[0]} predictions last week")

Parameters

The query_metrics API has the following parameters:

ParameterTypeDescription

model_id

str

Model ID to query metrics for.

metrics

List[MetricParameters]

List of metrics to query.

The API can request values for multiple metrics concurrently.

MetricParameters

Here are different fields for the MetricParameters object:

FieldTypeDescription

name

str

dataset

MetricDataset

Specifies what data to query (training / serving), what segment, and what timeframe. Required.

column

str

Name of the column to calculate the metric for. Required except for the count metric. For performance metrics, this should be the name of the prediction, not the actual.

k

int

K value for ranking metrics such as nDCG. Required only for ndcg_at_k, map_at_k, mrr_at_k, accuracy_at_k, precision_at_k, and recall_at_k.

threshold

float

Threshold to use when calculating binary performance metrics.

Required only if the prediction is numeric and the actual is boolean, and the metric is a binary performance metric such as accuracy, recall, precision, f1_score, etc.

custom_metric_id

str

Custom metric ID.

Required only if you want to query a custom metric.

baseline

MetricDataset

Specifies what data to use as baseline.

Required only for statistical distances such as js_distance, ks_distance, psi, and hellinger_distance.

MetricDataset

The MetricDataset object contains the following fields:

FieldTypeDescription

dataset_type

DatasetType

Can be either DatasetType.SERVING or DatasetType.TRAINING. Required.

time_range

TimeRange

Time range (contains start and end fields). Do not pass this for training.

model_version

str

Model version to filter by. Optional.

segment

MetricSegment

Used to query metrics in a specific data segment. Contains id and value fields.

Last updated