You can build fully custom Metrics/Tests to handle any column- or dataset-level evaluations. This lets you implement business metrics, weighted scores, etc.

There are ways to customize your evals that do not require creating Metrics from scratch:

Creating a custom Metric involves:

  • (Required). Implementing the Metric calculation method.

  • (Optional). Defining the default Test conditions that apply when you run Tests for this Metric (with or without Reference) without passing a custom condition.

  • (Optional). Creating a custom visualization for this Metric using Plotly. If you skip this, the Metric will appear as a simple counter in the Report.

Once you implement the Metric, you can use it as usual: include in Reports, view in the Evidently Cloud (or a self-hosted UI), and visualize over time on the Dashboard.

​
Example implementation

This is advanced functionality that assumes you’re comfortable working with the codebase. Refer to existing metrics for examples. To implement the visualization, you must be familiar with Plotly.

Let’s implement MyMaxMetric which calculates the maximum value in a column.

Imports:

from evidently.future.report import Report
from evidently.future.report import Context
from evidently.future.datasets import Dataset
from evidently.future.metric_types import Metric
from evidently.future.metric_types import MetricResult
from evidently.future.metric_types import MetricTestResult
from evidently.future.metric_types import SingleValue
from evidently.future.metric_types import SingleValueTest
from evidently.future.metric_types import MetricCalculation
from evidently.future.metric_types import MetricId
from evidently.future.metric_types import SingleValueMetric
from evidently.future.metric_types import TResult
from evidently.future.preset_types import PresetResult
from evidently.future.metric_types import BoundTest
from evidently.future.tests import Reference, eq
from typing import Optional
from typing import List
from plotly.express import line

Implementation:

class MyMaxMetric(SingleValueMetric):
    column: str

    def _default_tests(self) -> List[BoundTest]:
        return [eq(0).bind_single(self.get_fingerprint())]

    def _default_tests_with_reference(self) -> List[BoundTest]:
        return [eq(Reference(relative=0.1)).bind_single(self.get_fingerprint())]

# implementation
class MaxMetricImplementation(MetricCalculation[SingleValue, MyMaxMetric]):
    def calculate(self, context: Context, current_data: Dataset, reference_data: Optional[Dataset]) -> SingleValue:
        x = current_data.column(self.metric.column).data
        value = x.max()
        result = SingleValue(value=value)
        figure = line(x)
        figure.add_hrect(6, 10)
        #result.set_widget([plotly_figure(title=self.display_name(), figure=figure)])
        return result

    def display_name(self) -> str:
        return f"Max value for {self.metric.column}"

The default Test will checks if the max value is 0 (or within ±10% of the reference value). This applies if you invoke the Tests without setting a custom threshold.

This implementation uses the default (counter) render. Alternatively, you can define the widget as a Plotly figure.

​
Example use

Once implemented, you can reference your custom Metric in a Report as usual.

Let’s create a sample toy dataset:

import pandas as pd
import numpy as np
from evidently.future.datasets import DataDefinition

data = {
    "Item": [f"Item_{i}" for i in range(1, 11)],
    "Quantity": np.random.randint(1, 50, size=10),
    "Sales": np.random.uniform(100, 5000, size=10).round(2),
}

df = pd.DataFrame(data)

dataset = Dataset.from_pandas(
    pd.DataFrame(df),
    data_definition=DataDefinition()
)

Add my MyMaxMetric to the Report:

report = Report([
    MyMaxMetric(column="Sales")
])
my_eval = report.run(dataset, None)
my_eval

Want a Metric added to the core library? Share your idea or feature request by opening a GitHub issue.

Customize Data DriftData drift