Need help? Ask on Discord.

​
1. Set up your environment

This quickstart shows both local open-source and cloud workflows.

You will run a simple evaluation in Python and explore results in Evidently Cloud.

1.1. Set up Evidently Cloud

  • Sign up for a free Evidently Cloud account.

  • Create an Organization if you log in for the first time. Get an ID of your organization. (Link).

  • Get an API token. Click the Key icon in the left menu. Generate and save the token. (Link).

1.2. Installation and imports

Install the Evidently Python library:

!pip install evidently

Components to run the evals:

import pandas as pd
from sklearn import datasets

from evidently.future.datasets import Dataset
from evidently.future.datasets import DataDefinition

from evidently.future.report import Report
from evidently.future.metrics import *
from evidently.future.presets import *
from evidently.future.tests import *

Components to connect with Evidently Cloud:

from evidently.ui.workspace.cloud import CloudWorkspace

1.3. Create a Project

Connect to Evidently Cloud using your API token:

ws = CloudWorkspace(token="YOUR_API_TOKEN", url="https://app.evidently.cloud")

Create a Project within your Organization, or connect to an existing Project:

project = ws.create_project("My project name", org_id="YOUR_ORG_ID")
project.description = "My project description"
project.save()

# or project = ws.get_project("PROJECT_ID")

​
2. Prepare a toy dataset

Let’s import a toy dataset with tabular data:

adult_data = datasets.fetch_openml(name="adult", version=2, as_frame="auto")
adult = adult_data.frame

Let’s split the data into two and introduce some artificial drift for demo purposes. Prod data will include people with education levels unseen in the reference dataset:

adult_ref = adult[~adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]
adult_prod = adult[adult.education.isin(["Some-college", "HS-grad", "Bachelors"])]

Map the column types:

schema = DataDefinition(
    numerical_columns=["education-num", "age", "capital-gain", "hours-per-week", "capital-loss"],
    categorical_columns=["education", "occupation", "native-country", "workclass", "marital-status"],
    )

Create Evidently Datasets to work with:

eval_data_1 = Dataset.from_pandas(
    pd.DataFrame(adult_prod),
    data_definition=schema
)

eval_data_2 = Dataset.from_pandas(
    pd.DataFrame(adult_ref),
    data_definition=schema
)

​
3. Get a Report

Let’s a summary of all columns in the dataset, and run auto-generated Tests to check for data quality and core statistics between two datasets:

report = Report([
    DataSummaryPreset()
],
include_tests="True")
my_eval = report.run(eval_data_1, eval_data_2)

​
4. Explore the results

Upload the Report with summary results:

ws.add_run(project.id, my_eval, include_data=False)

View the Report. Go to Evidently Cloud, open your Project, navigate to “Reports” in the left and open the Report. You will see the summary with scores and Test results.

Get a Dashboard. As you run repeated evals, you may want to track the results in time. Go to the “Dashboard” tab in the left menu and enter the “Edit” mode. Add a new tab, and select the “Columns” template.

You’ll see a set of panels that show column stats. Each has a single data point. As you log ongoing evaluation results, you can track trends and set up alerts.

LLM EvaluationTracing