Customize Embedding Drift
How to set embedding drift detection conditions.
Pending implementation for the new API.
Pre-requisites:
-
You know how to generate Reports or Test Suites with default parameters.
-
You know how to pass custom parameters for Reports or Test Suites.
Default
When you calculate embeddings drift, Evidently automatically applies the default drift detection method (“model”).
In Reports:
In Test Suites:
It works the same inside presets, like DataDriftPreset
.
Embedding parameters - Metrics and Tests
You can override the defaults by passing a custom drift_method
parameter to the relevant Metric or Test. You can define the embeddings drift detection method, the threshold, or both.
Pass the drift_method
parameter:
Embedding parameters - Presets
When you use NoTargetPerformanceTestPreset
, DataDriftTestPreset
or DataDriftPreset
you can specify which subsets of columns with embeddings to include using embeddings
, and the drift detection method using embeddings_drift_method
.
By default, the Presets will include all columns mapped as containing embeddings in column_mapping
.
To exclude columns with embeddings:
To specify which sets of columns to include (with the default drift detection method):
To specify which sets of columns to include, and specify the method:
Embedding drift detection methods
Currently 4 embeddings drift detection methods are available.
Embeddings drift detection method | Description and default |
---|---|
drift_method=model (Default) |
|
drift_method=ratio |
|
drift_method=distance |
|
drift_method=mmd |
|
If you specify an embedding drift detection method but do not pass additional parameters, defaults will apply.
You can also specify parameters for any chosen method. Since the methods are different, each has a different set of parameters. Note that you should pass the parameters directly to the chosen drift detection method, not to the Metric.
Model-based (“model”)
Parameter | Description |
---|---|
threshold | Sets the threshold for drift detection (ROC AUC). Drift is detected when drift_score > threshold . Applies when bootstrap not equal True.Default: 0.55. |
bootstrap (optional) | Boolean parameter (True/False) to determine whether to apply statistical hypothesis testing. If applied, the ROC AUC of the classifier is compared to the ROC AUC of the random classifier at a set percentile. The calculation is repeated 1000 times with randomly assigned target class probabilities. This produces a distribution of random roc_auc scores with a mean of 0,5. We then take the 95th percentile (default) of this distribution and compare it to the ROC-AUC score of the classifier. If the classifier score is higher, data drift is detected. Default: True if less than or eq 1000 objects, False if > 1000 objects. |
quantile_probability (optional) | Sets the percentile of the possible ROC AUC values of the random classifier to compare against. This applies when bootstrap is True. Default: 0.95 |
pca_components (optional) | The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of pca_components .Default: None. |
Maximum mean discrepancy (“mmd”)
Parameter | Description |
---|---|
threshold | Sets the threshold for drift detection (ROC AUC). Drift is detected when drift_score > threshold . Applies when bootstrap not eq True.Default: 0.55. |
bootstrap (optional) | Boolean parameter (True/False) to determine whether to apply statistical hypothesis testing. If applied, the ROC AUC of the classifier is compared to the ROC AUC of the random classifier at a set percentile. The calculation is repeated 1000 times with randomly assigned target class probabilities. This produces a distribution of random roc_auc scores with a mean of 0,5. We then take the 95th percentile (default) of this distribution and compare it to the ROC-AUC score of the classifier. If the classifier score is higher, data drift is detected. Default: True if less thahn or eq 1000 objects, False if > 1000 objects. |
quantile_probability (optional) | Sets the percentile of the possible ROC AUC values of the random classifier to compare against. This applies when bootstrap is True. Default: 0.95 |
pca_components (optional) | The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of pca_components .Default: None. |
Share of drifted embedding components (“ratio”)
Parameter | Description |
---|---|
component_stattest (optional) | Sets the tabular drift detection method (any of the tabular drift detection methods for numerical features available in Evidently).< br /> Default: Wasserstein |
component_stattest_threshold (optional) | Sets the threshold for drift detection for individual embedding components. Drift is detected when drift_score > component_stattest_threshold in case of distance/divergence metrics where the threshold is the metric value or drift_score < component_stattest_threshold in case of statistical tests where the threshold is the p-value.Default: 0.1 (relevant for Wasserstein). |
threshold (optional) | Sets the threshold (share of drifted embedding components) for drift detection for the overall dataset. Default: 0.2 |
pca_components (optional) | The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of pca_components . Default: None. |
Distance-based methods (“distance”)
Parameter | Description |
---|---|
dist (optional) Available: euclidean cosine cityblock (manhattan distance)chebyshev | Sets the distance metric for drift detection. Default: Euclidean distance |
threshold (optional) | Sets the threshold for drift detection. Drift is detected when drift_score > threshold .Applies when bootstrap not equals True Default: 0.2 (relevant for euclidean distance) |
bootstrap (optional) | Boolean parameter (True/False) to determine whether to apply statistical hypothesis testing. If applied, the distance between reference and current is tested against possible distance values in reference. We randomly split the reference data into two parts and compute the distance between them. The calculation is repeated 100 times. This produces a distribution of distance values obtained for a reference dataset. We then take the 95th percentile (default) of this distribution and compare it to the distance between reference and current datasets. If the distance between the reference and current is higher than the 95th percentile of the distance obtained for the reference dataset, the drift is detected. Default: True if less than or eq 1000 objects, False if > 1000 objects. |
quantile_probability (optional) | Sets the percentile of the possible distance values in reference to compare against. Applies when bootstrap is True.Default: 0.95. |
pca_components (optional) | The number of PCA components. If specified, dimensionality reduction will be applied to project data to n-dimensional space based on the number of pca_components . Default: None. |