Drift Detection Guide

Drift is the divergence between the data a model was trained on and the data it

now scores. This guide explains the three statistical tests this framework uses

— PSI, the Kolmogorov-Smirnov test, and the chi-square test — how to

read their output, and how to tune the thresholds that turn a raw statistic into

a PASS / WARN / FAIL verdict.

Two kinds of drift

• Data drift (covariate shift) — the distribution of the *inputs* P(X)

changes. The relationship between inputs and target may still hold, but the

model is now extrapolating into regions it saw little of during training.

• Concept drift — the relationship P(Y|X) itself changes. The same inputs

now imply a different outcome (e.g. spending €500/month meant "low risk" in

2019 and "high risk" after an economic shock).

This framework detects data drift directly, by comparing input

distributions. That is its superpower: it needs no ground-truth labels, so

it warns you *today*, weeks before the labels that would reveal concept drift

arrive. Persistent, large data drift is also the most common practical *cause*

of concept drift, so a drifting feature is a strong leading indicator even when

you cannot measure P(Y|X) directly.

Reference vs current

Every drift check compares two datasets:

• reference — the baseline you trust. Usually the training set, or the most

recent window you validated and were happy with.

• current — the window you want to test, e.g. the last 7 days of production

traffic.

from model_validation import DataDriftDetector, DriftThresholds

detector = DataDriftDetector(
    thresholds=DriftThresholds(psi_warn=0.10, psi_fail=0.25, ks_p_value=0.05),
    categorical_features=["region", "plan"],   # the rest are auto-detected
)
report = detector.detect(reference_columns, current_columns)

Input can be a dict[str, sequence] (column name → values) or a pandas

DataFrame. Numeric columns are auto-detected by dtype; name any categorical

columns explicitly via categorical_features to be safe.

Population Stability Index (PSI)

PSI is the workhorse drift metric for tabular data. It buckets a feature using

the reference quantiles, then measures how much probability mass moved

between the reference and current samples:

PSI = Σ_i (c_i − r_i) · ln(c_i / r_i)

where r_i and c_i are the proportion of the reference and current samples

that fall in bucket *i*. Because the buckets are defined by reference quantiles,

an unshifted feature scores ≈ 0. The widely used interpretation bands:

PSI is an effect-size measure: it tells you *how much* the distribution

moved, not whether the move is statistically "significant". That is exactly what

you want for a stability gate — a change large enough to matter shows up

regardless of sample size.

from model_validation import population_stability_index
psi = population_stability_index(reference["age"], current["age"], bins=10)

Notes on the implementation:

• Bins come from reference quantiles, de-duplicated so constant or

low-cardinality features never create zero-width buckets.

• The outermost buckets are open-ended, so values that fall outside the

reference range (a classic drift symptom) are captured rather than dropped.

• A small epsilon floors each proportion so PSI never divides by — or takes the

log of — zero.

Kolmogorov-Smirnov test (numeric significance)

PSI tells you the size of a move; the two-sample KS test tells you whether a

move is real. It computes D, the maximum absolute gap between the two empirical

cumulative distribution functions, and converts it to a p-value via the

Kolmogorov distribution:

from model_validation import ks_2samp
d_statistic, p_value = ks_2samp(reference["income"], current["income"])

• D ranges from 0 (identical) to 1 (disjoint). Larger means more different.
• p-value is the probability of seeing a gap that large if both samples came

from the same distribution. p < ks_p_value (default 0.05) rejects "same

distribution".

The p-value uses the standard asymptotic Kolmogorov series with the

small-sample correction term, the same approximation SciPy uses — but

implemented here in pure NumPy, so the framework has no SciPy dependency.

Why PSI *and* KS together

With large production samples, the KS test becomes hypersensitive: a difference

far too small to affect the model becomes "statistically significant" simply

because you have a million rows. This is the classic *large-n* trap. The

detector therefore flags a numeric feature only when:

PSI ≥ psi_fail            (a large move on its own), OR
(KS significant AND PSI ≥ psi_warn)   (a real move that is also big enough)

Significance answers "is it real?"; PSI answers "is it big enough to care?". You

need both to avoid a wall of false alarms.

Chi-square test (categorical significance)

For categorical features, the detector builds a 2 × k contingency table

(reference vs current counts across the union of categories) and runs a

chi-square test of homogeneity:

from model_validation import chi_square_test
statistic, p_value, dof = chi_square_test(reference["plan"], current["plan"])

The p-value is computed from the chi-square survival function via the

regularised upper incomplete gamma function — again, pure NumPy. As with numeric

features, the detector pairs significance with a categorical PSI effect-size

gate, so a new rare category or a tiny proportion wobble does not trip a false

alarm on a large sample.

Reading a drift report

detector.detect(...) returns a DriftReport. The most useful members:

report.n_features        # how many features were compared
report.n_drifted         # how many drifted
report.drift_share       # fraction drifted (0.0–1.0)
report.has_major_drift   # any feature in the "major" severity band?
report.drifted_features  # list[FeatureDriftResult]
report.get("income")     # the result for one feature
report.to_dict()         # JSON-serialisable summary

Each FeatureDriftResult carries the feature name, kind (numeric /

categorical), drifted flag, severity (none / moderate / major),

the psi, the test statistic, and the p_value. Example console output from

examples/drift_check.py:

feature         kind         severity       psi     p_value
monthly_spend   numeric      major        0.839  2.7e-295  <== DRIFT
plan            categorical  moderate     0.235  2.6e-123  <== DRIFT
age             numeric      none         0.003     0.888

Here monthly_spend shifted enough to be a hard failure, plan's category mix

moved into the warning band, and age is stable.

Tuning thresholds

DriftThresholds controls both per-feature sensitivity and how the whole check

is graded:

The DataDriftCheck grade combines per-feature severity with the overall share:

it fails if any feature shows *major* drift or the drifted share exceeds

max_drifted_share_fail; it warns if the drifted share exceeds

max_drifted_share_warn (or any feature drifted at all); otherwise it passes.

Practical tuning advice

• Start loose, tighten later. Begin with the defaults and everything as

WARN. Watch a few weeks of real traffic to learn your baseline noise floor

before you let drift block a deployment.

• Segment your reference window. Comparing against a single static training

snapshot will eventually flag seasonal-but-benign change. A rolling

"last validated window" reference adapts to legitimate evolution.

• Mind the sample size. Very small current windows make PSI and the tests

noisy; aggregate to at least a few hundred rows per feature before trusting a

verdict.

• Watch the most important features harder. Set tighter thresholds (or a

stricter gate) for the handful of features that drive most of the model's

predictions, and looser ones for long-tail features.

A standalone drift-monitoring job

from model_validation import DataDriftDetector
import sys

detector = DataDriftDetector(categorical_features=["region", "plan"])
report = detector.detect(reference, last_24h)
print(report.to_dict())
sys.exit(1 if report.has_major_drift else 0)   # alert / page on major drift

Schedule this against a rolling production window and alert on a non-zero exit.

For the full version with formatted output, see examples/drift_check.py.

Limitations to keep in mind

• Drift detection sees the *inputs*, not the *outcome*. It cannot prove the

model got worse, only that the world it operates in has changed. Pair it with

PerformanceCheck once labels arrive.

• Univariate tests (one feature at a time) miss multivariate drift, where

each feature's marginal looks fine but their *joint* distribution shifts.

Treat a clean per-feature report as necessary, not sufficient.

• A drifting feature is a signal to *investigate*, not an automatic verdict that

the model is broken — sometimes the world genuinely changed and the model

still generalises. The gate exists to make sure a human looks.