Drift Detection Explained

A practical guide to understanding and implementing drift detection in ML systems.

Why Models Degrade in Production

A model trained on historical data makes one critical assumption: the future will look like the past. When that assumption breaks, the model's predictions become unreliable. This is "drift."

There are three distinct types, and they require different detection strategies:

1. Data Drift (Covariate Shift)

What changes: The distribution of input features.

Example: Your churn model was trained on customers aged 25-45, but a marketing campaign brings in users aged 55+. The model has never seen this population and may produce unreliable predictions.

Detection: Compare the feature distributions between training data and production data using statistical tests like PSI or KS-test.

Key insight: Data drift can happen WITHOUT the model being wrong — the model might still make correct predictions for the new population. But it's a warning sign that the model is operating outside its training domain.

2. Concept Drift

What changes: The relationship between features and the target variable.

Example: During a recession, the factors that predict customer churn change. Previously, low engagement predicted churn. Now, even engaged customers are churning due to budget cuts. The model's learned rules are no longer valid.

Detection: Monitor prediction errors over time. If the error rate increases while the input distribution stays stable, it's likely concept drift. ADWIN and Page-Hinkley are online algorithms that detect this without storing all historical data.

Key insight: Concept drift is more dangerous than data drift because even familiar-looking inputs produce wrong outputs. The model is confidently wrong.

3. Label Drift (Prior Probability Shift)

What changes: The proportion of positive/negative cases.

Example: A fraud detection model trained when 1% of transactions were fraudulent starts seeing 5% fraud (new attack vector). The model's threshold, calibrated for 1% base rate, now produces too many false negatives.

Detection: Track the prediction distribution and the ground-truth positive rate over time.

Statistical Tests in This Toolkit

Population Stability Index (PSI)

PSI measures how much a distribution has shifted by comparing binned probability distributions between a reference (training) set and the current (production) set.

How it works:

1. Bin the reference data into N equal-width buckets (default: 10)

2. Count what proportion of reference data falls in each bin

3. Count what proportion of current data falls in each bin

4. For each bin: psi_i = (P_current - P_reference) × ln(P_current / P_reference)

5. Sum across all bins: PSI = Σ psi_i

Interpretation:

Strengths:

• Easy to interpret (single number per feature)
• Industry standard in financial services / credit scoring
• Works well for continuous features with enough data

Weaknesses:

• Sensitive to bin count choice
• Can miss multimodal shifts within bins
• Requires a fixed reference dataset

Kolmogorov-Smirnov Test (KS Test)

The KS test compares two empirical CDFs and measures the maximum vertical distance between them. It's non-parametric — no assumptions about the underlying distribution.

How it works:

1. Sort both samples

2. Build the empirical CDF for each: F(x) = (count of values ≤ x) / n

3. Find the point where the CDFs are furthest apart: D = max |F_ref(x) - F_cur(x)|

4. Compare D against a critical value that depends on sample sizes and significance level

Interpretation:

• If D > critical value → reject null hypothesis (distributions are different)
• The critical value at α=0.05 is approximately 1.36 × √((n1+n2)/(n1×n2))

Strengths:

• Distribution-free
• Sensitive to changes in location, scale, and shape
• Mathematically rigorous (p-value interpretation)

Weaknesses:

• Less interpretable than PSI ("how much" is it drifting?)
• Sensitive to sample size (large samples = everything is "significant")
• Doesn't tell you WHERE the distribution shifted

Chi-Squared Test (Categorical Features)

For categorical features (country, device type, subscription tier), we use Pearson's chi-squared test to compare category frequencies.

How it works:

1. Count occurrences of each category in reference and current data

2. Scale reference counts to match current sample size (expected counts)

3. Compute: χ² = Σ (observed_i - expected_i)² / expected_i

4. Compare against chi-squared distribution with (k-1) degrees of freedom

When to use: Any feature with a small number of discrete values. For features with 100+ categories, consider grouping rare categories or using a different test.

Concept Drift Detectors

ADWIN (Adaptive Windowing)

ADWIN maintains a variable-length window of recent observations (typically prediction errors). At each step, it checks whether any split of the window shows a statistically significant mean difference.

Key properties:

• Adapts automatically — grows when stable, shrinks when drift occurs
• Memory-efficient via exponential histogram compression
• Theoretical guarantees on false positive/negative rates
• Controlled by delta parameter (smaller = fewer false alarms)

When to use: Gradual drift where you want automatic sensitivity adjustment.

Page-Hinkley Test

A cumulative sum (CUSUM) test that detects changes in the mean of a stream. It maintains a running sum of deviations from the mean and triggers when the cumulative deviation exceeds a threshold.

Key properties:

• Simple and fast (O(1) per observation)
• Two parameters: delta (dead zone) and lambda (threshold)
• Good for detecting abrupt mean shifts
• Resets after detection to find subsequent change points

When to use: Abrupt concept drift (sudden model degradation).

Decision Tree: Which Test to Use

Is the feature categorical?
├── YES → Chi-Squared Test
└── NO (continuous) →
    ├── Do you need a single interpretability score? → PSI
    ├── Do you need rigorous statistical significance? → KS Test
    └── For online streaming detection → ADWIN or Page-Hinkley

Running Drift Analysis in Practice

How Often?

How Much Data?

Both reference and current samples should have at least 200-500 observations per feature for reliable test results. With fewer than 100, statistical tests lack power and may miss real drift.

What To Do When Drift Is Detected

1. Investigate before retraining. Drift might be a data quality issue, not a real distribution change.

2. Check upstream data sources. Schema changes, ETL bugs, and missing imputation are common root causes.

3. Compare drift magnitude across features. If one feature drifts while others are stable, it's likely a data issue rather than a population shift.

4. Monitor performance alongside drift. Data drift without performance degradation may not require action.

5. When retraining, use a blend of old and new data unless the old distribution is completely irrelevant.