← Back to all products
$29
ML Testing Framework
Test ML code: data validation, model performance, fairness checks, integration tests, and regression detection.
PythonYAMLMarkdownJSON
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 42 files
ml-testing-framework/
├── LICENSE
├── README.md
├── configs/
│ ├── thresholds.yaml
│ └── validation_rules.yaml
├── free-sample.zip
├── guide/
│ ├── 01-fairness-testing.md
│ └── 02-ml-testing-strategy.md
├── guides/
│ ├── fairness-testing.md
│ └── ml-testing-strategy.md
├── index.html
├── interactive.html
├── requirements.txt
├── src/
│ ├── __init__.py
│ ├── __pycache__/
│ │ ├── __init__.cpython-312.pyc
│ │ ├── behavioral.cpython-312.pyc
│ │ ├── fairness.cpython-312.pyc
│ │ ├── fixtures.cpython-312.pyc
│ │ ├── performance.cpython-312.pyc
│ │ ├── regression.cpython-312.pyc
│ │ └── validators.cpython-312.pyc
│ ├── behavioral.py
│ ├── fairness.py
│ ├── fixtures.py
│ ├── performance.py
│ ├── regression.py
│ └── validators.py
└── tests/
├── __pycache__/
│ ├── test_behavioral.cpython-312-pytest-9.0.2.pyc
│ ├── test_behavioral.cpython-312.pyc
│ ├── test_fairness.cpython-312-pytest-9.0.2.pyc
│ ├── test_fairness.cpython-312.pyc
│ ├── test_performance.cpython-312-pytest-9.0.2.pyc
│ ├── test_performance.cpython-312.pyc
│ ├── test_regression.cpython-312-pytest-9.0.2.pyc
│ ├── test_regression.cpython-312.pyc
│ ├── test_validators.cpython-312-pytest-9.0.2.pyc
│ └── test_validators.cpython-312.pyc
├── test_behavioral.py
├── test_fairness.py
├── test_performance.py
├── test_regression.py
└── test_validators.py
Documentation Preview README excerpt
ML Testing Framework
A structured, reusable framework for testing machine-learning models and data pipelines. Covers the full testing lifecycle: data validation, performance gates, fairness auditing, behavioral tests, and regression detection between model versions.
Features
- Data Validators — Schema checks (column presence, dtype, nullable), range validation, PSI-based distribution drift detection, completeness and duplicate checks
- Performance Gates — Configurable threshold assertions for classification (accuracy, precision, recall, F1, AUC) and regression (MAE, RMSE, R², MAPE) metrics with warn-delta early warning
- Fairness Auditing — Demographic parity ratio, equalized odds (TPR/FPR gap), and disparate impact ratio with stdlib-only math for portable execution
- Behavioral Tests — Invariance tests (neutral perturbations shouldn't change output), directional expectation tests (increasing X should increase Y), metamorphic tests (input transforms should produce predictable output transforms), and noise robustness
- Regression Detection — Paired t-test and bootstrap confidence intervals to statistically determine if a new model is worse than the current one, with per-slice subgroup analysis
- Pytest Integration — Synthetic data generators, model stubs, and temporary artifact helpers for clean test setup
File Structure
ml-testing-framework/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│ ├── __init__.py # Package init with version
│ ├── validators.py # Schema, range, drift, completeness checks
│ ├── performance.py # Metric gates and auto-computation
│ ├── fairness.py # Demographic parity, equalized odds, subgroup metrics
│ ├── behavioral.py # Invariance, directional, metamorphic tests
│ ├── regression.py # Paired t-test, bootstrap, RegressionDetector
│ └── fixtures.py # Synthetic data, model stubs, temp artifacts
├── tests/
│ ├── test_validators.py # 14 tests: schema, PSI, completeness, duplicates
│ ├── test_performance.py # 12 tests: metrics, gates, auto-compute, parsing
│ ├── test_fairness.py # 10 tests: DP ratio, EO gap, audit, subgroups
│ ├── test_behavioral.py # 8 tests: invariance, directional, metamorphic, noise
│ └── test_regression.py # 9 tests: t-test, bootstrap, detector, slices
├── configs/
│ ├── thresholds.yaml # Performance, fairness, and behavioral thresholds
│ └── validation_rules.yaml # Per-column schema definition example
└── guides/
├── ml-testing-strategy.md # Layered testing pyramid with code examples
└── fairness-testing.md # Fairness metrics deep dive and remediation
Quick Start
pip install -r requirements.txt
pytest tests/ -v
Usage
Data Validation
from src.validators import SchemaValidator, ColumnRule, check_distribution_drift
rules = [
ColumnRule("age", dtype="integer", nullable=False, min_value=0, max_value=120),
ColumnRule("income", dtype="float", nullable=True, min_value=0),
]
*... continues with setup instructions, usage examples, and more.*
Code Sample .py preview
src/behavioral.py
"""
behavioral — Invariance, directional-expectation, and metamorphic tests.
Traditional unit tests assert on *specific* outputs for *specific* inputs.
Behavioral tests instead assert on *properties* that should hold across
families of inputs. This is much more natural for ML models where the
exact output is stochastic or hard to specify.
Three test families
-------------------
1. **Invariance tests** — Perturbing the input in a semantically neutral
way should NOT change the prediction. Example: adding whitespace to
a text classifier's input shouldn't flip the sentiment.
2. **Directional expectation tests** — Perturbing the input in a known
direction should move the prediction in a predictable direction.
Example: increasing a credit applicant's income should never *decrease*
their predicted creditworthiness.
3. **Metamorphic tests** — A transformation of the input should produce
a transformation of the output that satisfies a known relation.
Example: doubling all prices in a demand-forecasting model should
roughly halve predicted demand (within some tolerance).
The test runner accepts any callable ``model_fn(input) -> output`` so it
works with sklearn estimators, PyTorch modules, REST API wrappers, etc.
"""
from __future__ import annotations
import copy
import logging
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
import numpy as np
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Result types
# ---------------------------------------------------------------------------
@dataclass
class BehavioralTestResult:
"""Outcome of a single behavioral test case."""
test_name: str
test_type: str # "invariance" | "directional" | "metamorphic"
# ... 275 more lines ...