← Back to all products

ML Testing Framework

$29

Test ML code: data validation, model performance, fairness checks, integration tests, and regression detection.

42 files
PythonYAMLMarkdownJSON

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 42 files

ml-testing-framework/ ├── LICENSE ├── README.md ├── configs/ │ ├── thresholds.yaml │ └── validation_rules.yaml ├── free-sample.zip ├── guide/ │ ├── 01-fairness-testing.md │ └── 02-ml-testing-strategy.md ├── guides/ │ ├── fairness-testing.md │ └── ml-testing-strategy.md ├── index.html ├── interactive.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── behavioral.cpython-312.pyc │ │ ├── fairness.cpython-312.pyc │ │ ├── fixtures.cpython-312.pyc │ │ ├── performance.cpython-312.pyc │ │ ├── regression.cpython-312.pyc │ │ └── validators.cpython-312.pyc │ ├── behavioral.py │ ├── fairness.py │ ├── fixtures.py │ ├── performance.py │ ├── regression.py │ └── validators.py └── tests/ ├── __pycache__/ │ ├── test_behavioral.cpython-312-pytest-9.0.2.pyc │ ├── test_behavioral.cpython-312.pyc │ ├── test_fairness.cpython-312-pytest-9.0.2.pyc │ ├── test_fairness.cpython-312.pyc │ ├── test_performance.cpython-312-pytest-9.0.2.pyc │ ├── test_performance.cpython-312.pyc │ ├── test_regression.cpython-312-pytest-9.0.2.pyc │ ├── test_regression.cpython-312.pyc │ ├── test_validators.cpython-312-pytest-9.0.2.pyc │ └── test_validators.cpython-312.pyc ├── test_behavioral.py ├── test_fairness.py ├── test_performance.py ├── test_regression.py └── test_validators.py

Documentation Preview README excerpt

ML Testing Framework

A structured, reusable framework for testing machine-learning models and data pipelines. Covers the full testing lifecycle: data validation, performance gates, fairness auditing, behavioral tests, and regression detection between model versions.

Features

  • Data Validators — Schema checks (column presence, dtype, nullable), range validation, PSI-based distribution drift detection, completeness and duplicate checks
  • Performance Gates — Configurable threshold assertions for classification (accuracy, precision, recall, F1, AUC) and regression (MAE, RMSE, R², MAPE) metrics with warn-delta early warning
  • Fairness Auditing — Demographic parity ratio, equalized odds (TPR/FPR gap), and disparate impact ratio with stdlib-only math for portable execution
  • Behavioral Tests — Invariance tests (neutral perturbations shouldn't change output), directional expectation tests (increasing X should increase Y), metamorphic tests (input transforms should produce predictable output transforms), and noise robustness
  • Regression Detection — Paired t-test and bootstrap confidence intervals to statistically determine if a new model is worse than the current one, with per-slice subgroup analysis
  • Pytest Integration — Synthetic data generators, model stubs, and temporary artifact helpers for clean test setup

File Structure


ml-testing-framework/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│   ├── __init__.py            # Package init with version
│   ├── validators.py          # Schema, range, drift, completeness checks
│   ├── performance.py         # Metric gates and auto-computation
│   ├── fairness.py            # Demographic parity, equalized odds, subgroup metrics
│   ├── behavioral.py          # Invariance, directional, metamorphic tests
│   ├── regression.py          # Paired t-test, bootstrap, RegressionDetector
│   └── fixtures.py            # Synthetic data, model stubs, temp artifacts
├── tests/
│   ├── test_validators.py     # 14 tests: schema, PSI, completeness, duplicates
│   ├── test_performance.py    # 12 tests: metrics, gates, auto-compute, parsing
│   ├── test_fairness.py       # 10 tests: DP ratio, EO gap, audit, subgroups
│   ├── test_behavioral.py     # 8 tests: invariance, directional, metamorphic, noise
│   └── test_regression.py     # 9 tests: t-test, bootstrap, detector, slices
├── configs/
│   ├── thresholds.yaml        # Performance, fairness, and behavioral thresholds
│   └── validation_rules.yaml  # Per-column schema definition example
└── guides/
    ├── ml-testing-strategy.md # Layered testing pyramid with code examples
    └── fairness-testing.md    # Fairness metrics deep dive and remediation

Quick Start


pip install -r requirements.txt
pytest tests/ -v

Usage

Data Validation


from src.validators import SchemaValidator, ColumnRule, check_distribution_drift

rules = [
    ColumnRule("age", dtype="integer", nullable=False, min_value=0, max_value=120),
    ColumnRule("income", dtype="float", nullable=True, min_value=0),
]

*... continues with setup instructions, usage examples, and more.*

Code Sample .py preview

src/behavioral.py """ behavioral — Invariance, directional-expectation, and metamorphic tests. Traditional unit tests assert on *specific* outputs for *specific* inputs. Behavioral tests instead assert on *properties* that should hold across families of inputs. This is much more natural for ML models where the exact output is stochastic or hard to specify. Three test families ------------------- 1. **Invariance tests** — Perturbing the input in a semantically neutral way should NOT change the prediction. Example: adding whitespace to a text classifier's input shouldn't flip the sentiment. 2. **Directional expectation tests** — Perturbing the input in a known direction should move the prediction in a predictable direction. Example: increasing a credit applicant's income should never *decrease* their predicted creditworthiness. 3. **Metamorphic tests** — A transformation of the input should produce a transformation of the output that satisfies a known relation. Example: doubling all prices in a demand-forecasting model should roughly halve predicted demand (within some tolerance). The test runner accepts any callable ``model_fn(input) -> output`` so it works with sklearn estimators, PyTorch modules, REST API wrappers, etc. """ from __future__ import annotations import copy import logging from dataclasses import dataclass, field from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple import numpy as np logger = logging.getLogger(__name__) # --------------------------------------------------------------------------- # Result types # --------------------------------------------------------------------------- @dataclass class BehavioralTestResult: """Outcome of a single behavioral test case.""" test_name: str test_type: str # "invariance" | "directional" | "metamorphic" # ... 275 more lines ...