ML Monitoring Suite
Model monitoring dashboards, alert configurations, data quality checks, and performance tracking.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 17 files
Documentation Preview README excerpt
ML Monitoring Suite
Production model monitoring with zero third-party dependencies. A small,
batteries-included Python package that keeps a served model healthy: prediction /
latency / throughput metrics, data-quality expectations, online drift detection
(PSI / KS / Jensen-Shannon), a stateful alert engine, Prometheus + JSON exporters,
and a ready-to-import Grafana dashboard.
Everything is pure Python standard library (3.9+), so it imports and runs in any
environment your model already runs in — no pip install required to use it.
Table of Contents
- [Why this exists](#why-this-exists)
- [What's included](#whats-included)
- [Requirements](#requirements)
- [Installation](#installation)
- [Project structure](#project-structure)
- [Quick start](#quick-start)
- [The modules](#the-modules)
- [Metrics reference](#metrics-reference)
- [Data-quality checks](#data-quality-checks)
- [Drift detection](#drift-detection)
- [Alerting](#alerting)
- [Grafana dashboard](#grafana-dashboard)
- [Running the examples](#running-the-examples)
- [Running the tests](#running-the-tests)
- [FAQ](#faq)
- [Support](#support)
- [License](#license)
Why this exists
A model in production needs four signals: volume/throughput, latency,
errors, and drift — plus data quality underneath them all. Most teams
bolt this together from prometheus_client, great_expectations, scipy.stats,
and a pile of glue code, then fight the dependency tree every time their serving
image is rebuilt.
This suite gives you all five signals from a single stream of
record_prediction() calls, implemented from scratch on the standard library so it
can drop into a constrained serving environment (a locked-down base image, a
lambda, an air-gapped box) without adding a single dependency.
What's included
- Prediction metrics (
metrics.py) — a thread-safePredictionMonitorthat
derives volume, error rate, latency percentiles (p50/p90/p95/p99), throughput,
output-value distribution, and per-class counts from one call per request.
- Rolling storage (
store.py) — time- and size-boundedRollingWindow/
MetricStore primitives with deterministic, event-time eviction (fully testable
by passing explicit timestamps).
- Data-quality expectations (
data_quality.py) — a pandas-free, 8-expectation
framework (DataQualitySuite) for schema, null, range, type, set-membership,
cardinality and table-shape checks, with error/warning severities.
- Drift detection (
drift_monitor.py) — online Population Stability Index,
two-sample Kolmogorov-Smirnov (with an asymptotic p-value), and Jensen-Shannon
divergence, wrapped in numeric/categorical detectors and a DriftMonitor.
- Alerting (
alerting.py) — a statefulAlertManagerwithThresholdRule
(with Prometheus-style for_seconds dwell) and a self-tuning z-score
... continues with setup instructions, usage examples, and more.