← Back to all products

ML Monitoring Suite

$39

Model monitoring dashboards, alert configurations, data quality checks, and performance tracking.

17 filesv1.0.0
PythonMarkdownJSONGrafanaPrometheus

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 17 files

ml-monitoring-suite/ ├── LICENSE ├── README.md ├── dashboards/ │ └── grafana-ml-monitoring.json ├── examples/ │ ├── data_quality_check.py │ └── monitor_predictions.py ├── guides/ │ ├── alerting-setup.md │ └── monitoring-guide.md ├── src/ │ └── ml_monitoring/ │ ├── __init__.py │ ├── alerting.py │ ├── data_quality.py │ ├── drift_monitor.py │ ├── exporters.py │ ├── metrics.py │ └── store.py └── tests/ ├── test_data_quality.py └── test_metrics.py

Documentation Preview README excerpt

ML Monitoring Suite

Production model monitoring with zero third-party dependencies. A small,

batteries-included Python package that keeps a served model healthy: prediction /

latency / throughput metrics, data-quality expectations, online drift detection

(PSI / KS / Jensen-Shannon), a stateful alert engine, Prometheus + JSON exporters,

and a ready-to-import Grafana dashboard.

Everything is pure Python standard library (3.9+), so it imports and runs in any

environment your model already runs in — no pip install required to use it.

Table of Contents

  • [Why this exists](#why-this-exists)
  • [What's included](#whats-included)
  • [Requirements](#requirements)
  • [Installation](#installation)
  • [Project structure](#project-structure)
  • [Quick start](#quick-start)
  • [The modules](#the-modules)
  • [Metrics reference](#metrics-reference)
  • [Data-quality checks](#data-quality-checks)
  • [Drift detection](#drift-detection)
  • [Alerting](#alerting)
  • [Grafana dashboard](#grafana-dashboard)
  • [Running the examples](#running-the-examples)
  • [Running the tests](#running-the-tests)
  • [FAQ](#faq)
  • [Support](#support)
  • [License](#license)

Why this exists

A model in production needs four signals: volume/throughput, latency,

errors, and drift — plus data quality underneath them all. Most teams

bolt this together from prometheus_client, great_expectations, scipy.stats,

and a pile of glue code, then fight the dependency tree every time their serving

image is rebuilt.

This suite gives you all five signals from a single stream of

record_prediction() calls, implemented from scratch on the standard library so it

can drop into a constrained serving environment (a locked-down base image, a

lambda, an air-gapped box) without adding a single dependency.

What's included

  • Prediction metrics (metrics.py) — a thread-safe PredictionMonitor that

derives volume, error rate, latency percentiles (p50/p90/p95/p99), throughput,

output-value distribution, and per-class counts from one call per request.

  • Rolling storage (store.py) — time- and size-bounded RollingWindow /

MetricStore primitives with deterministic, event-time eviction (fully testable

by passing explicit timestamps).

  • Data-quality expectations (data_quality.py) — a pandas-free, 8-expectation

framework (DataQualitySuite) for schema, null, range, type, set-membership,

cardinality and table-shape checks, with error/warning severities.

  • Drift detection (drift_monitor.py) — online Population Stability Index,

two-sample Kolmogorov-Smirnov (with an asymptotic p-value), and Jensen-Shannon

divergence, wrapped in numeric/categorical detectors and a DriftMonitor.

  • Alerting (alerting.py) — a stateful AlertManager with ThresholdRule

(with Prometheus-style for_seconds dwell) and a self-tuning z-score

... continues with setup instructions, usage examples, and more.

Code Sample .py preview

src/ml_monitoring/alerting.py """ Rule engine and pluggable alert sinks for model monitoring. You register :class:`Rule` objects with an :class:`AlertManager`, then feed the manager a flat ``{metric_name: value}`` mapping each evaluation cycle. The manager tracks alert *state* across cycles, so it fires an alert once when a rule starts breaching and emits a matching *resolved* event when it recovers -- no duplicate spam, and you always know when an incident clears. Two rule families are provided: * :class:`ThresholdRule` -- the workhorse: ``metric <op> threshold`` with an optional ``for_seconds`` dwell time (the breach must persist before firing, exactly like a Prometheus ``for:`` clause), which suppresses flapping. * :class:`AnomalyRule` -- a self-tuning z-score detector that learns a rolling mean/standard deviation and fires when a value is more than ``z_threshold`` standard deviations away. Good for metrics with no obvious fixed threshold. Alerts are dispatched to any number of :class:`AlertSink` implementations (:class:`ConsoleSink`, :class:`JSONLinesSink`, :class:`InMemorySink`, :class:`CallableSink`); a failing sink never blocks the others. """ from __future__ import annotations import json import logging import time from dataclasses import dataclass, field from enum import Enum from typing import Callable, Optional from .store import RollingWindow __all__ = [ "Severity", "Alert", "Rule", "ThresholdRule", "AnomalyRule", "AlertSink", "ConsoleSink", "InMemorySink", "JSONLinesSink", "CallableSink", "AlertManager", ] _LOGGER = logging.getLogger("ml_monitoring.alerting") # ... 346 more lines ...