Alerting Setup Guide

This guide covers how to turn the metrics from the ML Monitoring Suite into

actionable alerts using the bundled rule engine (ml_monitoring.alerting). It

explains rule design, how to suppress flapping, how to route by severity, and how

to wire alerts into the destination of your choice — all with zero third-party

dependencies.

How the engine works

You register Rule objects with an AlertManager, then call

manager.evaluate(metrics, timestamp=...) once per cycle with a flat

{metric_name: value} mapping. The manager is stateful: it remembers which

alerts are currently firing, so each rule fires exactly once when it starts

breaching and emits a matching resolved event when it recovers. You get incident

*transitions*, not a wall of duplicate notifications.

from ml_monitoring import AlertManager, ThresholdRule, AnomalyRule, Severity
from ml_monitoring import ConsoleSink, JSONLinesSink

manager = AlertManager(
    sinks=[ConsoleSink(), JSONLinesSink("/var/log/ml-alerts.jsonl")],
    default_labels={"model": "churn_classifier", "env": "production"},
)

Each evaluation returns the list of new firing/resolved Alert events, and

manager.active_alerts() returns everything currently firing.

Building the metric snapshot

The engine evaluates a plain dict, which you assemble from the monitors each cycle.

Keep the keys stable — they are what rules match on.

latency = monitor.latency_window().stats()
drift_results = drift.check({"tenure_months": recent_batch})

snapshot = {
    "error_rate": monitor.error_rate(),
    "latency_p95_ms": latency.p95,
    "throughput_per_min": monitor.throughput_per_minute(),
    "tenure_drift_psi": drift_results["tenure_months"].drift_score,
    "dq_failures": len(report.failures),
}
events = manager.evaluate(snapshot, timestamp=time.time())

Threshold rules

ThresholdRule fires when metric threshold holds. Operators are

">", ">=", "<", "<=", "==", "!=". This is the right tool when you have a

known SLO.

manager.add_rule(ThresholdRule(
    "high_error_rate", "error_rate", ">", 0.05,
    severity=Severity.CRITICAL,
    for_seconds=300,          # must breach continuously for 5 minutes
))

Suppressing flapping with for_seconds

A metric that hovers around its threshold will otherwise fire and resolve

repeatedly. for_seconds is a dwell time, exactly like a Prometheus for: clause:

the rule must breach *continuously* for that long before the manager fires it. Use

it on any noisy signal (latency, error rate). A for_seconds=0 rule (the default)

fires immediately — appropriate for hard, must-never-happen conditions like a

data-quality error.

Because the dwell is measured against the timestamp you pass in, it is fully

deterministic and testable: feed timestamps 0, 5, 10 and a rule with

for_seconds=10 fires precisely on the third evaluation.

Anomaly rules

When a metric has no obvious fixed threshold (a per-model throughput, a feature

mean that varies by season), use AnomalyRule. It maintains its own rolling

baseline and fires when a value is more than z_threshold standard deviations from

the recent mean. It only starts alerting once min_samples of baseline have

accumulated, so it never fires on cold start.

manager.add_rule(AnomalyRule(
    "throughput_anomaly", "throughput_per_min",
    z_threshold=4.0,          # 4 sigma — conservative, few false positives
    window_size=200,          # baseline = last 200 observations
    min_samples=30,           # warm up before alerting
    severity=Severity.WARNING,
))

The baseline adapts over time, so a slow, legitimate ramp in traffic will not keep

alerting — only abrupt deviations do.

A complete starter rule set

These rules cover the four golden signals plus data quality. They are a good

production starting point for an online classifier.

manager.add_rule(ThresholdRule("high_error_rate", "error_rate", ">", 0.05,
                                severity=Severity.CRITICAL, for_seconds=300))
manager.add_rule(ThresholdRule("slow_p95", "latency_p95_ms", ">", 250.0,
                                severity=Severity.WARNING, for_seconds=300))
manager.add_rule(ThresholdRule("feature_drift", "tenure_drift_psi", ">=", 0.25,
                                severity=Severity.WARNING))
manager.add_rule(ThresholdRule("dq_failure", "dq_failures", ">", 0,
                                severity=Severity.CRITICAL))   # any failure pages
manager.add_rule(AnomalyRule("throughput_anomaly", "throughput_per_min",
                             z_threshold=4.0, min_samples=30))

Severity and routing

Severity has three levels — INFO, WARNING, CRITICAL. Map them to your

on-call policy:

• CRITICAL — wake someone up. Reserve for customer-visible breakage: high error

rate, data-quality errors that would feed garbage to the model.

• WARNING — create a ticket / Slack message. Drift, slow latency, throughput

anomalies — things to fix this week, not tonight.

• INFO — record only. Useful for audit trails and post-incident timelines.

The severity is attached to every Alert, so your sink can route on it.

Sinks: where alerts go

Sinks are pluggable; a failing sink never blocks the others (the manager catches and

logs sink exceptions). Built-in sinks:

A webhook bridge is just a CallableSink:

import json, urllib.request

def post_to_webhook(alert):
    payload = json.dumps(alert.as_dict()).encode("utf-8")
    req = urllib.request.Request(
        "https://hooks.docs.example.com/services/your-webhook",
        data=payload, headers={"Content-Type": "application/json"},
    )
    urllib.request.urlopen(req, timeout=5)

manager.add_sink(CallableSink(post_to_webhook))

Only fire the webhook for the severities you care about by filtering inside the

callable (if alert.severity is Severity.CRITICAL: ...).

Avoiding alert fatigue

Alert fatigue is the failure mode that makes monitoring worthless. Defend against

it deliberately:

1. Use for_seconds on every noisy signal. Latency and error rate should never

page on a single spiky scrape.

2. Page only on CRITICAL. Everything else is a ticket. If a WARNING pages you at

3am, downgrade it or add dwell time.

3. Alert on symptoms, not causes. Page on the error rate the customer feels, not

on every internal counter that might contribute to it.

4. Make drift a ticket, not a page. Drift is a slow process; a same-day ticket is

the right urgency. Only the significant band (PSI ≥ 0.25) is worth a rule.

5. Resolve loudly. The resolved events the manager emits are as valuable as the

firing ones — they tell on-call the incident is over without a manual all-clear.

Incident runbook (paste into your own docs)

When high_error_rate fires:

1. Check throughput — did errors rise, or did successful predictions fall? A

throughput collapse points upstream (caller/feature service), not at the model.

2. Inspect the latest data-quality report. A spike in ml_data_quality_failures_total

at the same time means malformed features, not a model fault.

3. Check ml_drift_score. Significant drift plus rising errors is the classic

"the world changed" signal — start a retrain, and consider rolling back to the

previous model version in the meantime.

4. If latency p95 also breached, suspect resource saturation (CPU/GPU, batch size,

a slow dependency) rather than the model weights.

5. Confirm recovery via the matching resolved alert before closing the incident.