← Back to all products

Experiment Tracking Pack

$29

Experiment tracking with W&B and MLflow, custom dashboards, metrics comparison, and reproducibility tooling.

18 filesv1.0.0
PythonYAMLMarkdownJSONMLflow

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 18 files

experiment-tracking-pack/ ├── LICENSE ├── README.md ├── config.example.yaml ├── examples/ │ ├── compare_experiments.py │ └── track_training_run.py ├── guides/ │ ├── choosing-a-tracker.md │ └── experiment-tracking-guide.md ├── src/ │ └── exp_tracking/ │ ├── __init__.py │ ├── base.py │ ├── compare.py │ ├── custom_tracker.py │ ├── factory.py │ ├── mlflow_tracker.py │ ├── sweep.py │ └── wandb_tracker.py └── tests/ ├── test_base.py └── test_custom_tracker.py

Documentation Preview README excerpt

Experiment Tracking Pack

One experiment-tracking interface, three interchangeable backends. Instrument

your ML training loop once and send the results to Weights & Biases,

MLflow, or a zero-dependency local SQLite store — switching between them

with a single line of configuration.

The W&B and MLflow SDKs are imported lazily, so import exp_tracking (and the

entire test suite) works with nothing but the Python standard library

installed. You only need wandb or mlflow when you actually use that backend.

Features

  • Unified ExperimentTracker interface — start_run, log_params,

log_metrics, log_artifact, end_run, plus list_runs / get_metric_history

for reading results back.

  • Three backends, identical API:
  • CustomTracker — local, SQLite-backed, pure stdlib (great for tests/CI/offline).
  • WandbTracker — Weights & Biases, with artifact and sweep support.
  • MlflowTracker — MLflow Tracking, local mlruns/ or a self-hosted server.
  • run() context manager that always finalizes a run — FINISHED on success,

FAILED (and re-raised) on exception.

  • Hyperparameter sweeps — grid and random search expansion, plus

build_wandb_sweep_config for W&B Bayesian sweeps.

  • Cross-backend comparison — compare_runs builds a table from any backend,

rendered as ASCII, Markdown, or CSV, with best_run to pick the winner.

  • Config-driven backend selection — choose the backend from a YAML/JSON file.
  • Metric validation — non-numeric / NaN / inf values are rejected with a

clear local error before they reach a backend.

Installation

The pack itself needs no installation — drop src/exp_tracking/ onto your

PYTHONPATH (the examples and tests do this automatically). Install a backend

SDK only if you use it:


pip install wandb        # for the W&B backend  (then: wandb login)
pip install mlflow       # for the MLflow backend
pip install pyyaml       # only if you load a .yaml config (JSON needs nothing)
# the local SQLite backend, sweeps, and comparison need no third-party packages

Quick Start


from exp_tracking import get_tracker, compare_runs, format_table

# Pick a backend by name. "custom" runs locally with zero setup.
tracker = get_tracker("custom", db_path="runs.db", project="image-classification")

# Track a training loop. The context manager finalizes the run for you.
with tracker.run(run_name="adam-lr1e-3", config={"lr": 1e-3, "epochs": 10}) as run:
    for epoch in range(10):
        loss = 1.0 / (epoch + 1)
        tracker.log_metrics({"train_loss": loss, "val_accuracy": 0.5 + epoch / 20},
                            step=epoch)

# Compare every run in the project and print a table.
table = compare_runs(tracker, metric_keys=["val_accuracy", "train_loss"])

*... continues with setup instructions, usage examples, and more.*

Code Sample .py preview

src/exp_tracking/base.py """ Abstract experiment-tracking interface and shared data models. Every backend in this pack — Weights & Biases (:mod:`exp_tracking.wandb_tracker`), MLflow (:mod:`exp_tracking.mlflow_tracker`), and the bundled SQLite-backed local tracker (:mod:`exp_tracking.custom_tracker`) — implements the same :class:`ExperimentTracker` interface defined here. Code written against this interface can switch backends by changing a single line; see :mod:`exp_tracking.factory`. This module is intentionally pure standard library so it imports cleanly in any environment, including CI runners that have neither ``wandb`` nor ``mlflow`` installed. Backend-specific imports live inside the concrete tracker modules and are loaded lazily, only when that backend is actually used. """ from __future__ import annotations import abc import enum from collections.abc import Iterator, Mapping from contextlib import contextmanager from dataclasses import dataclass, field from numbers import Real from typing import Any, Optional __all__ = [ "TrackerError", "RunStatus", "RunInfo", "MetricPoint", "ExperimentTracker", "flatten_params", "coerce_metric_value", ] class TrackerError(RuntimeError): """Raised when a tracker is used incorrectly. Examples include logging metrics before :meth:`ExperimentTracker.start_run` has been called, or passing a non-numeric value where a metric is expected. Using a dedicated exception type lets callers distinguish *their* misuse from genuine backend/network failures raised by ``wandb`` or ``mlflow``. """ class RunStatus(enum.Enum): """Lifecycle state of a single experiment run. # ... 276 more lines ...