Choosing a Tracker: W&B vs MLflow vs Local SQLite

This pack ships three backends behind one interface so you are never locked in.

But which should you actually *use*? This guide gives you a decision framework,

an honest comparison, and migration notes. Because every backend implements the

same ExperimentTracker interface, switching later is a one-line change — so

optimize for where you are today, not where you might be in two years.

TL;DR decision tree

Are you offline, in CI, or writing tests?
└─ Yes → custom (local SQLite). Zero setup, zero network, fully deterministic.
└─ No → Do you need a hosted UI, sweeps, and team sharing with no ops work?
        └─ Yes → wandb. Best-in-class dashboards and hyperparameter sweeps.
        └─ No → Do you want self-hosted, open-source, and model-registry features
                under your own control?
                └─ Yes → mlflow. Run your own server, own your data.
                └─ Unsure → start with custom, switch when a real need appears.

Side-by-side comparison

When to pick the local SQLite backend

The custom tracker is not a toy — it is the right default in several real

situations:

• Unit tests and CI. It needs no credentials and no network, so test suites

stay fast and hermetic. This pack's own tests use it.

• Air-gapped or regulated environments where data cannot leave the machine.
• Quick local experiments where spinning up a server is more work than the

experiment itself.

• Teaching and demos where you want the audience to run code immediately.

Its limits are equally clear: there is no hosted UI (you compare via

format_table/to_markdown), and "sharing" means sharing the SQLite file. When

you outgrow it, the interface makes moving up painless.

When to pick Weights & Biases

Choose wandb when you want a polished, hosted experience with essentially no

operational burden:

• Rich dashboards with interactive plots, parallel-coordinates views, and

run tables your whole team can browse.

• Hyperparameter sweeps with Bayesian optimization out of the box — build the

config with build_wandb_sweep_config and launch agents.

• Team workflows: reports, run sharing, and commenting are first-class.

Trade-offs: it is a third-party SaaS (your run data lives on their servers,

though an offline mode exists), and seats cost money beyond the free tier. Use

mode="offline" on locked-down machines and wandb sync later.

When to pick MLflow

Choose mlflow when you want open-source tracking you control end to end:

• Self-hosting: point the tracker at your own server

(tracking_uri="http://mlflow.internal:5000") or even just a local mlruns/

directory with no server at all.

• Model registry: MLflow's registry is excellent for staging/production model

promotion workflows.

• No per-seat cost: you pay for infrastructure, not licenses.

Trade-offs: someone has to run and maintain the server for team use, and the UI,

while solid, is less flashy than W&B's. For many organizations that control is

exactly the point.

Cost and hosting at a glance

• Local SQLite: free; the only "cost" is that collaboration is manual.
• W&B: generous free tier for individuals; paid plans for teams and private

hosting. You trade dollars for zero ops.

• MLflow: the software is free; you pay in infrastructure and maintenance

time for a shared server (a small VM is often enough).

Migrating between backends

Because all three implement ExperimentTracker, your *training code* never

changes — only how you construct the tracker:

# Local during development...
tracker = get_tracker("custom", db_path="runs.db", project="demo")

# ...W&B when you want dashboards...
tracker = get_tracker("wandb", project="demo", entity="your-org")

# ...MLflow when you self-host.
tracker = get_tracker("mlflow", experiment_name="demo",
                      tracking_uri="http://mlflow.internal:5000")

Better still, select the backend from config so it's an ops decision, not a code

change:

from exp_tracking import load_config, tracker_from_config

tracker = tracker_from_config(load_config("config.yaml"))

There is no automated cross-backend *history* migration — runs already recorded

in W&B don't move to MLflow. But because comparison reads through the same

interface, you can point compare_runs at whichever backend holds the runs you

care about. In practice teams run custom in CI and a hosted backend for real

training, which this design supports natively.

A reasonable default path

1. Start on custom while you prototype — instant, offline, testable.

2. Graduate to wandb when you want dashboards and sweeps without ops, or to

self-hosted mlflow when data control and a model registry matter more.

3. Keep custom in your test suite forever — it is the fastest, most

reliable way to verify your tracking code without external dependencies.

You are not choosing a tracker for life. You are choosing the one that removes

the most friction from your *next* experiment.