ML Data Versioning
DVC setup, data pipeline versioning, experiment reproducibility, and artifact management workflows.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 17 files
Documentation Preview README excerpt
ML Data Versioning
A standard-library Python toolkit for **reproducible dataset versioning, lineage,
and DVC pipelines**. Hash datasets deterministically, snapshot and diff them
without any infrastructure, generate a valid dvc.yaml, build a provenance graph,
and capture the run environment that makes an experiment reproducible.
Every module is pure Python (no third-party runtime dependencies), so it runs
anywhere Python 3.9+ does and is fully unit-tested.
Why this exists
A versioned dataset is only half of reproducibility. To recreate a result you
need to pin four things together — the data, the code, the configuration, and
the environment. DVC handles the first three superbly; this toolkit fills the
gaps:
- Before/without DVC — content-addressable hashing and a lightweight
snapshot store give you an exact "did this dataset change?" check with zero
setup (great for CI scratch space, notebooks, and audits).
- Alongside DVC — generate
dvc.yamlfrom code, render the pipeline as a
lineage graph for provenance/impact analysis, and capture the run environment
(seeds, params, package versions, git commit) that DVC does not record itself.
Features
- Deterministic content hashing (
hashing.py) — Merkle-styletree_hash
for a whole directory; identical bytes → identical hash on any OS, independent
of walk order and file mtimes. Streams large files; configurable ignore rules.
- Dataset snapshots + diffs (
snapshots.py) — capture, store, list, and
diff dataset versions in a tiny JSON-backed .snapshots/ registry, referenced
by label or hash prefix (git-style short SHAs).
- Lineage graph (
lineage.py) — a typed dataset → stage → output DAG with
topological ordering, cycle detection, provenance() (where did this come
from?) and impacted_by() (what must I recompute?), serializable to JSON and
Graphviz DOT.
- DVC pipeline builder (
pipeline.py) — define stages in code and emit a
valid dvc.yaml / params.yaml without PyYAML, including the `path:
{cache: false}` long form for metrics and plots.
- Reproducibility capture (
reproducibility.py) — seed every RNG, snapshot
the run environment, diff two environments to explain drift, and bundle
everything into a single RunManifest.
Quick start
No installation needed — the package is importable straight from src/.
# Run the test suite (standard library only; no pip installs required)
python -m unittest discover -s tests -v
# See dataset hashing, snapshots, and diffs in action
python examples/version_dataset.py
# See a generated dvc.yaml, a lineage graph, and an environment-drift report
python examples/reproduce_experiment.py
Use it in your own code:
... continues with setup instructions, usage examples, and more.