← Back to all products

ML Data Versioning

$29

DVC setup, data pipeline versioning, experiment reproducibility, and artifact management workflows.

17 filesv1.0.0
PythonYAMLMarkdownJSON

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 17 files

ml-data-versioning/ ├── LICENSE ├── README.md ├── dvc/ │ ├── dvc.yaml │ └── params.yaml ├── examples/ │ ├── reproduce_experiment.py │ └── version_dataset.py ├── guides/ │ ├── dvc-setup.md │ └── reproducibility-guide.md ├── src/ │ └── data_versioning/ │ ├── __init__.py │ ├── hashing.py │ ├── lineage.py │ ├── pipeline.py │ ├── reproducibility.py │ └── snapshots.py └── tests/ ├── test_hashing.py └── test_lineage.py

Documentation Preview README excerpt

ML Data Versioning

A standard-library Python toolkit for **reproducible dataset versioning, lineage,

and DVC pipelines**. Hash datasets deterministically, snapshot and diff them

without any infrastructure, generate a valid dvc.yaml, build a provenance graph,

and capture the run environment that makes an experiment reproducible.

Every module is pure Python (no third-party runtime dependencies), so it runs

anywhere Python 3.9+ does and is fully unit-tested.

Why this exists

A versioned dataset is only half of reproducibility. To recreate a result you

need to pin four things together — the data, the code, the configuration, and

the environment. DVC handles the first three superbly; this toolkit fills the

gaps:

  • Before/without DVC — content-addressable hashing and a lightweight

snapshot store give you an exact "did this dataset change?" check with zero

setup (great for CI scratch space, notebooks, and audits).

  • Alongside DVC — generate dvc.yaml from code, render the pipeline as a

lineage graph for provenance/impact analysis, and capture the run environment

(seeds, params, package versions, git commit) that DVC does not record itself.

Features

  • Deterministic content hashing (hashing.py) — Merkle-style tree_hash

for a whole directory; identical bytes → identical hash on any OS, independent

of walk order and file mtimes. Streams large files; configurable ignore rules.

  • Dataset snapshots + diffs (snapshots.py) — capture, store, list, and

diff dataset versions in a tiny JSON-backed .snapshots/ registry, referenced

by label or hash prefix (git-style short SHAs).

  • Lineage graph (lineage.py) — a typed dataset → stage → output DAG with

topological ordering, cycle detection, provenance() (where did this come

from?) and impacted_by() (what must I recompute?), serializable to JSON and

Graphviz DOT.

  • DVC pipeline builder (pipeline.py) — define stages in code and emit a

valid dvc.yaml / params.yaml without PyYAML, including the `path:

{cache: false}` long form for metrics and plots.

  • Reproducibility capture (reproducibility.py) — seed every RNG, snapshot

the run environment, diff two environments to explain drift, and bundle

everything into a single RunManifest.

Quick start

No installation needed — the package is importable straight from src/.


# Run the test suite (standard library only; no pip installs required)
python -m unittest discover -s tests -v

# See dataset hashing, snapshots, and diffs in action
python examples/version_dataset.py

# See a generated dvc.yaml, a lineage graph, and an environment-drift report
python examples/reproduce_experiment.py

Use it in your own code:

... continues with setup instructions, usage examples, and more.

Code Sample .py preview

src/data_versioning/hashing.py """ Content-addressable hashing for datasets and ML artifacts. This module computes deterministic, reproducible content hashes for individual files and whole directory trees. It is the foundation for dataset versioning: two byte-identical datasets always produce the same ``tree_hash`` regardless of the machine, the filesystem walk order, or the files' modification times. The approach mirrors how content-addressable stores work in practice (Git objects, DVC's cache): the *content* determines the address, not the path or the timestamp. A directory is hashed as a small Merkle-style manifest — every file is hashed independently, the (relpath, hash, size) triples are sorted into a canonical order, and the manifest itself is hashed to produce one stable ``tree_hash`` for the whole dataset. Design goals ------------ * **Deterministic** — identical bytes produce an identical hash on any OS, every run. ``os.walk`` order, inode numbers, and mtimes never leak into the result. * **Streaming** — large files are read in fixed-size chunks, so a multi-gigabyte parquet shard never has to fit in memory. * **Self-contained** — standard library only (``hashlib``, ``os``, ``fnmatch``). Typical use:: from data_versioning.hashing import hash_directory, diff_directories before = hash_directory("data/raw") print(before.short_hash, before.file_count, before.total_size) # ... the dataset changes on disk ... after = hash_directory("data/raw") delta = diff_directories(before, after) if delta.changed: print(delta.summary()) """ from __future__ import annotations import fnmatch import hashlib import os from dataclasses import dataclass from pathlib import Path from typing import Iterator, Sequence # Hash algorithms we accept. md5 is DVC's default cache key (chosen for speed, # not cryptographic strength); sha1/sha256 are offered for stronger collision # resistance when you care more about integrity than throughput. SUPPORTED_ALGORITHMS: tuple[str, ...] = ("md5", "sha1", "sha256") # ... 353 more lines ...