← Back to all products

Data Labeling Pipeline

$29

Annotation workflow tools, quality assurance scripts, active learning selectors, and labeling interface templates.

35 files
PythonYAMLMarkdownJSON

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 35 files

data-labeling-pipeline/ ├── LICENSE ├── README.md ├── configs/ │ ├── labeling_config.yaml │ └── schema_config.yaml ├── free-sample.zip ├── guide/ │ ├── 01-annotation-workflow.md │ └── 02-quality-assurance.md ├── guides/ │ ├── annotation-workflow.md │ └── quality-assurance.md ├── index.html ├── interactive.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── active_learning.cpython-312.pyc │ │ ├── agreement.cpython-312.pyc │ │ ├── quality_audit.cpython-312.pyc │ │ ├── schema.cpython-312.pyc │ │ └── workflow.cpython-312.pyc │ ├── active_learning.py │ ├── agreement.py │ ├── quality_audit.py │ ├── schema.py │ └── workflow.py ├── templates/ │ └── labeling_interface.html └── tests/ ├── __pycache__/ │ ├── test_active_learning.cpython-312-pytest-9.0.2.pyc │ ├── test_active_learning.cpython-312.pyc │ ├── test_agreement.cpython-312-pytest-9.0.2.pyc │ ├── test_agreement.cpython-312.pyc │ ├── test_quality_audit.cpython-312-pytest-9.0.2.pyc │ └── test_quality_audit.cpython-312.pyc ├── test_active_learning.py ├── test_agreement.py └── test_quality_audit.py

Documentation Preview README excerpt

Data Labeling Pipeline

A complete toolkit for managing ML data labeling workflows — from annotator assignment through quality auditing to model-ready export. Includes inter-annotator agreement metrics (Cohen's κ, Fleiss' κ, Krippendorff's α), active learning sample selectors, label quality auditing, and a workflow orchestration engine.

Why This Exists

Labeling is the bottleneck of supervised ML. Most teams either:

  • Over-label: waste budget on redundant annotations without measuring agreement
  • Under-validate: ship noisy labels that poison model performance
  • Re-do work: discover label quality issues too late and re-label from scratch

This toolkit gives you the measurement and workflow tools to label efficiently, catch errors early, and know when you're done.

Features

  • Inter-annotator agreement — Cohen's κ (2 raters), Fleiss' κ (N raters), Scott's π, Krippendorff's α (with missing data support). All implemented in stdlib math — no scipy dependency for the core metrics.
  • Active learning selectors — Uncertainty, margin, entropy, random baseline, and query-by-committee strategies. Feed in predict_proba output, get back the most informative sample indices.
  • Label quality auditing — Annotator scorecards (agreement, speed, bias), suspect label detection, distribution drift testing (chi-squared), and automated audit reports.
  • Schema management — Define labeling taxonomies with categories, hierarchy, colors, and keyboard shortcuts. Validate annotations against schemas. Version and migrate schemas over time.
  • Workflow engine — Task creation, batch assignment (round-robin and difficulty-stratified), label submission, consensus resolution (majority vote / unanimous), adjudication, and training data export.
  • Labeling UI template — Ready-to-deploy HTML/CSS/JS annotation interface with keyboard shortcuts, progress tracking, and guidelines sidebar.

File Structure


data-labeling-pipeline/
├── README.md                          # This file
├── LICENSE                            # MIT License
├── requirements.txt                   # Python dependencies
├── src/
│   ├── __init__.py                    # Package metadata
│   ├── agreement.py                   # Cohen's κ, Fleiss' κ, Scott's π, Krippendorff's α
│   ├── active_learning.py             # Sample selection strategies
│   ├── quality_audit.py               # Annotator scoring, suspect detection, drift
│   ├── schema.py                      # Label schema definition and validation
│   └── workflow.py                    # Annotation workflow orchestration
├── tests/
│   ├── test_agreement.py              # Agreement metric tests
│   ├── test_active_learning.py        # Selector tests
│   └── test_quality_audit.py          # Audit function tests
├── configs/
│   ├── labeling_config.yaml           # Master workflow configuration
│   └── schema_config.yaml             # Label taxonomy definition
├── templates/
│   └── labeling_interface.html        # Browser-based annotation UI
└── guides/
    ├── annotation-workflow.md          # End-to-end workflow guide
    └── quality-assurance.md            # QA strategy and metrics guide

Quick Start

1. Compute Inter-Annotator Agreement


from src.agreement import cohens_kappa, fleiss_kappa

# Two raters
rater_a = ["pos", "neg", "pos", "neg", "pos", "neg", "pos", "neg"]
rater_b = ["pos", "neg", "neg", "neg", "pos", "neg", "pos", "pos"]

*... continues with setup instructions, usage examples, and more.*

Code Sample .py preview

src/active_learning.py """ Active learning sample selectors — choose the most informative unlabeled samples to send to annotators next, so you label fewer items while training a better model. All selectors take a probability matrix (N_samples × N_classes) and return indices of the top-k most informative samples. The matrix typically comes from `model.predict_proba(X_unlabeled)`. Strategies implemented: 1. Uncertainty sampling — lowest max-class probability 2. Margin sampling — smallest gap between top-two probabilities 3. Entropy sampling — highest Shannon entropy 4. Random baseline — uniform random (for comparison) 5. Committee disagreement — query-by-committee via vote entropy All math uses stdlib only (math.log, sorted, etc.). """ from __future__ import annotations import logging import math import random from dataclasses import dataclass, field from typing import Callable, Dict, List, Optional, Sequence, Tuple logger = logging.getLogger(__name__) # --------------------------------------------------------------------------- # Type alias — a probability matrix is List[List[float]] where # probs[i][j] = P(class j | sample i) # --------------------------------------------------------------------------- ProbMatrix = List[List[float]] @dataclass class SelectionResult: """Returned by every selector: the chosen indices plus the score that ranked them, so you can inspect *why* each sample was selected.""" indices: List[int] scores: List[float] strategy: str pool_size: int budget: int def top(self, n: int = 5) -> List[Tuple[int, float]]: """Return top-n (index, score) pairs for quick inspection.""" # ... 335 more lines ...