← Back to all products
$29
Data Labeling Pipeline
Annotation workflow tools, quality assurance scripts, active learning selectors, and labeling interface templates.
PythonYAMLMarkdownJSON
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 35 files
data-labeling-pipeline/
├── LICENSE
├── README.md
├── configs/
│ ├── labeling_config.yaml
│ └── schema_config.yaml
├── free-sample.zip
├── guide/
│ ├── 01-annotation-workflow.md
│ └── 02-quality-assurance.md
├── guides/
│ ├── annotation-workflow.md
│ └── quality-assurance.md
├── index.html
├── interactive.html
├── requirements.txt
├── src/
│ ├── __init__.py
│ ├── __pycache__/
│ │ ├── __init__.cpython-312.pyc
│ │ ├── active_learning.cpython-312.pyc
│ │ ├── agreement.cpython-312.pyc
│ │ ├── quality_audit.cpython-312.pyc
│ │ ├── schema.cpython-312.pyc
│ │ └── workflow.cpython-312.pyc
│ ├── active_learning.py
│ ├── agreement.py
│ ├── quality_audit.py
│ ├── schema.py
│ └── workflow.py
├── templates/
│ └── labeling_interface.html
└── tests/
├── __pycache__/
│ ├── test_active_learning.cpython-312-pytest-9.0.2.pyc
│ ├── test_active_learning.cpython-312.pyc
│ ├── test_agreement.cpython-312-pytest-9.0.2.pyc
│ ├── test_agreement.cpython-312.pyc
│ ├── test_quality_audit.cpython-312-pytest-9.0.2.pyc
│ └── test_quality_audit.cpython-312.pyc
├── test_active_learning.py
├── test_agreement.py
└── test_quality_audit.py
Documentation Preview README excerpt
Data Labeling Pipeline
A complete toolkit for managing ML data labeling workflows — from annotator assignment through quality auditing to model-ready export. Includes inter-annotator agreement metrics (Cohen's κ, Fleiss' κ, Krippendorff's α), active learning sample selectors, label quality auditing, and a workflow orchestration engine.
Why This Exists
Labeling is the bottleneck of supervised ML. Most teams either:
- Over-label: waste budget on redundant annotations without measuring agreement
- Under-validate: ship noisy labels that poison model performance
- Re-do work: discover label quality issues too late and re-label from scratch
This toolkit gives you the measurement and workflow tools to label efficiently, catch errors early, and know when you're done.
Features
- Inter-annotator agreement — Cohen's κ (2 raters), Fleiss' κ (N raters), Scott's π, Krippendorff's α (with missing data support). All implemented in stdlib math — no scipy dependency for the core metrics.
- Active learning selectors — Uncertainty, margin, entropy, random baseline, and query-by-committee strategies. Feed in
predict_probaoutput, get back the most informative sample indices. - Label quality auditing — Annotator scorecards (agreement, speed, bias), suspect label detection, distribution drift testing (chi-squared), and automated audit reports.
- Schema management — Define labeling taxonomies with categories, hierarchy, colors, and keyboard shortcuts. Validate annotations against schemas. Version and migrate schemas over time.
- Workflow engine — Task creation, batch assignment (round-robin and difficulty-stratified), label submission, consensus resolution (majority vote / unanimous), adjudication, and training data export.
- Labeling UI template — Ready-to-deploy HTML/CSS/JS annotation interface with keyboard shortcuts, progress tracking, and guidelines sidebar.
File Structure
data-labeling-pipeline/
├── README.md # This file
├── LICENSE # MIT License
├── requirements.txt # Python dependencies
├── src/
│ ├── __init__.py # Package metadata
│ ├── agreement.py # Cohen's κ, Fleiss' κ, Scott's π, Krippendorff's α
│ ├── active_learning.py # Sample selection strategies
│ ├── quality_audit.py # Annotator scoring, suspect detection, drift
│ ├── schema.py # Label schema definition and validation
│ └── workflow.py # Annotation workflow orchestration
├── tests/
│ ├── test_agreement.py # Agreement metric tests
│ ├── test_active_learning.py # Selector tests
│ └── test_quality_audit.py # Audit function tests
├── configs/
│ ├── labeling_config.yaml # Master workflow configuration
│ └── schema_config.yaml # Label taxonomy definition
├── templates/
│ └── labeling_interface.html # Browser-based annotation UI
└── guides/
├── annotation-workflow.md # End-to-end workflow guide
└── quality-assurance.md # QA strategy and metrics guide
Quick Start
1. Compute Inter-Annotator Agreement
from src.agreement import cohens_kappa, fleiss_kappa
# Two raters
rater_a = ["pos", "neg", "pos", "neg", "pos", "neg", "pos", "neg"]
rater_b = ["pos", "neg", "neg", "neg", "pos", "neg", "pos", "pos"]
*... continues with setup instructions, usage examples, and more.*
Code Sample .py preview
src/active_learning.py
"""
Active learning sample selectors — choose the most informative
unlabeled samples to send to annotators next, so you label fewer
items while training a better model.
All selectors take a probability matrix (N_samples × N_classes) and
return indices of the top-k most informative samples. The matrix
typically comes from `model.predict_proba(X_unlabeled)`.
Strategies implemented:
1. Uncertainty sampling — lowest max-class probability
2. Margin sampling — smallest gap between top-two probabilities
3. Entropy sampling — highest Shannon entropy
4. Random baseline — uniform random (for comparison)
5. Committee disagreement — query-by-committee via vote entropy
All math uses stdlib only (math.log, sorted, etc.).
"""
from __future__ import annotations
import logging
import math
import random
from dataclasses import dataclass, field
from typing import Callable, Dict, List, Optional, Sequence, Tuple
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Type alias — a probability matrix is List[List[float]] where
# probs[i][j] = P(class j | sample i)
# ---------------------------------------------------------------------------
ProbMatrix = List[List[float]]
@dataclass
class SelectionResult:
"""Returned by every selector: the chosen indices plus the score that
ranked them, so you can inspect *why* each sample was selected."""
indices: List[int]
scores: List[float]
strategy: str
pool_size: int
budget: int
def top(self, n: int = 5) -> List[Tuple[int, float]]:
"""Return top-n (index, score) pairs for quick inspection."""
# ... 335 more lines ...