← Back to all products

Data Cleaning Playbook

$19

Step-by-step data quality frameworks: deduplication, standardization, outlier detection, missing value strategies, and validation rules.

34 files
PythonYAMLMarkdownJSON

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 34 files

data-cleaning-playbook/ ├── LICENSE ├── README.md ├── examples/ │ ├── __pycache__/ │ │ └── cleaning_pipeline.cpython-312.pyc │ ├── cleaning_pipeline.py │ └── messy_customers.csv ├── free-sample.zip ├── guide/ │ ├── deduplication_strategies.md │ ├── missing_data_playbook.md │ └── outlier_detection_guide.md ├── guides/ │ ├── deduplication_strategies.md │ ├── missing_data_playbook.md │ └── outlier_detection_guide.md ├── index.html ├── interactive.html ├── requirements.txt ├── rules/ │ ├── common_patterns.yaml │ └── data_quality_rules.yaml ├── src/ │ └── cleaners/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── deduplication.cpython-312.pyc │ │ ├── main.cpython-312.pyc │ │ ├── missing_values.cpython-312.pyc │ │ ├── outliers.cpython-312.pyc │ │ ├── standardization.cpython-312.pyc │ │ └── validation_engine.cpython-312.pyc │ ├── deduplication.py │ ├── main.py │ ├── missing_values.py │ ├── outliers.py │ ├── standardization.py │ └── validation_engine.py └── tests/ ├── __pycache__/ │ └── test_cleaners.cpython-312-pytest-9.0.2.pyc └── test_cleaners.py

Documentation Preview README excerpt

Data Cleaning Playbook

A comprehensive toolkit combining deep playbook guides with runnable Python cleaners. Covers deduplication, standardization, outlier detection, missing-value strategies, and a rule-based validation engine.

Features

  • Deduplication Engine — Exact-match, fuzzy-match, and time-window dedup strategies
  • Standardization — Date formats, addresses, names, phone numbers, categorical encoding
  • Outlier Detection — IQR, z-score, Modified Z-Score (MAD), and isolation-based methods
  • Missing Value Strategies — Pattern analysis, imputation hierarchy, and indicator generation
  • Validation Rule Engine — Declarative YAML rules for automated data quality checks

Quick Start


pip install -r requirements.txt
python -m src.cleaners.main

Structure


src/cleaners/
├── __init__.py            # Package exports
├── main.py                # Demo runner
├── deduplication.py       # Exact and fuzzy deduplication
├── standardization.py     # Date/name/address standardizers
├── outliers.py            # IQR, z-score, MAD outlier detection
├── missing_values.py      # Missing value analysis and imputation
├── validation_engine.py   # YAML-driven rule validation
rules/
├── data_quality_rules.yaml  # Example validation ruleset
├── common_patterns.yaml     # Regex patterns for data standardization
examples/
├── messy_customers.csv      # Sample messy data for testing
├── cleaning_pipeline.py     # End-to-end cleaning example
guides/
├── deduplication_strategies.md   # When to use which dedup approach
├── outlier_detection_guide.md    # Choosing the right detection method
├── missing_data_playbook.md      # Decision framework for missing values

Usage

Detect and Remove Duplicates


from src.cleaners.deduplication import find_duplicates, deduplicate
dupes = find_duplicates(df, subset=["email"], method="exact")
clean_df = deduplicate(df, subset=["email"], keep="latest", date_col="created_at")

Standardize Dates and Names


from src.cleaners.standardization import standardize_dates, standardize_names
df["date"] = standardize_dates(df["date_raw"], output_format="%Y-%m-%d")
df["name"] = standardize_names(df["name_raw"])

Detect Outliers



*... continues with setup instructions, usage examples, and more.*

Code Sample .py preview

src/cleaners/deduplication.py """ Deduplication engine supporting exact-match, fuzzy-match, and time-window strategies. Identifies and removes duplicate records using configurable matching logic. """ from __future__ import annotations import logging import re from dataclasses import dataclass from typing import Literal import numpy as np import pandas as pd logger = logging.getLogger(__name__) @dataclass class DuplicateReport: """Summary of duplicate detection results.""" total_rows: int duplicate_groups: int duplicate_rows: int duplicate_pct: float sample_duplicates: pd.DataFrame def _is_text(series: pd.Series) -> bool: """True for text columns under both classic (object) and pandas>=3 (str) dtypes.""" return pd.api.types.is_object_dtype(series) or pd.api.types.is_string_dtype(series) def find_duplicates( df: pd.DataFrame, subset: list[str] | None = None, method: Literal["exact", "normalized", "fuzzy_simple"] = "exact", ) -> DuplicateReport: """Identify duplicate records in a DataFrame. Args: df: Input DataFrame. subset: Columns to check for duplicates. None means all columns. method: Matching method. - 'exact': Exact string match on raw values - 'normalized': Match after lowercasing and stripping whitespace - 'fuzzy_simple': Match after removing non-alphanumeric chars Returns: # ... 125 more lines ...