← Back to all products
$19
Data Cleaning Playbook
Step-by-step data quality frameworks: deduplication, standardization, outlier detection, missing value strategies, and validation rules.
PythonYAMLMarkdownJSON
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 34 files
data-cleaning-playbook/
├── LICENSE
├── README.md
├── examples/
│ ├── __pycache__/
│ │ └── cleaning_pipeline.cpython-312.pyc
│ ├── cleaning_pipeline.py
│ └── messy_customers.csv
├── free-sample.zip
├── guide/
│ ├── deduplication_strategies.md
│ ├── missing_data_playbook.md
│ └── outlier_detection_guide.md
├── guides/
│ ├── deduplication_strategies.md
│ ├── missing_data_playbook.md
│ └── outlier_detection_guide.md
├── index.html
├── interactive.html
├── requirements.txt
├── rules/
│ ├── common_patterns.yaml
│ └── data_quality_rules.yaml
├── src/
│ └── cleaners/
│ ├── __init__.py
│ ├── __pycache__/
│ │ ├── __init__.cpython-312.pyc
│ │ ├── deduplication.cpython-312.pyc
│ │ ├── main.cpython-312.pyc
│ │ ├── missing_values.cpython-312.pyc
│ │ ├── outliers.cpython-312.pyc
│ │ ├── standardization.cpython-312.pyc
│ │ └── validation_engine.cpython-312.pyc
│ ├── deduplication.py
│ ├── main.py
│ ├── missing_values.py
│ ├── outliers.py
│ ├── standardization.py
│ └── validation_engine.py
└── tests/
├── __pycache__/
│ └── test_cleaners.cpython-312-pytest-9.0.2.pyc
└── test_cleaners.py
Documentation Preview README excerpt
Data Cleaning Playbook
A comprehensive toolkit combining deep playbook guides with runnable Python cleaners. Covers deduplication, standardization, outlier detection, missing-value strategies, and a rule-based validation engine.
Features
- Deduplication Engine — Exact-match, fuzzy-match, and time-window dedup strategies
- Standardization — Date formats, addresses, names, phone numbers, categorical encoding
- Outlier Detection — IQR, z-score, Modified Z-Score (MAD), and isolation-based methods
- Missing Value Strategies — Pattern analysis, imputation hierarchy, and indicator generation
- Validation Rule Engine — Declarative YAML rules for automated data quality checks
Quick Start
pip install -r requirements.txt
python -m src.cleaners.main
Structure
src/cleaners/
├── __init__.py # Package exports
├── main.py # Demo runner
├── deduplication.py # Exact and fuzzy deduplication
├── standardization.py # Date/name/address standardizers
├── outliers.py # IQR, z-score, MAD outlier detection
├── missing_values.py # Missing value analysis and imputation
├── validation_engine.py # YAML-driven rule validation
rules/
├── data_quality_rules.yaml # Example validation ruleset
├── common_patterns.yaml # Regex patterns for data standardization
examples/
├── messy_customers.csv # Sample messy data for testing
├── cleaning_pipeline.py # End-to-end cleaning example
guides/
├── deduplication_strategies.md # When to use which dedup approach
├── outlier_detection_guide.md # Choosing the right detection method
├── missing_data_playbook.md # Decision framework for missing values
Usage
Detect and Remove Duplicates
from src.cleaners.deduplication import find_duplicates, deduplicate
dupes = find_duplicates(df, subset=["email"], method="exact")
clean_df = deduplicate(df, subset=["email"], keep="latest", date_col="created_at")
Standardize Dates and Names
from src.cleaners.standardization import standardize_dates, standardize_names
df["date"] = standardize_dates(df["date_raw"], output_format="%Y-%m-%d")
df["name"] = standardize_names(df["name_raw"])
Detect Outliers
*... continues with setup instructions, usage examples, and more.*
Code Sample .py preview
src/cleaners/deduplication.py
"""
Deduplication engine supporting exact-match, fuzzy-match, and time-window strategies.
Identifies and removes duplicate records using configurable matching logic.
"""
from __future__ import annotations
import logging
import re
from dataclasses import dataclass
from typing import Literal
import numpy as np
import pandas as pd
logger = logging.getLogger(__name__)
@dataclass
class DuplicateReport:
"""Summary of duplicate detection results."""
total_rows: int
duplicate_groups: int
duplicate_rows: int
duplicate_pct: float
sample_duplicates: pd.DataFrame
def _is_text(series: pd.Series) -> bool:
"""True for text columns under both classic (object) and pandas>=3 (str) dtypes."""
return pd.api.types.is_object_dtype(series) or pd.api.types.is_string_dtype(series)
def find_duplicates(
df: pd.DataFrame,
subset: list[str] | None = None,
method: Literal["exact", "normalized", "fuzzy_simple"] = "exact",
) -> DuplicateReport:
"""Identify duplicate records in a DataFrame.
Args:
df: Input DataFrame.
subset: Columns to check for duplicates. None means all columns.
method: Matching method.
- 'exact': Exact string match on raw values
- 'normalized': Match after lowercasing and stripping whitespace
- 'fuzzy_simple': Match after removing non-alphanumeric chars
Returns:
# ... 125 more lines ...