PySpark Utils Library
Battle-tested PySpark utility functions for transformations, data quality, SCD, schema evolution, and lineage.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 31 files
Documentation Preview README excerpt
PySpark Utils Library
Battle-tested utility functions for PySpark data engineering — transformations, data quality, SCD, schema evolution, logging, dedup, and DataFrame diffing.
Stop rewriting the same PySpark boilerplate on every project. This library gives you the production-ready building blocks that data engineering teams use daily — fully typed, tested, and documented.
What's Inside
| Module | What It Does |
|---|---|
| transformations | 15 reusable DataFrame transforms: column cleaning, casting, flattening, pivoting, hashing |
| data_quality | Chainable DQ validation framework with structured reports and severity levels |
| scd | SCD Type 1 (overwrite) and Type 2 (full history) merge utilities for Delta Lake |
| schema_utils | Schema comparison, evolution, DDL conversion, and compatibility checking |
| logging_utils | Structured pipeline logging with correlation IDs, metrics, and Delta table sink |
| dedup | Window-based, hash-based, and fuzzy deduplication strategies |
| diff | DataFrame comparison with row-level, column-level, and schema diffs |
20 files — every one fully type-hinted, tested, and documented.
Quick Start
1. Install
pip install pyspark-utils-library
Or install from the source directory:
pip install -e .
For fuzzy deduplication support (Levenshtein distance):
pip install pyspark-utils-library[fuzzy]
For development (testing, linting, type checking):
pip install pyspark-utils-library[dev]
2. Import and Use
from pyspark_utils.transformations import clean_column_names, add_metadata_columns
from pyspark_utils.data_quality import DQValidator
from pyspark_utils.scd import scd2_merge
from pyspark_utils.logging_utils import PipelineLogger
# Clean and enrich a raw DataFrame
df = spark.read.table("bronze.raw_orders")
*... continues with setup instructions, usage examples, and more.*