← Back to all products

Spark ETL Framework

$29

Production-ready medallion architecture ETL framework for Databricks and Apache Spark.

16 filesv1.0.0
PythonYAMLMarkdownJSONDatabricksPySparkSparkDelta LakePostgreSQL

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 16 files

spark-etl-framework/ ├── LICENSE ├── README.md ├── configs/ │ ├── pipeline_config.yaml │ └── quality_rules.yaml ├── guides/ │ └── etl-patterns.md ├── notebooks/ │ ├── backfill.py │ └── run_pipeline.py ├── src/ │ ├── bronze_loader.py │ ├── config_manager.py │ ├── etl_base.py │ ├── gold_aggregator.py │ ├── quality_gate.py │ └── silver_transformer.py └── tests/ ├── conftest.py └── test_etl_base.py

Documentation Preview README excerpt

Spark ETL Framework

Production-ready medallion architecture ETL framework for Databricks and Apache Spark.

Build reliable, observable, and maintainable data pipelines with a battle-tested extract-transform-load pattern that scales from prototype to petabyte.

What You Get

  • Abstract ETL base class with built-in logging, metrics collection, and error handling
  • Medallion architecture (Bronze / Silver / Gold) with production patterns baked in
  • Data quality gates between every layer — catch issues before they propagate
  • YAML-driven configuration with environment overrides and secret scope integration
  • Source extractors for JDBC databases, file systems, and REST APIs with pagination and retry logic
  • SCD handler for Type 1 and Type 2 slowly changing dimension merges with soft deletes
  • Deduplication engine with exact, window, fuzzy, and hash-based strategies
  • Metrics collector for pipeline observability, SLA tracking, and batch comparison
  • Lineage tracker for data provenance, impact analysis, and Mermaid diagram export
  • Databricks notebooks for orchestration and date-range backfills
  • Comprehensive test suite with mock Spark sessions and fixture data
  • Architecture guide covering idempotency, partitioning, extraction, SCD, and dedup patterns

File Tree


spark-etl-framework/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── etl_base.py                    # Abstract base ETL class
│   ├── bronze_loader.py               # Bronze layer ingestion
│   ├── silver_transformer.py          # Silver layer transforms
│   ├── gold_aggregator.py             # Gold layer aggregations
│   ├── quality_gate.py                # Inter-layer quality checks
│   ├── config_manager.py              # YAML config + env overrides
│   ├── extractors/
│   │   ├── __init__.py
│   │   ├── jdbc_extractor.py          # JDBC database extraction (PostgreSQL, SQL Server, Oracle, MySQL)
│   │   ├── file_extractor.py          # File extraction (CSV, JSON, Parquet, Avro, ORC)
│   │   └── api_extractor.py           # REST API extraction with pagination & rate limiting
│   ├── transformers/
│   │   ├── __init__.py
│   │   ├── scd_handler.py             # SCD Type 1 & 2 merge operations
│   │   └── deduplication.py           # Dedup strategies (exact, window, fuzzy, hash)
│   └── utils/
│       ├── __init__.py
│       ├── metrics_collector.py       # Pipeline execution metrics & SLA tracking
│       └── lineage_tracker.py         # Data lineage DAG with Mermaid export
├── configs/
│   ├── pipeline_config.yaml           # Pipeline configuration
│   └── quality_rules.yaml             # Quality rule definitions
├── notebooks/
│   ├── run_pipeline.py                # Orchestration entry point
│   └── backfill.py                    # Date-range backfill utility
├── tests/
│   ├── conftest.py                    # Spark fixtures & sample data
│   ├── test_etl_base.py              # Core framework & utility tests
│   └── test_extractors.py            # Extractor unit tests

*... continues with setup instructions, usage examples, and more.*

Code Sample .py preview

src/bronze_loader.py """ Bronze Layer Loader — Spark ETL Framework ============================================ Ingests raw data into the Bronze (landing) layer of the medallion architecture. Automatically infers schema, appends audit metadata columns, and writes to Delta format. Metadata columns added: _source_file — originating file path _ingested_at — UTC ingestion timestamp _batch_id — unique batch identifier By Datanest Digital — https://datanest.dev """ from __future__ import annotations from datetime import datetime, timezone from typing import Any, Dict, List, Optional from pyspark.sql import DataFrame, SparkSession from pyspark.sql import functions as F from pyspark.sql.types import StringType, StructType, TimestampType from src.etl_base import ETLBase class BronzeLoader(ETLBase): """ Load raw files into the Bronze Delta table with metadata enrichment. Supports JSON, CSV, Parquet, and Avro source formats. Schema can be explicitly provided via config or auto-inferred from the source data. Config keys (under ``source``): format: str — file format (json | csv | parquet | avro) path: str — source directory path options: dict — extra reader options (e.g. ``{"header": "true"}``) schema: dict — optional explicit StructType definition Config keys (under ``destination``): database: str — target database / catalog table: str — target table name path: str — Delta table storage path (optional) """ def __init__( self, spark: SparkSession, pipeline_name: str, # ... 127 more lines ...