Data Quality Monitoring Framework
Cross-cutting DQ layer for any pipeline: freshness, completeness, schema drift, null rates, and row-count anomalies with a standardized summary per schema.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
⚙ Try Demo ToolsFile Structure 17 files
Documentation Preview README excerpt
Data Quality Monitoring Framework
A cross-cutting quality layer you can attach to any pipeline. Tracks
freshness, completeness/documentation, and row-count anomalies, and produces a
standardized summary per schema — built on system.information_schema, so it
works on any Unity Catalog workspace with no per-table wiring.
Fully generic and environment-agnostic: point it at a list of catalogs and
deploy.
What's inside
- Databricks Asset Bundle — a daily scan job (freshness, completeness,
row-count history → summary).
- Metadata-driven scans — no per-table SQL to maintain; monitored catalogs
are a single variable.
- A dependency-free checks library (
lib/dq_checks.py): freshness/
completeness classification, null rates, z-score row-count anomalies, and a
0–100 quality score — with an offline test suite (7 tests).
Quickstart
pip install pytest && pytest tests/ -v # validate check logic offline
databricks bundle deploy -t dev -p <profile>
Set target_catalog, target_schema, monitored_catalogs, warehouse_id.
License
MIT — see LICENSE.