← Back to all products

Data Quality Monitoring Framework

$1490

Cross-cutting DQ layer for any pipeline: freshness, completeness, schema drift, null rates, and row-count anomalies with a standardized summary per schema.

17 filesv1.0.0
PythonMarkdownSQLYAMLJSONDatabricks

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

⚙ Try Demo Tools

File Structure 17 files

gf-data-quality-framework/ ├── LICENSE ├── README.md ├── conftest.py ├── databricks.yml ├── guide/ │ ├── 01_what-you-get.md │ ├── 02_getting-started.md │ ├── 03_architecture.md │ └── 04_support.md ├── guides/ │ └── data-quality-methodology.md ├── lib/ │ └── dq_checks.py ├── resources/ │ └── jobs.yml ├── src/ │ ├── 01_freshness_scan.sql │ ├── 02_completeness_scan.sql │ ├── 03_row_count_anomalies.sql │ └── 04_quality_summary.sql └── tests/ └── test_dq_checks.py

Documentation Preview README excerpt

Data Quality Monitoring Framework

A cross-cutting quality layer you can attach to any pipeline. Tracks

freshness, completeness/documentation, and row-count anomalies, and produces a

standardized summary per schema — built on system.information_schema, so it

works on any Unity Catalog workspace with no per-table wiring.

Fully generic and environment-agnostic: point it at a list of catalogs and

deploy.

What's inside

  • Databricks Asset Bundle — a daily scan job (freshness, completeness,

row-count history → summary).

  • Metadata-driven scans — no per-table SQL to maintain; monitored catalogs

are a single variable.

  • A dependency-free checks library (lib/dq_checks.py): freshness/

completeness classification, null rates, z-score row-count anomalies, and a

0–100 quality score — with an offline test suite (7 tests).

Quickstart


pip install pytest && pytest tests/ -v      # validate check logic offline
databricks bundle deploy -t dev -p <profile>

Set target_catalog, target_schema, monitored_catalogs, warehouse_id.

License

MIT — see LICENSE.

Code Sample .py preview

conftest.py import os,sys sys.path.insert(0, os.path.dirname(__file__))