Pipeline Observability Framework
Job/pipeline observability from system tables: run history, failures, durations, and SLA tracking with alerting.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
⚙ Try Demo ToolsFile Structure 17 files
Documentation Preview README excerpt
Pipeline Observability Framework
Job execution monitoring, health scoring, and lineage/impact analysis from
Databricks system.lakeflow tables. Answers the three questions every data team
asks at 3am: what ran, did it succeed, and what depends on it?
Works on any Databricks workspace. Fully parameterized, no org-specific deps.
What's inside
- Databricks Asset Bundle — a 4-hourly refresh (inventory → run history →
health metrics).
- Health scoring — success rate and a Healthy/Degraded/Unhealthy/Critical
classification per job over a 30-day window.
- Lineage + blast radius — a job↔table lineage table and a downstream-impact
query that shows which consumers an unhealthy job affects.
- A dependency-free library (
lib/job_health.py) mirroring the scoring +
impact logic, with an offline test suite (5 tests).
Health classification
| Status | Success rate | Action |
|---|---|---|
| Healthy | ≥ 99% | none |
| Degraded | 95–99% | monitor |
| Unhealthy | 80–95% | investigate |
| Critical | < 80% | fix now |
Quickstart
pip install pytest && pytest tests/ -v
databricks bundle deploy -t dev -p <profile>
Set target_catalog, target_schema, warehouse_id. Populate d_job_lineage
with your job↔table mappings to enable impact analysis.
License
MIT — see LICENSE.