← Back to all products

Pipeline Observability Framework

$1490

Job/pipeline observability from system tables: run history, failures, durations, and SLA tracking with alerting.

17 filesv1.0.0
PythonMarkdownSQLYAMLJSONDatabricks

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

⚙ Try Demo Tools

File Structure 17 files

pipeline-observability-framework/ ├── LICENSE ├── README.md ├── conftest.py ├── databricks.yml ├── guide/ │ ├── 01_what-you-get.md │ ├── 02_getting-started.md │ ├── 03_architecture.md │ └── 04_support.md ├── guides/ │ └── observability-methodology.md ├── lib/ │ └── job_health.py ├── resources/ │ └── jobs.yml ├── src/ │ ├── 01_job_inventory.sql │ ├── 02_run_history.sql │ ├── 03_health_metrics.sql │ └── 04_lineage_and_impact.sql └── tests/ └── test_job_health.py

Documentation Preview README excerpt

Pipeline Observability Framework

Job execution monitoring, health scoring, and lineage/impact analysis from

Databricks system.lakeflow tables. Answers the three questions every data team

asks at 3am: what ran, did it succeed, and what depends on it?

Works on any Databricks workspace. Fully parameterized, no org-specific deps.

What's inside

  • Databricks Asset Bundle — a 4-hourly refresh (inventory → run history →

health metrics).

  • Health scoring — success rate and a Healthy/Degraded/Unhealthy/Critical

classification per job over a 30-day window.

  • Lineage + blast radius — a job↔table lineage table and a downstream-impact

query that shows which consumers an unhealthy job affects.

  • A dependency-free library (lib/job_health.py) mirroring the scoring +

impact logic, with an offline test suite (5 tests).

Health classification

StatusSuccess rateAction
Healthy≥ 99%none
Degraded95–99%monitor
Unhealthy80–95%investigate
Critical< 80%fix now

Quickstart


pip install pytest && pytest tests/ -v
databricks bundle deploy -t dev -p <profile>

Set target_catalog, target_schema, warehouse_id. Populate d_job_lineage

with your job↔table mappings to enable impact analysis.

License

MIT — see LICENSE.

Code Sample .py preview

conftest.py import os,sys sys.path.insert(0, os.path.dirname(__file__))