Ingestion Latency Framework
End-to-end ingestion latency measurement and alerting across source-to-gold hops.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
⚙ Try Demo ToolsFile Structure 16 files
Documentation Preview README excerpt
Ingestion Latency Monitoring Framework
Measures end-to-end data ingestion latency across medallion layers:
batch-to-batch lag ("how often is new data arriving?") and source-to-landing
time ("how fresh is it vs the source?"), with p50/p95/p99 summaries and SLA
status. Works for any streaming or micro-batch pipeline.
Fully generic and environment-agnostic — point it at your bronze/silver
catalogs and monitored tables.
What's inside
- Databricks Asset Bundle — a 2-hourly measure + summarize job.
- Two latency measures — batch-to-batch lag and end-to-end (source→landing).
- Percentile summaries — p50/p95/p99 per table/layer/day.
- A dependency-free stats library (
lib/latency_stats.py): batch lags,
end-to-end latency, percentiles, summaries, and SLA status — with an offline
test suite (5 tests).
Quickstart
pip install pytest && pytest tests/ -v
databricks bundle deploy -t dev -p <profile>
Set the target/bronze/silver catalogs and edit the table references in the SQL
templates to your monitored tables.
License
MIT — see LICENSE.