← Back to all products

Ingestion Latency Framework

$990

End-to-end ingestion latency measurement and alerting across source-to-gold hops.

16 filesv1.0.0
PythonMarkdownSQLYAMLJSONDatabricks

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

⚙ Try Demo Tools

File Structure 16 files

ingestion-latency-framework/ ├── LICENSE ├── README.md ├── conftest.py ├── databricks.yml ├── guide/ │ ├── 01_what-you-get.md │ ├── 02_getting-started.md │ ├── 03_architecture.md │ └── 04_support.md ├── guides/ │ └── latency-methodology.md ├── lib/ │ └── latency_stats.py ├── resources/ │ └── jobs.yml ├── src/ │ ├── 01_batch_latency.sql │ ├── 02_e2e_latency.sql │ └── 03_latency_summary.sql └── tests/ └── test_latency_stats.py

Documentation Preview README excerpt

Ingestion Latency Monitoring Framework

Measures end-to-end data ingestion latency across medallion layers:

batch-to-batch lag ("how often is new data arriving?") and source-to-landing

time ("how fresh is it vs the source?"), with p50/p95/p99 summaries and SLA

status. Works for any streaming or micro-batch pipeline.

Fully generic and environment-agnostic — point it at your bronze/silver

catalogs and monitored tables.

What's inside

  • Databricks Asset Bundle — a 2-hourly measure + summarize job.
  • Two latency measures — batch-to-batch lag and end-to-end (source→landing).
  • Percentile summaries — p50/p95/p99 per table/layer/day.
  • A dependency-free stats library (lib/latency_stats.py): batch lags,

end-to-end latency, percentiles, summaries, and SLA status — with an offline

test suite (5 tests).

Quickstart


pip install pytest && pytest tests/ -v
databricks bundle deploy -t dev -p <profile>

Set the target/bronze/silver catalogs and edit the table references in the SQL

templates to your monitored tables.

License

MIT — see LICENSE.

Code Sample .py preview

conftest.py import os,sys sys.path.insert(0, os.path.dirname(__file__))