Market Data Ingestion Accelerator
Generic balancing-energy / market-data ingestion pipeline for open market APIs, environment-agnostic and configurable by endpoint and series.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
⚙ Try Demo ToolsFile Structure 18 files
Documentation Preview README excerpt
Market Data Ingestion Accelerator
A generic, deploy-ready ingestion pipeline for balancing-energy / market-data
open APIs (ENTSO-E Transparency Platform-style): bronze → silver → gold with
XML time-series flattening, revision de-duplication, and data-quality checks.
Fully generic and environment-agnostic: configure your bronze/silver/gold
catalogs, bidding-zone EIC codes, and API token (via a Databricks **secret
scope** — never hard-coded). No organization-specific dependencies.
What's inside
- Databricks Asset Bundle — a 30-minute realtime ingest → transform → DQ job.
- A thin REST API client (
src/ingest/api_client.py) with request building
and secret-scoped auth.
- Bronze→silver→gold transforms — XML parse, point flattening, enrich, dedup.
- Data-quality SQL — freshness, null, and duplicate assertions per zone.
- A dependency-free ingestion library (
lib/market_api.py): query-param
building, ISO-8601 resolution mapping, Period/Point flattening, exponential
backoff, and dedup — with an offline test suite (5 tests).
Quickstart
pip install pytest && pytest tests/ -v # validate parsing/dedup offline
databricks bundle deploy -t dev -p <profile>
Store your API token in a secret scope, set bidding_zones (EIC codes), and the
bronze/silver/gold catalogs.
License
MIT — see LICENSE.