← Back to all products

Market Data Ingestion Accelerator

$1490

Generic balancing-energy / market-data ingestion pipeline for open market APIs, environment-agnostic and configurable by endpoint and series.

18 filesv1.0.0
PythonMarkdownSQLYAMLJSONDatabricks

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

⚙ Try Demo Tools

File Structure 18 files

market-data-ingestion-accelerator/ ├── LICENSE ├── README.md ├── conftest.py ├── databricks.yml ├── guide/ │ ├── 01_what-you-get.md │ ├── 02_getting-started.md │ ├── 03_architecture.md │ └── 04_support.md ├── guides/ │ └── market-ingestion-methodology.md ├── lib/ │ └── market_api.py ├── resources/ │ └── jobs.yml ├── src/ │ ├── ingest/ │ │ ├── api_client.py │ │ └── realtime_ingest.py │ ├── quality/ │ │ └── dq_checks.sql │ └── transform/ │ ├── bronze_to_silver.py │ └── silver_to_gold.py └── tests/ └── test_market_api.py

Documentation Preview README excerpt

Market Data Ingestion Accelerator

A generic, deploy-ready ingestion pipeline for balancing-energy / market-data

open APIs (ENTSO-E Transparency Platform-style): bronze → silver → gold with

XML time-series flattening, revision de-duplication, and data-quality checks.

Fully generic and environment-agnostic: configure your bronze/silver/gold

catalogs, bidding-zone EIC codes, and API token (via a Databricks **secret

scope** — never hard-coded). No organization-specific dependencies.

What's inside

  • Databricks Asset Bundle — a 30-minute realtime ingest → transform → DQ job.
  • A thin REST API client (src/ingest/api_client.py) with request building

and secret-scoped auth.

  • Bronze→silver→gold transforms — XML parse, point flattening, enrich, dedup.
  • Data-quality SQL — freshness, null, and duplicate assertions per zone.
  • A dependency-free ingestion library (lib/market_api.py): query-param

building, ISO-8601 resolution mapping, Period/Point flattening, exponential

backoff, and dedup — with an offline test suite (5 tests).

Quickstart


pip install pytest && pytest tests/ -v      # validate parsing/dedup offline
databricks bundle deploy -t dev -p <profile>

Store your API token in a secret scope, set bidding_zones (EIC codes), and the

bronze/silver/gold catalogs.

License

MIT — see LICENSE.

Code Sample .py preview

conftest.py import os,sys sys.path.insert(0, os.path.dirname(__file__))