Why loading ENTSO-E data is harder than it looks

The ENTSO-E Transparency Platform is one of the best open-data resources in energy. It is also one of the most painful to load correctly. Teams routinely budget an afternoon to "just pull the prices" and lose a week. Here is where the time actually goes.

1. The XML is nested, and the schema shifts

ENTSO-E does not hand you a tidy table. You get deeply nested XML — TimeSeries inside Period inside Point, with the resolution declared once at the period level and the values implied by position, not stamped on each row. A single response can mix PT15M, PT30M and PT60M resolutions across neighbouring periods. If you flatten naively you silently misalign every fifteen-minute series against every hourly one.

2. Timezones are a trap, twice

Delivery hours are published in CET/CEST, but the API speaks UTC offsets that change across the year. The classic bug: you parse the period start as a local timestamp, forget that the clocks moved on the last Sunday of March, and now your spring DST day has 23 rows where it should have 24 — or 25 in October. The only safe move is to anchor everything to UTC on ingest and treat local wall-clock as a presentation concern, never a join key.

3. The B-codes

Generation and load come tagged with PSR B-codes — B01 is biomass, B16 is solar, B19 is wind offshore, and so on. There are dozens, the list evolves, and the platform happily returns codes that are not in last year's reference table. You cannot resolve fuels by guessing; you need the current mapping and a strategy for codes you have never seen.

4. The gaps you do not notice

This is the expensive one. Missing data in ENTSO-E is mostly absence, not nulls — the row simply is not there. A zone goes dark for a maintenance window, a TSO is late publishing, an outage suppresses a feed. If your loader assumes contiguous timestamps you will compute a daily average over a day that only had eighteen hours and never know.

• Reindex every series onto its native cadence before you aggregate.
• Distinguish a real zero (a negative-price hour clearing at €0) from a gap.
• Keep a per-zone, per-day completeness count so anomalies surface.

What "done" looks like

A clean ENTSO-E table is: one row per zone per timestamp, UTC, deduplicated, gap-aware, with fuels resolved against the current B-code map. Getting there is not hard conceptually — it is a hundred small correctness decisions, each of which fails quietly if you get it wrong.

That is exactly the work we do once, so you do not have to. Our European Day-Ahead Prices dataset is the output of this pipeline: flat Parquet, UTC, deduplicated, gaps documented in the QC report. You join it; you do not parse XML.