Stop using CSVs for time-series market data

The CSV is the cockroach of data science: it survives everything, opens in Notepad, and is universally understood. But if you are building models on multi-year, high-frequency energy or financial data, leaning on CSVs is an active drag on your productivity. Here is why time-series data belongs in Apache Parquet.

1. Type inference is a silent killer

A CSV is just text with commas. When you call pd.read_csv(), pandas scans the file and *guesses* the types. Is 2025-01-01 12:00:00 a string or a datetime? What happens when an API returns "N/A" in a float column halfway through ten million rows? Your column silently coerces to object, memory balloons, and your vectorised maths breaks downstream in ways that do not raise an error — they just produce wrong numbers.

Parquet is strongly typed. The schema is baked into the file's metadata. A float stays a float; a UTC timestamp stays a UTC timestamp. No guessing, no silent coercion, no "why is this column an object" at 2am.

2. Row-based vs columnar

CSVs are written row by row. If a dataset has fifty columns but you only need to plot wind generation against price, a CSV parser still has to read the entire multi-gigabyte file into memory to get at those two columns.

Parquet is columnar. Query two columns and the engine — pandas, Polars or DuckDB — reads only the bytes on disk for those two columns. A scan that takes tens of seconds on a CSV drops to a fraction of a second on Parquet, because you are no longer paying to read the 48 columns you did not ask for.

3. Compression and partitioning

Because Parquet groups like-typed values together, it compresses extremely well: a couple of gigabytes of day-ahead power prices as CSV shrink to a fraction of that as Parquet, with zero loss of fidelity. And Parquet supports Hive-style partitioning — laying data out in folders like zone=FR/year=2024/ — so an engine like DuckDB can push filters down and run SQL over a folder of files without ever loading the whole thing into RAM.

The ReadySet standard

We process hundreds of millions of rows of messy public XML and JSON. We never hand it back as a raw CSV. Every dataset in the catalog is quality-checked, normalised to one documented grain, strongly typed, and shipped as analysis-ready Parquet — flat where that is simplest, partitioned by zone and year where that helps. Each ships with a data dictionary, a QC report and a free sample so you can confirm the schema before you buy. (Prefer CSV for a quick look? Every sample is available that way too — just not for the multi-million-row full sets, where it is the wrong tool.)

> Query gigabytes in milliseconds. Browse the catalog of > pre-cleaned Parquet datasets, or get All-Access for unlimited > downloads of the whole thing.