Vortex vs Arrow vs Parquet on 10 years of energy data
The columnar storage fork — fast-but-huge (Arrow) vs small-but-slow-to-decode (Parquet) — and whether Vortex really gives you both. Measured on a real energy-markets dataset, including the Vortex-vs-Arrow-IPC comparison that isn't published anywhere.
LinkedIn teaser (paste-ready):
Same query — one series out of 60M rows of real energy-market data. Read from a Vortex file on local disk, it beat Parquet 5.6×. Read from object storage, Parquet beat Vortex 4.3×. Same data, same query — the ranking inverts the moment storage moves off the box.
The physics: Vortex latency ≈ (number of requests) × (cost per request). Its fine-grained layout issues many small reads — near-free as an mmap, expensive as network round-trips; Parquet's coarse row-groups issue few large ones and barely notice. And this doesn't contradict SpiralDB's "137× faster random access" headline — that's a local NVMe number; their object-storage wins are high-concurrency scan throughput, the opposite of one-user-one-query.
Along the way I had to correct my own conclusion. I thought I'd shown disk-Vortex beating in-RAM Arrow — until I realised my RAM baseline was a naïve scan, not the indexed O(1) lookup a real server actually uses. Fixed it and reran: indexed RAM wins point queries outright (sub-ms); Vortex's real edge is near-RAM latency at ~10× less memory.
The real takeaway isn't "which format is fastest?" — it's that resident memory, local NVMe, and object storage are three different optimisation problems, and the right format depends on the layer you're building. Full numbers + honest caveats (pre-1.0, no .NET binding) 👇
Years ago I reverse-engineered the QVD file format — QlikView's binary store — for fun, because I wanted to understand why QlikView loaded data so fast. The answer was dictionary encoding, bit-packed index tokens, and querying the encoded columns directly; those ideas stuck with me. The open ecosystem now has Arrow, Parquet, Iceberg — and Vortex. I wanted to know whether Vortex finally delivers the thing QVD hinted at: compact storage without giving up serving speed. So I measured it, on real energy time-series rather than synthetic TPC-H.
The problem
If you serve columnar data to analysts, you live on a fork:
- Arrow IPC — the in-memory layout is the wire layout, so reads are zero-copy and instant. But it's uncompressed by design: the files are big.
- Parquet — dictionary + run-length + bit-packing + compression make it tiny. But to use it you decode all of that back into memory. Read costs CPU.
You end up pairing them: Parquet at rest, Arrow in flight. Vortex (SpiralDB → LF AI & Data, incubation) claims to collapse the fork — compressed and zero-copy and fast random access, via cascading FastLanes / ALP / FSST encodings with FlatBuffer O(1) column access. I wanted real numbers on my own data, not TPC-H.
Setup
Two slices of EnergyScope's energy-market time-series (schema series_id | period | value): a 13.8M-row petroleum set and a 59.6M-row electricity set. Each written as Arrow IPC (raw + zstd), Parquet (zstd) and Vortex 0.75.0, then read back to a usable Arrow table (materialised, not just "file opened"), with a column-projection pass. Median of N runs, warm cache, on a Minisforum MS-02 (Core Ultra 5, NVMe).
Storage — 59.6M rows (electricity)
| Format | Size (MB) | Write (ms) | Cold read (ms) | Project (ms) |
|---|
| arrow_ipc_raw | 3245 | 1524 | 1550 | 1006 |
| arrow_ipc_zstd | 563 | 1827 | 1448 | 1266 |
| parquet_zstd | 206 | 3261 | 1009 | 1324 |
| vortex | 319 | 7012 | 879 | 623 |
And at 13.8M rows (petroleum) the same shape held: Arrow-raw 587 MB / read 293 ms; Parquet 31.6 MB / read 237 ms; Vortex 60.6 MB / read 206 ms, project 88 ms — again the fastest read and projection.
The finding
Vortex lands in the quadrant the fork says shouldn't exist. On 60M rows it read a 10× smaller file than raw Arrow faster than raw Arrow read itself (879 ms vs 1550 ms), and projected columns 2.1× faster than Parquet (623 ms vs 1324 ms) — at ~1.5× Parquet's size. Its one real cost is write time (7 s — the encoding tax), which is fine for write-once storage.
Why can a smaller-but-encoded file out-read an uncompressed zero-copy one? Because there's 10× less bytes to move and its encodings decode fast (and it can operate on compressed data for many ops). Less I/O + fast decode beats more I/O + zero decode — and the advantage grows with data size. Vortex's founder Will Manning puts numbers on the "decode fast" in his CMU Future Data Systems talk: FastLanes bit-packing decompresses at ~53 GB/s versus Zstandard's 1–2 GB/s. The cascaded lightweight encodings (FastLanes / ALP / FSST) are chosen precisely to avoid the data dependencies that make traditional compression slow to decode.
Notably, every published Vortex benchmark I could find compares it to Parquet. The Vortex-vs-Arrow-IPC comparison — the one that actually matters if your serving path is Arrow — wasn't out there. Now it is.
Execution — DuckDB vs DataFusion
I also swapped the query engine (both reading the source Parquet). On these workloads (scan / point-filter / group-by), DuckDB was consistently ahead — at 60M rows, 7 ms vs 47 ms on a point filter, 70 ms vs 93 ms on a group-by. DataFusion tops complex ClickBench-style suites, but for straightforward analytical queries DuckDB won. The takeaway: DataFusion is an architecture choice (Rust single-binary, Arrow-native, native Vortex integration), not a performance upgrade.
A workload-dependent footnote, since I later ran it: DuckDB reads Vortex natively (a community read_vortex extension). Manning reports that on wide ClickBench-style scans on NVMe, DuckDB reading Vortex beats DuckDB's own native storage format — but on my narrow point/aggregate queries over 3-column series data, DuckDB native won by ~6×. Storage verdicts are workload-shaped: his wide analytical scan is Vortex's sweet spot; my narrow point query is native's.
End-to-end: disk → query → Arrow → Flight → client
Isolated storage numbers are one thing; what a platform actually cares about is the latency the client sees. So I put each format behind an Arrow Flight server and measured time-to-first-batch and total — for a full scan and a selective one-series query — comparing reading storage per query (Parquet / Vortex) against holding the data in RAM as Arrow. Crucially I ran the in-RAM baseline two ways: a naïve linear filter (arrow_mem), and the way a real server does it (arrow_idx) — a startup series index (series_id → a pre-sorted contiguous row range) so a lookup is O(1) + a take(), never a scan. That's exactly how EnergyScope serves, and it's the honest baseline to beat.
Selective query — one series (the pattern that dominates real analytics):
| Source | 13.8M rows (ms) | 60M rows (ms) |
|---|
| arrow_idx — indexed in-RAM (real server) | 0.6 | 0.6 |
| vortex — read per query | 9.2 | 14.1 |
| arrow_mem — naïve RAM scan | 13.0 | 42.9 |
| parquet — read per query | 121.4 | 222.6 |
The honest result — and the correction to every "disk beats RAM" hot take, a draft of my own included: a properly indexed in-RAM server wins selective queries outright, ~0.6 ms flat at both sizes, because an O(1) index lookup into RAM can't be beaten. The striking part is second place: Vortex serves the same one-series query in 9–14 ms reading from disk — ~15–23× behind indexed RAM, but from a file ~10× smaller than the resident Arrow table, while beating the naïve scan ~3× and Parquet ~15×. Note the scaling: indexed RAM and Vortex both stay flat as data grows (0.6→0.6; 9→14), while the naïve scan blows out (13→43). The lesson was never "disk beats RAM" — it's that an index beats a scan, the same thing QVD's symbol tables and bit-packed tokens taught, whether the data lives in RAM or in a well-encoded file.
Selective query — "give me one series" (60M rows)
arrow_mem (naïve scan) arrow_idx (indexed RAM) vortex (on disk, 10× smaller)
scan 60M rows O(1) lookup + take read metadata, prune to 1 series
| | |
43 ms 0.6 ms 14 ms
fastest — but 3.2 GB near-interactive, from a
resident in RAM 319 MB file
For a full scan the economy tips back to Vortex: reading + decoding a 319 MB Vortex file and streaming it delivers the whole result to the client faster than streaming the 3.2 GB resident Arrow table — 936 ms vs 1634 ms total at 60M rows — and first-batch in 307 ms vs Parquet's 859 ms. (In-RAM still yields the very first batch soonest, 37 ms, since it's already materialised — but it's slowest to finish.) So Vortex is the fastest full-scan source to complete, and the lightest. Net for a serving layer: indexed RAM if you need the last microsecond on point queries; Vortex for near-interactive selective latency, the fastest complete scans, and ~10× less memory — which is what matters once the data outgrows RAM or you run many datasets at once.
Loopback gRPC (no network — real network adds a fixed latency equally to all sources); warm cache; median of 5. The indexed-RAM baseline replicates EnergyScope's startup series index (series_id → contiguous, pre-sorted row range) so take() is a slice, not a gather — without it, a scattered take() loses even to a full scan.
Over object storage — where the ranking inverts
The numbers so far assume storage sits next to compute. But a cold tier lives on object storage, and that changes the physics: every read becomes an HTTP range request with real latency. I put both formats behind MinIO (S3-compatible) and ran the same one-series query three ways — as a local file, via MinIO on the same box (loopback, no network), and via MinIO on another machine over a 2.5 GbE LAN:
| One-series query | Parquet | Vortex |
|---|
| Local filesystem file | 26 ms | 4.7 ms |
| MinIO on same box (loopback S3) | 26.7 ms | 23.1 ms |
| MinIO over LAN (~2 ms / request) | 41 ms | 178 ms |
The local win inverts over the network: Vortex goes from 5.6× faster (local file) to 4.3× slower (LAN S3). The cause is request granularity — Vortex latency ≈ (number of requests) × (cost per request). Its fine-grained layout issues many small reads — superb when a read is a near-free mmap, costly when each is a network round-trip; Parquet's coarse row-groups issue few large ones and barely notice the network. Even loopback object storage costs Vortex ~5× over a plain file — the S3/HTTP protocol tax, before any network at all. Concurrency does scale for both (object storage's real gift), but Parquet peaks higher against a single endpoint.
Does this contradict SpiralDB? No — it locates the boundary. Manning's headline 137× faster random access than Parquet is, in his own words, "Zstandard Parquet on NVMe, six arbitrary rows" — a local number that matches our local result exactly. His object-storage wins are scan throughput under massive connection concurrency (his target is streaming to GPUs — "~500 MB/s per connection, terabytes/sec by having boatloads of connections"), not single-query latency against one endpoint — and that throughput comes from Vortex's Rust scan operator, which the PyArrow-dataset path I used doesn't fully exercise. So the honest placement — for the workloads measured here: Vortex is best suited to local or attached storage; over a remote cold tier its many-small-requests model loses to Parquet, and a real cloud/WAN endpoint (higher per-request latency) would only widen that gap.
It's worth understanding why Vortex makes these trade-offs, because it explains our result. Manning's design premise is a hardware inversion: on a modern GPU box (he cites AWS P5), the network card has ~8× the bandwidth of the PCIe bus to the GPU — up to an order of magnitude. So the fastest way to feed a GPU is no longer "load from local disk over PCIe" — it's to stream compressed bytes straight from object storage over the NIC to the device, bypassing the CPU and PCIe entirely, and decompress on the GPU. That world rewards exactly what Vortex optimises for: object-storage-native layout, throughput via many concurrent connections (~500 MB/s each, terabytes/sec in aggregate), and lightweight, data-dependency-free encodings that decode on GPU SIMT. It's a throughput-at-massive-concurrency design — which is precisely why a single-user, single-query, latency-sensitive workload like ours (one analyst, one series, one endpoint) is the one operating point where its advantages don't show up.
The honest caveats
- Pre-1.0. Vortex is LF AI & Data incubation stage, ~weekly releases. The file format is stable/backwards-compatible since 0.36.0; the APIs are not frozen.
- No .NET / C# binding. Bindings are Rust, Python, Java, C/C++ (FFI). If your client is .NET (mine is an ExcelDNA add-in), Vortex stays server-side — you serve Arrow to the client and keep Vortex behind the transport.
- Write cost & size are data-dependent. On my narrow, string-heavy, highly-repetitive series data I measured Vortex ~3× slower to write and ~2× larger than Parquet — but that's near-worst-case for it (Parquet's dictionary + Zstd love exactly this data). Manning reports the opposite geometric mean across wider datasets: Vortex ~5× faster to write, ~1.16× the size. Benchmark on your data — either way it's a storage, not a serving, concern. Follow-up measured: the size gap was partly our own schema's fault — after a schema fix (date32 instead of date strings + sort by series) it narrows to 1.4–1.8× while Vortex reads get ~2× faster. See the companion study, Floats that are secretly integers — ALP + a schema fix.
What I'd actually do
Place Vortex where storage sits next to compute — a local / attached-disk scan layer behind Arrow Flight, co-located on the serving box (an on-site mini-server, say), where its random-access and decode speed shine and the client keeps consuming Arrow unchanged. Keep Parquet for the object-storage cold tier — it's smaller and it beats Vortex over the network — and for portability. The composable stack — Iceberg → (Parquet on object storage / Vortex on local disk) → DuckDB / DataFusion → Arrow Flight → client — is real and each piece works; it's a staged adoption, not a rewrite. (Engine is an open choice: DuckDB won these queries; DataFusion earns its place only if you want the Rust single-binary for client-site deployment.)
The real takeaway
The biggest lesson wasn't that Vortex replaces Parquet — it doesn't. It's that the right format depends on where the data lives: resident memory, local NVMe, and object storage are three different optimisation problems, and a format that wins one can lose another. So the question worth asking isn't "which format is fastest?" but "which format fits this layer of the architecture?" — which is why this piece runs three separate tests rather than one: how fast is the format, how fast is the end-to-end serving path, and how does the answer change when storage moves from local NVMe to object storage. Answer all three before you commit a platform to a format.
Method + harness: a small Python script (pyarrow / duckdb / datafusion / vortex-data) that writes each format, times cold-read-to-Arrow and projection, and runs the engine queries. Reproducible on your own Parquet.