EnergyScope · Engineering Blog

Notes from building a resident Arrow Flight serving engine

Storage formats, serving architecture, and experiments that end in a number, not an opinion.


July 2026 · ~8 min

Iceberg stops at the files. Serving starts at the index.

Table formats say which files are the table; nothing says how a resident server should expose them without rebuilding its in-memory state. A proposal for that missing layer — snapshots that ship their own serving index — validated in four measured experiments: mmap faults pages not files, a reload gap of 30,825 failed queries goes to zero, live updates merge in 0.57 ms, and a Vortex tail tier cuts 59.6M rows to 374 MB.

Read the post →

Technical companions