Persist a resident server's index inside the snapshot, and memory-map it instead of rebuilding it on every load — a serving layer that could sit above any open table format.
Modern table formats — Iceberg, Delta, Hudi — solve storage management. They tell you which files make up a consistent table snapshot, and add atomic commits, time-travel, and schema evolution on top. They're excellent. And they deliberately stop short of one thing: telling a serving engine how to expose that snapshot with microsecond-latency lookups. That isn't a gap in Iceberg — it's a missing layer above it.
I hit this building EnergyScope — a Bloomberg-style data service for energy markets. An analyst types =ES.Get("WTI", …) in Excel and expects the answer now. Under the hood, an Arrow Flight server holds ~2.8 million time series (136 million rows, ~49 GB) resident in RAM, indexed so that "one series, a date range" is an O(1) lookup plus a slice, not a scan. That index — series ID to a contiguous row range, plus the latest value per series, plus date bounds — is the piece that matters here. Call it the serving index: the in-memory structure that turns "find this series" into a lookup instead of a scan. Iceberg gives you the files; something still has to build that index. And every server I know rebuilds it from scratch.
Rebuilding means: read the Parquet, sort and group with Polars, dictionary-encode the strings, hold it all resident. Seconds of work, gigabytes of RAM, every time the process comes up — and partially repeated on every dataset reload, which is where a worse problem lives: because there's no atomic handoff between "new files on disk" and "new index in RAM," a reload briefly takes the dataset offline.
The index is the expensive, volatile derivative the server keeps recomputing. So the idea is almost embarrassingly simple: what if the snapshot shipped its own index, and the server just mapped it?
Before writing a line, I found I'd half-built this already, without naming it. One hot-path optimization in the server keeps, next to each dataset, a flat sorted recent.parquet and a recent_index.json — a map from series ID to [start_row, end_row]. At query time it reads the index and does table.slice(offset, length): a zero-copy view, not a scan. My own comment recorded the payoff — slice() at ~0 ms where the equivalent gather cost ~600 ms.
That's the whole idea in miniature: data file, a persisted index beside it, an O(1) slice to serve. The plan wrote itself — generalize it from "the recent rows" to "the whole dataset," make it a proper snapshot, and make it the primary load path instead of a side cache. A snapshot becomes a pair: an immutable, sorted data.arrow (or Vortex) file, and a compact binary index.arrow sidecar that is the index. Loading a dataset stops being "read and rebuild" and becomes "map and read the index."
Then the important part: instead of believing any of that, measure it. Four experiments, on real data, production untouched.
The obvious objection: the OS page cache already keeps hot pages warm, so is a memory-mapped file really better than just holding the Arrow table resident? I built the PET dataset (13.8M rows, 175K series) as one sorted Arrow-IPC file plus a 6.7 MB binary sidecar, and loaded it two ways — a plain resident read versus a memory-map — then touched every series.
The surprise is in that "after scanning every series." I expected mmap to win only on the cold tail — series never queried, whose pages never fault in. But it stayed at 171 MB even after I touched all of them. Why? Because I'd only read the value column, and mmap faults in pages, not files. Arrow stores each column in its own region of the file, so touching one column faults in only the pages backing that column; the string columns I never read never became resident. The win isn't "cold series don't cost you" — it's the stronger "columns you don't read don't cost you either." The resident table holds all four columns whether you want them or not. Load time dropped too: 0.25 s mapped versus 0.55 s resident, and versus ~1.5 s for the old read-and-reindex path.
The reload outage was the failure I most wanted gone, and the fix is almost trivial: never take the dataset offline. Build the new snapshot fully off to the side, then swap a single pointer — the server serves the old one until the instant the new one is ready. Compare that to the old sequence, unload → sleep → load, where the dataset is None for the whole window and every query fails. To measure the difference I stood up a minimal Flight server, pointed eight concurrent workers at it, and fired a reload mid-flight, each way.
Old: a 1.2-second hard outage, and because rejected requests return instantly, a busy client hammers out tens of thousands of failures in that window — every one of them an #N/A in someone's spreadsheet. New: zero failures. The only trace of the swap is a single 169 ms latency ripple as the fresh file's pages fault in — no query ever fails. Steady-state latency is identical, so the swap itself costs nothing.
An immutable, mapped file can't take an insert — that's the catch. Live data has to land somewhere. The answer turns out to be a shape anyone who's read about Hudi or an LSM tree will recognize: the mapped base stays immutable; writes append to a small in-RAM delta; a read merges the base slice with the delta, the delta shadowing the base by key (so a revision overwrites rather than duplicates); and periodically you compact — fold the delta into a fresh base and atomic-swap it in.
New periods appear; revisions correctly shadow the base with the row count stable; the value survives compaction. And the merge cost is paid only by series that actually have a pending update — everything else takes the 0.012 ms fast path. Compaction is a background job, off the serving path. It works, and it's cheap. (One bug I hit was instructive: the shadow key has to be the normalized period, not the raw one — a small correctness contract that only surfaces when you build the thing.)
Real data has a hot core and a long tail. The core — the daily prices a desk actually trades — wants true zero-copy: Arrow IPC. The tail — millions of rarely-touched series — wants to be small on disk and cheap to decode on demand: Vortex, a newer columnar format built for exactly that. So: two tiers under one sidecar. I built the electricity dataset (59.6M rows, 765K series) as Vortex and ran a query spanning both tiers.
The tail dataset is seven times smaller on disk than Arrow IPC, ranged slices are cheap through the same sidecar, and a single query gathers correctly across the mapped hot tier and the Vortex tail. Vortex is ~3× slower per series than a zero-copy map — the decode cost — which is precisely why the hot core stays on Arrow IPC and only the tail goes to Vortex. The sidecar routes uniformly; a query doesn't care which tier a series lives in.
Here's where I keep myself honest, because it's what makes the rest believable. None of the pieces are new. Memory-mapping a file, a sidecar index, merge-on-read compaction — these are old, well-understood ideas. DuckDB stores persistent metadata alongside its data. Lance does something similar. Merge-on-read is decades old. If someone reads this and thinks "that's just mmap plus an index file," they are not wrong about the parts.
The contribution isn't a new mechanism. It's a specific bridge, in a spot where two worlds rarely meet.
Query engines read open table formats but hold no resident index — they scan and plan per query. Serving systems hold resident indexes but over proprietary storage you can't also query with DuckDB. Time-series databases have brilliant indexes but they are databases — which, for a system whose founding thesis was "the database was the bottleneck, skip it," is the wrong answer. The unclaimed seam is the one in between: an open, memory-mappable, time-series-sorted snapshot whose sidecar is precisely a resident server's serving index, with a base-plus-delta merge for liveness. It's unclaimed because few systems are both a lakehouse store and a Bloomberg-style resident server. This one happens to be both.
And it's a prototype. "Validated end to end" means the design is sound and every claim is measured — not that it's in production. The load path, the reload swap, the liveness, the tiering all work in isolation; wiring them into the live server is a separate, careful step I haven't taken. The honest scorecard: four experiments, four green, one deploy still ahead.
Because the two answer different questions. An Iceberg snapshot answers "which files belong to this table, right now?" — the storage question. This answers the next one: "how does a resident server expose those files without rebuilding its in-memory state?" — the serving question. They're complementary, not competing.
In fact the clean version of this idea sits directly on top of Iceberg. Let Iceberg own the snapshot, its atomicity, and its history — then persist the serving index as one more artifact beside the data files inside that same snapshot. The server memory-maps the data and the index straight out of the Iceberg snapshot, instead of rebuilding its in-memory state on every load. Iceberg, Delta, or Hudi define the storage snapshot; this is a serving layer that could sit above any of them. It isn't an alternative to the table format — it's the tier the table format leaves unspecified.
Two things, and neither is about mmap.
The first is the number that surprised me — mmap faults pages, not files — which is worth internalizing beyond this project. When you memory-map a columnar file and read one column, the others cost you nothing. For a wide table served narrowly, that's a real, free win most people leave on the table by defaulting to a resident read.
The second is the method, which mattered more than any result. The whole thing is a design that could have stayed a satisfying architecture argument forever. Instead: build the smallest real version, measure the three numbers that decide it, and let the table tell you whether it's a project or a footnote. Every experiment here was scoped to touch nothing in production and to end in a number, not an opinion. The design flagged one risk as central — "does mmap actually pay, or is the value just the prebuilt index?" — and a two-minute A/B answered it more sharply than any amount of arguing would have.
One last framing, because it's the reason this reaches past my own server. Iceberg snapshots are designed to be portable between query engines. A serving snapshot extends that portability to resident serving engines — the same open files, now also mappable straight into a live server's memory.
The seam is still open, if you want it. Ship the index; don't rebuild it.