The Serving Snapshot

Where the lake stops, and serving begins

An Iceberg or Delta snapshot answers one question perfectly: which files are the table right now? It says nothing about how to serve that table with sub-millisecond latency — because that was never its job.

To serve, EnergyScope's Flight server takes the files a snapshot names and does substantial work at startup:

1. Read the Parquet year-partitions into memory.
2. Build series_index — series_id → a pre-sorted contiguous row range — with Polars.
3. Build latest_index, the category tree, and precompute period_norm.
4. Dictionary-encode the string columns in RAM.

That indexing is the price of O(1) serving — and it is rebuilt from scratch on every restart measured: the resident footprint is ~49 GB across 2.82M series; the live server reports ~4.3 s of eager index-build timing even with lazy year-partitioning deferring most data (the pre-lazy eager-load figure was ~22 s, itself down from 52 s) — and lazy loading only moves that cost, since the first query to a cold dataset then pays the load tax; the dict-encode step exists because raw string columns dwarf their encoded form (660 MB → 253 MB on the base dataset). Worse, the same work is partially redone on every dataset reload — and because there is no atomic handoff between "new files on disk" and "new index in RAM," a reload briefly unloads the dataset. That unload→load window is the serving gap called out in the code review.

The v0 you already ship

This isn't a leap into the unknown. The recent-tier prewarm in the Flight server is, structurally, exactly this idea — built as a hot-path optimization, never framed as a serving snapshot.

Each dataset already has, on disk, next to its year partitions:

At query time the server reads the index and does table.slice(offset, length) — a zero-copy view, not a scan. Its own comment records the payoff measured: slice() is ~0 ms where take() costs ~600 ms on a 14.8M-row EUROSTAT extract, because slice respects the Arrow chunk structure instead of gathering across it.

The v0 even surfaces a design lesson. The sidecar is 64% the size of the data measured — absurd for an index — purely because it's JSON repeating the long series_id strings 127,520 times. A binary sidecar (an Arrow table of series_id dict-encoded + two int columns) would be a small fraction of that: the string keys, which dominate, dict-encode away. Lesson banked: the sidecar must be columnar/binary, not JSON.

Anatomy of a serving snapshot

A serving snapshot is a pair: an immutable, sorted data file, and a compact sidecar that is the index. A tiny manifest ties them together so the swap is atomic.

Two properties make the index trivially small and the load trivially cheap:

• Sorting collapses the index to arithmetic. Once rows are sorted by (series_id, period), a series isn't "scattered rows to gather" — it's a slice. The index degenerates to series_id → (offset, length), a few bytes per series. This is why the deferred date32 + sort schema migration is a prerequisite, not a parallel nicety: the sort is what makes the serving index a slice table instead of a gather map.
• The index is the derivative, precomputed. Everything the server rebuilds at startup — series_index, latest_index, per-series date bounds — is written into index.arrow at snapshot-build time, once, by whoever cooks the data. The server never recomputes it.

The load path, before & after

The whole value is in one substitution: replace "read files and rebuild the index" with "map files and read the index."

And because the new snapshot is built fully before the pointer flips, the load is atomic: the server serves the old snapshot until the new one is ready, then swaps. The unload→load reload gap is gone — not patched, structurally absent projected.

Serving latency itself doesn't need to change: a query is still index lookup → slice → filter → stream. What changes is that the index arrived ready-made, and (for the mmap'd portion) the data pages fault in from disk on demand rather than all sitting resident. That's the lever for the RAM reduction.

mmap vs decode — the file-format choice

"Memory-map the data" is cleaner in the slogan than in reality. Whether a file is truly zero-copy-mappable, or fast-to-decode-but-not-mapped, splits the design into two tiers.

The honest reading of the Vortex benchmark: its reads are fast (60M-row selective query in 9–14 ms) but the write-up is careful to say "read to a materialised Arrow table" — i.e. a decode, not a raw mmap. So the pure zero-copy-map win belongs to Arrow IPC; Vortex's win is small-file + fast-decode + the index is still prebuilt.

Format placement — the whole map

Four layers, four formats, each chosen where its strengths line up. The manifest records which file a series lives in, so the query path is identical across them:

The dividing line isn't "big vs small" — it's what each layer needs: zero-copy (Arrow IPC), compact-fast-local (Vortex), mutable (in-RAM Arrow), or portable-over-network (Parquet). Three of the four live on the local serving box; only the cold copy is object storage, which is exactly why only it is Parquet.

Live updates & the immutability problem

A memory-mapped file wants to be read-only. do_put wants to mutate. Reconciling the two is the one genuinely hard design question — and its answer is a shape you'll recognize.

An immutable, sorted, mmap'd snapshot cannot accept a live row insert in place — inserting into the middle of a sorted file means rewriting it. So the update path can't touch the base snapshot. The resolution is standard database machinery:

1. The base snapshot is immutable & mmap'd — the bulk of history, never mutated.
2. do_put appends to a small, hot delta buffer — a plain in-RAM Arrow table (or Arrow-IPC append log), unsorted, cheap to grow. This is where today's data lands intra-day.
3. A query reads both — slice the base via the index, concat the matching delta rows, merge. The delta is small, so the merge is cheap. (For revisions, the delta row shadows the base row by key — merge-on-read.)
4. Periodically, compact — fold the delta into a new base serving snapshot, rebuild index.arrow, atomic-swap, discard the delta. Nightly, or on a size threshold.

Why the session's threads converge here

The serving snapshot isn't a new fifth idea bolted on. It's the artifact that makes the other four earn their keep at once.

Read that list against the wishlist in the Iceberg doc: items 6 (pluggable hot/cold file encodings under one table), 7 (time-series-native primitives — the sort key, ALP, the composite key), and 8 (the serving-tier handoff) are not three separate asks — they're one artifact seen from three angles. The serving snapshot is the thing that, if built, closes all three. That is why this is the wishlist item worth prototyping: it's the keystone, and the other pieces already exist to support it.

Prior art — is this actually new?

The components are all off-the-shelf. The specific combination — a persisted serving index that a resident Flight server memory-maps from an open lake snapshot — appears to be the unclaimed seam. Here's the honest survey.

• Query engines (DuckDB, Trino, DataFusion) read Iceberg/Parquet and plan per query — but they hold no persistent resident index; they scan-and-plan each time. They optimize the scan, not a warm serving structure. Different problem.
• Feature stores & serving DBs (Feast, Redis-backed stores, Bloomberg-style services) hold resident indices for low-latency serving — but over proprietary storage, not an open table format you can also query with DuckDB. They don't compose with the lake.
• Time-series databases (ClickHouse, InfluxDB, kdb+) have brilliant resident indices — but they are databases, which the founding thesis rejects, and they don't sit on open Parquet/Vortex snapshots.
• Arrow Flight is the transport, not a storage/index spec — it says nothing about how the server holds data resident.
• The closest real thing is probably Lance (a columnar format with a built-in vector/scalar index) and DuckDB's own storage format — both bundle an index with data, but neither targets "mmap a snapshot into a resident Flight server's O(1) series index," and neither is built around time-series slice semantics.

The staged plan

Each stage is independently measurable and independently useful — you can stop at any rung and keep the gain. Measure before proceeding.

1. Already shipped — quantify it. The recent tier is v0. Measure load-by-index (slice) vs load-by-reindex head-to-head on one dataset to bank the premise as a hard number.deliverable: a before/after table · effort: hours
2. Full-dataset serving snapshot, Arrow-IPC. Take PET, write it sorted as one Arrow-IPC file + a binary index.arrow sidecar. Add an mmap+sidecar load path to the Flight server behind a flag. Measure startup, RAM, and selective-query latency vs today.validates: the whole premise · effort: a weekend
3. Atomic swap via a lite manifest. Wrap the snapshot in a manifest + pointer; make reload build-then-flip. Confirm the serving gap is gone under a concurrent read load.retires: the reload outage
4. Vortex tier + base/delta. Swap the cold tail's data file to Vortex for the RAM win; add the do_put delta buffer + merge-on-read + nightly compaction for liveness.delivers: 10× RAM tail + live updates
5. Iceberg-backed & generalized. Replace the lite manifest with real Iceberg metadata (vintages, rollback for free) and package the sidecar+load-path as a reusable component.outcome: the publishable artifact

Risks & open questions

The section that opened these questions can now close most of them. Status after v1–v3 — resolved, partial, or still open:

New questions the prototype surfaced

What to build first

One weekend experiment decides whether this is a project or a footnote. Here's its exact spec.

The beauty of starting here: it touches nothing in production (flagged, one dataset), reuses the sort the schema migration already wants, and turns the entire question from architecture debate into a table of three numbers. Measure, then build.