Iceberg turns a directory of Parquet files into a table with ACID commits, time-travel, and safe schema change — no database. But the reason it matters here is narrower and sharper: EnergyScope's Dagster pipeline is a hand-rolled, weaker version of Iceberg's delta engine. This document explains the format, then goes deep on the one mechanism that would replace 200 lines of Python row-diffing with a metadata read.
Apache Iceberg (born at Netflix, now an Apache top-level project) is an open table format — a metadata layer that sits on top of a pile of Parquet files in object storage and makes that pile behave like a real database table.
Before Iceberg, a "table" in a data lake was just a directory of Parquet you agreed to treat as one thing. That convention is brittle: you have to list the directory to know what's in it (slow and, on eventually-consistent object storage, sometimes wrong); you can't write to it atomically (a reader mid-write sees half a table); and you can't roll back a bad load. Iceberg replaces the directory-as-table convention with an explicit, versioned metadata tree — and gets ACID transactions, time-travel, and schema evolution as a result.
Crucially, Iceberg is not a query engine and not a database. It stores no data of its own beyond metadata, and runs no compute. Spark, Trino, DuckDB, Snowflake, Flink and pyiceberg all read and write the same Iceberg table. It is the shared contract, not the processor.
The obvious objection: EnergyScope already has Parquet files in folders — why is a table format better than the filesystem I've got? The answer is one reframing. A filesystem gives you files. It does not give you a table.
Everything Iceberg adds is the gap between "a folder of Parquet" and "a table." And here is the part that lands: everything EnergyScope hand-rolled — the .manifest.json, the prev/ copy, the drop-guard, the Python row-diff — is you building table semantics on a filesystem that doesn't have them. Those aren't features; they're symptoms of the missing layer.
| The gap | Why a filesystem can't | EnergyScope's hand-rolled patch |
|---|---|---|
| Atomic publish | You can atomically replace one file (write-temp-then-rename), never a set of 168 year files. On R2 there's no rename at all — just copy+delete. A crash mid-write leaves a half-table. | drop-guard + prev/ snapshot — patches over "there is no commit" |
| "Which files are the table?" | The table is an implicit convention — "whatever .parquet is under here" — discovered by a slow, eventually-consistent LIST. Two versions can't cleanly coexist in one folder. | the Flight autoload glob, re-deriving the layout at startup |
| What's inside a file | A file is opaque. To know if it holds WTI or covers 2026, you open it. A hash says that it changed, never what's in it. | MD5-per-file manifest — detects change, can't prune |
| History | Overwrite destroys the old bytes. To keep history you copy directories — and you keep one. | a single prev/ copy — "yesterday," nothing older |
| Which rows changed | The filesystem knows a file changed; it has no idea which rows. | the Phase-2 to_pylist() + Python row loop |
| Schema change | Nothing coordinates a column add / type change across 168 files; rename is by name/position, fragile. | none — which is why the date32 migration is a risky coordinated rewrite |
| Self-description | The "table" is tribal knowledge encoded in whatever reads it. | the layout is hardcoded independently in the cook, the autoload, and the delta function |
For what EnergyScope has right now — one writer (Dagster), one box (MS-02), local NVMe — a filesystem of Parquet genuinely works. That's not a mistake; it's the right call at this scale, and it's why it's what runs. The metadata layer earns its keep at specific thresholds: object storage (R2, where multi-file atomicity and listing bite), readers during writes (the reload gap), history / vintages (the product feature), multiple writers, or millions of files (planning cost). EnergyScope is now brushing against three of those — the R2 origin, the reload gap, and the vintage opportunity — which is exactly why Iceberg moved from "no" to "next."
The filesystem stores your bytes; a table format makes those same bytes behave like a database — atomic, versioned, self-describing, queryable through time — without being one. EnergyScope currently supplies a thin, weaker slice of that behaviour in Python, which is fine until the thresholds above, at which point hand-rolling it stops being cheaper than adopting it.
Every write is an atomic snapshot, committed by swapping one metadata pointer. Readers always see a consistent version; query AS OF any past timestamp; roll back a bad write by re-pointing to an old snapshot.
Planning reads metadata manifests — each listing every data file with per-column min/max/null stats — instead of listing the filesystem. Fast on millions of files, correct on object storage where a listing can lie.
Partitions are derived from a column by a transform (days(period)) and tracked in metadata. Users query WHERE period > … and pruning is automatic. The partition scheme can change without rewriting old data.
Columns carry stable IDs, not positions or names — so add / drop / rename / reorder is a metadata-only change that never breaks a reader or rewrites a file.
All four rest on the same foundation: a rich, versioned metadata tree describing exactly which files make up the table right now, and at every past commit. Understanding that tree is the key to the delta — so let's open it up.
An Iceberg table is a four-level tree. Data at the bottom is ordinary Parquet; everything above it is small metadata that records history and statistics.
metadata.json ← table root: schema, partition spec, and the
│ list of ALL snapshots + the current pointer
├─ snapshot (per commit) — snapshot-id, timestamp, parent, summary
│ │
│ └─ manifest list (Avro) — the manifests live in this snapshot
│ │
│ └─ manifest file (Avro) — lists data files; per entry:
│ ├─ status: ADDED │ EXISTING │ DELETED
│ ├─ record_count
│ └─ per-column min / max / null_count
│ │
│ └─ data file (Parquet) ← your actual rowsTwo things about this tree do all the work:
metadata.json whose "current snapshot" pointer is flipped in one atomic operation. Until that flip, readers see the old table entirely; after it, the new table entirely. Never a torn half-state.Here is the one sentence the whole document turns on: Iceberg records the delta at write time, in metadata. EnergyScope's pipeline recomputes it at read time, by scanning.
When Dagster commits today's data into an Iceberg table, the commit itself knows which files are new versus carried over — and writes that fact into the manifest as the status field. So the question "what changed between yesterday's snapshot and today's?" is answered by reading the manifest entries with status = ADDED in the snapshots since yesterday. A few small Avro reads. You never open, materialize, or diff a single row of the old data.
That is the IncrementalAppendScan. In Spark it reads:
SELECT * FROM prices
-- only files ADDED between two snapshots — pure metadata planning
FOR SYSTEM_VERSION /* incremental */
.option("start-snapshot-id", yesterday)
.option("end-snapshot-id", today)Now contrast what the pipeline does today, in assets.py Phase 2 — the exact code from the review:
# reconstruct the delta by brute force, every run
old = pq.read_table(prev_parquet).to_pylist() # entire prev year → Python
new = pq.read_table(curr_parquet).to_pylist() # entire new year → Python
lookup = {(r.series_id, r.period): r.value for r in old}
for r in new: # O(all rows), in Python
if (r.series_id, r.period) not in lookup: new_row()
elif abs(lookup[...] - r.value) > 1e-4: revised_row()The pipeline reconstructs the delta by brute force every run because nothing recorded it. Iceberg recorded it at commit time, so the read is O(changed files), not O(all rows). For a year partition where one price was revised, that's the difference between reading a few kilobytes of manifest and materializing millions of rows into Python dicts.
Write-time bookkeeping vs read-time scanning. Iceberg pays a tiny cost at commit (write a manifest entry) to make every future "what changed?" nearly free. The hand-rolled pipeline pays nothing at write and everything at read — a full re-scan and diff, on every dataset, every day.
The diff code raises a question: where is the key (series_id, period) actually defined? The honest answer is that it isn't declared anywhere — it's hardcoded as a Python tuple, once, in the body of _compute_delta:
old_lookup[(old_s[i], old_p[i])] = old_v[i]
└──── this tuple literal IS the key definition ────┘There is no config, no schema, no per-dataset setting. "A row's identity is (series_id, period); its payload is value" lives inside that one function and is applied identically to all 24 datasets. Nothing outside the diff loop knows the key exists, nothing enforces it, and changing it for one source means editing the code. The key is code, not metadata.
Iceberg inverts that. You declare the key once, in the table schema, as the table's identifier fields — the columns a revision's delete matches on:
# pyiceberg — the delta key is a first-class property of the TABLE
schema = Schema(
NestedField(1, "series_id", StringType(), required=True),
NestedField(2, "period", StringType(), required=True),
NestedField(3, "value", DoubleType()),
identifier_field_ids=[1, 2], # ← the key, declared ONCE
)Now the key is a property of the table — discoverable through the catalog and honoured by every engine that touches it: the MERGE that writes equality-deletes, the changelog scan, a rollback. It can differ per dataset without a code change, and the engine enforces it (an equality-delete on (series_id, period) is only meaningful because those are the declared identifier fields).
Today you define the key — by hand, as a tuple, and only the diff loop knows it. Iceberg has you declare it once as schema metadata, and the whole system honours it. It's the same shift as the delta itself: move the fact out of imperative code and into versioned metadata, where every reader and writer can see and trust it.
EnergyScope's data hits two kinds of change, and the pipeline's new vs revised split maps exactly onto Iceberg's two mechanisms.
A fresh daily observation is a new row in a new file, tagged ADDED. The cheap IncrementalAppendScan above handles it completely — read the added files, done.
This is the more interesting case — the pipeline's old_val != new_v[i] branch with the 1e-4 float tolerance. Iceberg v2 represents an update as a delete + an insert, using a second kind of manifest content called delete files:
| Delete file type | Says | Used when |
|---|---|---|
| position delete | "row 5 of file X is gone" | the row's exact location is known |
| equality delete | "any row where series_id='PET.RWTC.D' AND period='2024-03-05' is gone" | revising by key — EnergyScope's case |
So a MERGE that revises WTI's price for one day writes an equality-delete on the key plus a new data file carrying the corrected row. Readers apply the deletes on the fly — this is merge-on-read. The old value is never mutated in place; it's shadowed by the delete and superseded by the new row.
To get a clean stream of what changed and how, Iceberg offers a changelog scan (Spark's create_changelog_view). It reconstructs change rows across a snapshot range and tags each one:
A genuinely new row — a new period. Maps to the pipeline's new_count.
A row removed — e.g. a series discontinued or a bad row retracted.
A revision, emitted as a pair: the old value and the new. Maps exactly to the pipeline's revised_count.
Each change row also carries its _change_type and the originating snapshot id. That is precisely the "new vs revised" distinction the Python loop computes by hand — but here it's derived from metadata the writer already produced, not from a row-by-row comparison of two full tables.
There's a real asymmetry hiding here. The pipeline's Phase-2 diff keys rows on (series_id, period) and compares value — but it iterates over the new rows only, checking each against a lookup of the old ones. So it catches a key absent in old → INSERT and a value moved → UPDATE, but it never walks the old rows to find keys that vanished. A (series_id, period) that was in prev/ and is gone from today's cook produces no delta at all — the stale row simply lingers in the Flight server's memory until a full reload.
For EIA-style data this is usually harmless — agencies append and revise, they rarely retract history — which is why it has never bitten. But it is a structural blind spot: the hand-rolled diff computes INSERT and UPDATE and is incapable of emitting DELETE. Iceberg's changelog scan surfaces all three, because the deletion is recorded at write time (a removed data file, or an equality-delete) rather than inferred from a one-directional row comparison. If a vendor ever does drop rows — a discontinued series, a retracted print — this is exactly the case the current design misses and Iceberg would catch for free.
Line the two systems up and the pipeline reveals itself as a hand-rolled subset of Iceberg — one snapshot deep, hashes instead of statistics, a Python loop instead of a metadata read.
.manifest.json — an MD5 hash per year fileprev/ — exactly one snapshot copy (yesterday)to_pylist() + Python dict + loopnew / revised counters — but no deletes (the loop walks new rows only)Two capabilities fall out that the hand-rolled version simply cannot do:
period ranges 2020–2022; a query for 2026 skips it without opening it. An MD5 hash can only tell you that a file changed — never what's inside it.prev/ copy can express.The delta mechanism isn't just an efficiency win. Retaining every snapshot turns "what did the data look like on any past date?" into a first-class query — and for energy market data, that is a product feature, not plumbing.
EIA and most statistical agencies revise figures constantly — a weekly inventory number published Wednesday is refined over the following weeks. With Iceberg snapshots, every daily commit is a queryable vintage:
SELECT * FROM prices
FOR TIMESTAMP AS OF '2024-03-05 00:00:00' -- data as known that day
WHERE series_id = 'PET.WCESTUS1.W' -- US crude stocks, weeklyThis returns the data exactly as it was known on 5 March 2024 — before any later revision. For a quant, that is the difference between an honest backtest and a lie:
Backtesting a strategy on today's revised history secretly feeds it numbers no one had at the time — the model looks prescient because it saw corrected data. Querying the vintage as-of each decision date removes that lookahead bias entirely. Selling revision-safe history is exactly what data vendors like Bloomberg and Haver charge a premium for. Today, EnergyScope's single prev/ snapshot can't answer "what did you show me last Tuesday"; Iceberg makes it a one-line query.
Three more capabilities come from the same snapshot chain, at no extra cost:
date32 + sort + freq-split change is a coordinated schema/partition move — precisely what Iceberg's metadata-only evolution is built to do incrementally, with no big-bang rewrite of history.Iceberg is the strongest architectural upgrade on EnergyScope's board — but it earns a next, not a now, and the reasons are specific.
Iceberg is a lake/table format, not a query engine. The Flight server still reads a snapshot into RAM and serves it exactly as it does now. The in-memory O(1) index — the thing that wins point queries at 0.6 ms — is untouched. Iceberg improves ingest, storage, and history; it does not touch the hot path.
Something has to track "which metadata.json is current." That's a catalog — a REST catalog service, or pyiceberg's lighter SQL / filesystem catalog. One more moving part to run and back up.
The native, mature engine is Java/Spark/Trino. EnergyScope is pure Python → pyiceberg, whose read / append / snapshot-inspection are solid, but full merge-on-read CDC (equality deletes, changelog scan) is where you'd verify current support before betting on it.
Iceberg shines over R2 / S3 — the cold origin tier. The hot local-NVMe scan tier is Vortex's turf. They compose rather than compete.
Start read-only: register the existing R2 Parquet as an Iceberg table and prove vintage / time-travel queries against real history. Then move writes: have Dagster commit snapshots instead of maintaining .manifest.json + prev/, and retire the Phase-2 row-diff in favour of the incremental scan. Because Iceberg wraps the Parquet you already have, this is a staged adoption, not a rewrite — the same files, a metadata layer on top.
Where dbt is the right tool for a job EnergyScope's architecture deliberately avoided (a warehouse full of interdependent SQL), Iceberg is the right tool for a job EnergyScope is currently doing badly by hand — the delta. That contrast is the whole case for prioritising it.
Iceberg didn't invent the category; it won a race. Understanding the runners — what they all fight, how each fights it, and where each is strong and weak — is what turns "we picked Iceberg" into a defensible decision rather than a fashion.
Object storage (S3, R2) is a bucket of immutable blobs: cheap, durable, effectively infinite — and dumb. It has no transactions, no schema, no atomic multi-file operations, and a directory listing that is slow and, being eventually consistent, occasionally a liar. A "table" is a fiction you paint over a folder of Parquet. Every table format is an answer to the same question: how do you get database guarantees on top of a dumb object store? Concretely, each must solve some subset of:
LIST — know exactly which files are the table right now, fast and correctly.Iceberg's answers are the subject of this whole document — a metadata tree, atomic pointer swap, v2 delete files, snapshot history, ID-based schema evolution, and a required external catalog. Here is how the neighbours answer differently.
Before the differences, the surprising part: the three rivals share almost everything. Learn one and ~80% transfers to the others — they diverge in exactly one place.
The same substrate — immutable Parquet data files in object storage, interchangeable between them (and translatable by XTable). The same guarantees — atomic ACID commits, snapshot time-travel + rollback, schema evolution, and hidden / evolvable partitioning. The same dependency — an external catalog to hold the current-version pointer. The same purpose — database guarantees on a dumb object store. None of that is a differentiator; it's the price of admission to the category.
They differ on a single axis: how each records change — and therefore what it is fastest at. That one choice cascades into everything else. Here is each, and then the axis laid out plainly.
Where Iceberg keeps a tree of snapshots, Delta keeps an ordered transaction log — a _delta_log/ directory of numbered JSON commits, each listing add / remove file actions, with a Parquet checkpoint every ~10 commits so readers don't replay the whole log. The log is the single source of truth; a commit is an atomic write of the next log entry. Row-level change is handled by deletion vectors (mark rows deleted without rewriting the file, like Iceberg's position deletes) plus MERGE INTO.
delta-rs gives real non-Spark (Rust/Python) access.Built for a different pain: high-volume upserts and CDC from operational databases. Its differentiator is a record-level index — a map from record key to the file that holds it — so an update is targeted to one file group, never a full-table scan. It offers two table types you pick per workload: Copy-on-Write (rewrite the file on update — read-fast, write-heavy) and Merge-on-Read (append delta logs, merge at read — write-fast, read-heavier). Ships its own ingestion tooling (Hudi Streamer) and async compaction/clustering.
A database storage engine on a lake: primary-key tables backed by an LSM-tree (log-structured merge tree — sorted runs that compact in the background), the same structure that powers write-heavy databases. Streaming-first and Flink-native, it treats a table and a changelog stream as the same object, so real-time upserts land with low latency.
Given the shared foundation above, each format's identity reduces to one design choice — how it records change — and the single strength that follows. Iceberg is in the table too, to place it among its rivals:
| Format | Records change as… | Uniquely best at |
|---|---|---|
| Delta Lake | a linear transaction log | simplicity + the most mature tooling |
| Iceberg | a snapshot tree with per-file stats | pruning + planning at massive file counts |
| Hudi | a record-level index | targeted upserts + native deletes |
| Paimon | an LSM-tree | high-frequency streaming upserts |
So the choice is not really "which format" — it's "which pain": scale of files points to Iceberg, simplicity to Delta, update frequency to Hudi or Paimon. For EnergyScope — many files, revision-heavy, mostly-daily batch — Iceberg's file-scale pruning is the fit, with Hudi's upsert index the one genuinely tempting thing it doesn't match.
Catalogs — the format needs an external authority to hold the atomic "current pointer" and answer discovery/governance. Nessie is the interesting one: git-like branches, tags, commits, and merges over tables, with multi-table transactions — "data as code," and a natural home for tagging vintages or branching a migration. Apache Polaris (Snowflake) and Unity Catalog (Databricks, open-sourced) are the big open REST catalogs; AWS Glue is the managed default. Interop — Apache XTable (formerly OneTable) translates metadata between Iceberg / Delta / Hudi, and Delta UniForm writes Delta while exposing Iceberg metadata. The trend is unmistakable: the data (Parquet) is shared; the formats are competing metadata layers, and translation is making the choice less permanent every year.
No existing format is ideal for a time-series data vendor that serves from RAM across an edge fleet. Each solves part of it. Imagining the format that fits this exact shape — and noting who does each piece today:
delta-rs and pyiceberg are crawling this way; it should be the default, not the hard path.today: partial (delta-rs, pyiceberg)(series_id, period) as a natural composite key, encode value columns with ALP-style float compression, and cluster by (series, period) automatically — instead of treating time-series as generic rows.today: nobody (specialist TSDBs, but those are databases)mmap and index directly — closing the gap between a lake table and an in-RAM serving index. → Explored in full: The Serving Snapshot — design, the v0 EnergyScope already ships, base+delta liveness, prior art, and a weekend experiment to validate it.today: nobody — the unclaimed seamNo single project is that format, and probably none should be — items 1–5 are convergence bets the big three are already racing toward (XTable interop, delta-rs/pyiceberg, deletion vectors), while 6–8 are the genuinely open frontier where a time-series serving vendor could contribute. The closest real assembly today is the composable stack the ecosystem doc keeps returning to: Iceberg + Nessie for the table and its vintages, Vortex + Parquet as the hot/cold file formats beneath it, a Python/Rust writer, and the Flight server as the serving handoff. The wishlist isn't a product to go build — it's the map of which seams to watch, and which two or three (hot/cold file pluggability, the serving handoff) are worth prototyping because nobody has closed them yet.