Apache Iceberg
Also known as: Iceberg, Iceberg table format, Iceberg tables
- Apache Iceberg
- Apache Iceberg is an open table format for huge analytic datasets that gives files in object storage database-like features: ACID transactions, full schema and partition evolution, and snapshot-based time travel, while staying engine-agnostic across tools like Spark, Trino, Flink, and Snowflake.
Apache Iceberg is an open table format that adds database-style reliability — ACID transactions, schema evolution, and snapshot-based time travel — to large datasets stored as plain files in cloud object storage.
What It Is
A modern data lake is often just a pile of files — usually Parquet — sitting in cloud object storage like Amazon S3. That is cheap and flexible, but it has no notion of a “table.” Nothing tracks which files belong together, what the schema is, or what the data looked like last week, and if two jobs write at once, readers can see half-finished results. Apache Iceberg adds a metadata layer that makes a directory of files behave like a real, versioned database table. For anyone building a data versioning system, that last word is the point: Iceberg records every change as a distinct, queryable version of the dataset.
The mechanism is a stack of metadata files that sit above the raw data. Every time you write — an insert, update, or delete — Iceberg does not overwrite anything. It writes new data files and then records a new snapshot: an immutable, point-in-time picture of exactly which files make up the table at that moment. The previous snapshot stays intact. This is close to how a version-control system keeps a commit history: each change produces a new committed state you can inspect or roll back to, without copying the whole dataset.
Three properties make this useful for data lineage work. Time travel lets you query the table as it existed at any past snapshot, so you can reproduce the exact dataset a model trained on. Schema and partition evolution let you rename columns or repartition without rewriting petabytes of files. And it is engine-agnostic — Spark, Trino, Flink, and Snowflake can all read and write the same Iceberg table. According to the Apache Iceberg Spec, format versions 1, 2, and 3 are adopted, with version 3 generally available and a version 4 still in development; version 3 adds row lineage and deletion vectors, which track row-level changes and speed up deletes. According to the Apache Iceberg Docs, the current library release is 1.11.0.
How It’s Used in Practice
Most people meet Apache Iceberg through a cloud data platform, not by installing anything themselves. Snowflake, Databricks, AWS, and Google BigQuery all support Iceberg, so a data team can keep one copy of their large datasets in open Iceberg tables and point several engines at it, instead of loading the same data into a proprietary warehouse and getting locked in. The pattern is usually called a “lakehouse”: warehouse-grade reliability on cheap lake storage.
In a data versioning or MLOps setting, the snapshot history is the draw. When a machine learning team trains a model, they can pin the exact snapshot of the training table and record its ID next to the model. Months later, when someone asks why the model behaves the way it does, the team can time-travel back to that exact snapshot and see the data as it was — the foundation of reproducible experiments and auditable data lineage.
Pro Tip: Old snapshots are not free — every retained version keeps its data files alive in storage. Schedule snapshot expiration and orphan-file cleanup from the start, and treat Iceberg as the table layer, not a full replacement for dedicated dataset-versioning tools like lakeFS or DVC. It versions tables well, but not your loose files or model binaries.
When to Use / When Not
| Scenario | Use | Avoid |
|---|---|---|
| Large datasets several engines (Spark, Trino, Snowflake) must read | ✅ | |
| Reproducible training data needing time-travel to past snapshots | ✅ | |
| Schemas or partitioning change often; rewriting all files is infeasible | ✅ | |
| A small dataset that fits in one database or spreadsheet | ❌ | |
| High-frequency, low-latency transactional writes (OLTP) | ❌ | |
| Versioning loose files, images, or model binaries, not tables | ❌ |
Common Misconception
Myth: Apache Iceberg is a file format that competes with Parquet, or a database engine you query directly. Reality: Iceberg is a table format — a metadata layer. It organizes files (usually Parquet) into versioned tables and tells engines like Spark or Trino how to read them. It stores no data of its own and runs no queries itself; the engine does that.
One Sentence to Remember
Apache Iceberg turns a folder of files in object storage into a versioned table with a full snapshot history, which is exactly what lets a data lake support the time travel, reproducibility, and lineage tracking that a serious data versioning system depends on.
FAQ
Q: What is the difference between Apache Iceberg and Parquet? A: Parquet is a file format that stores the actual data on disk. Iceberg is a table format that groups many Parquet files into one versioned, queryable table with schema tracking and snapshots.
Q: How does Iceberg support data versioning? A: Every write creates a new immutable snapshot instead of overwriting files. You can query or roll back to any past snapshot by its ID, giving you point-in-time versions of a dataset.
Q: Is Apache Iceberg the same as Delta Lake? A: No. Both are open table formats with ACID transactions and time travel, but they use different specs and metadata designs. Iceberg is engine-agnostic; Delta Lake grew out of the Spark and Databricks ecosystem.
Sources
- Apache Iceberg Spec: Spec — Apache Iceberg - The official table format specification, including adopted format versions 1–3 and v3 features.
- Apache Iceberg Docs: Releases — Apache Iceberg - Official release notes listing current library versions.
Expert Takes
The core idea is immutability. Iceberg never edits a file in place; it writes new files and records a new snapshot that points at the current set. A table is therefore not a thing but a sequence of states. Not storage. Bookkeeping. The data already sat in object storage — what Iceberg adds is a disciplined ledger of which files count as the table at each moment.
Treat the snapshot ID as part of your pipeline’s contract. A reproducible workflow should not say “read the customers table” — it should say “read the customers table at this snapshot.” Write that ID into your run metadata next to the model version. Once the exact dataset state is named explicitly, a whole class of “it worked last week” debugging arguments disappears, because the inputs are pinned, not assumed.
The strategic shift is the separation of storage from compute. Lock your data inside one vendor’s warehouse and you rent their engine forever. Keep it in an open table format and the engines compete for your workload instead. That is why every major platform raced to support Iceberg rather than fight it. Owning the format stopped being the moat. The data stays put; the tools come to it.
A perfect history is also a perfect memory, and that cuts both ways. Snapshots make it trivial to prove what data a model was trained on, a gift for accountability. But the same retained history can preserve personal records long after someone asked to be forgotten. Who decides when a snapshot expires, and does the deletion vector that hides a row also truly erase it? Versioning is not the same as forgetting.