MONA explainer 10 min read August 6, 2026

Storage Bloat, Binary Merge Conflicts, and the Scaling Limits of Data Versioning

How data versioning tools bloat storage, fail to merge binary files, and hit scaling limits at petabyte size

ELI5

Data versioning tracks every change to a dataset the way Git tracks code. But datasets are enormous binary blobs, so the tools bloat storage, cannot merge two versions of a file, and strain once data climbs toward petabyte size.

Edit a single byte inside a ten-gigabyte training file. Commit it. Watch your repository grow by another ten gigabytes. Nothing new was created; the same data, almost identical to its previous self, now lives twice. This is not a bug to file. It is the predictable arithmetic of how Data Versioning tools store what they track, and it is the first clue to why Git’s elegant model quietly fails when you aim it at data instead of code.

The pitch is seductive: Git, but for datasets. Branch an experiment, diff two snapshots, roll back a poisoned label. The promise assumes datasets behave like source code. They do not, and the distance between those two assumptions is exactly where storage costs detonate and merges fail without warning.

What Git Does to a Ten-Gigabyte File

Git was designed for text that changes line by line. It stores history cheaply because it can describe a new version as a small set of edits against the old one. A dataset offers no such courtesy. A model checkpoint, an image, a Parquet shard: these are opaque blobs with no line structure to compare. So the first generation of data versioning tools never tried. They performed a sleight of hand instead.

Git LFS replaces each large file in your repository with a pointer file under a kilobyte in size, a few lines of text holding a SHA-256 hash that identifies the real bytes; the actual data lives on a separate server and is pulled down only when you check that version out (git-lfs.com). DVC plays the same trick, writing a small .dvc pointer that holds a content hash while the data sits in a content-addressed cache backed by remote storage like S3. Both rely on Content Addressable Storage: a file is stored and retrieved by a fingerprint computed from its own contents.

That fingerprint is the entire story. Content addressing means a file’s identity is its bytes. Change one byte and the fingerprint changes completely, so to the system the edited file is not a near-twin of the original; it is a brand-new object with no relationship to what came before. The tool deduplicates by whole object, never looking inside the file. The opening anomaly follows directly: a one-byte edit to a ten-gigabyte file stores another full ten-gigabyte copy, and those copies accumulate version after version. Not a bug. Arithmetic.

What are the limitations of data versioning tools?

Three limits show up in practice, and they compound on each other.

The first is storage bloat from whole-file deduplication. Because the file-pointer tools dedup at the level of the entire object, every revision of a large binary is a full new copy. Versioning a dataset that changes often does not cost you the size of your changes; it costs you the size of the whole dataset, again, every single time.

The second is the merge that cannot happen. Git reconciles two edited text files by comparing them line against line and splicing the differences together, a three-way merge. A binary has no lines. Ask Git to merge two versions of an image or a checkpoint and it has nothing to compare, so it declines to blend them at all, forcing you to keep one whole version or the other with --ours or --theirs (git-scm Docs). Calling this a “merge conflict” flatters it. There are no conflict markers, no middle path. Not a conflict. A standoff: one version survives, the other’s changes are discarded.

The third is platform quotas that arrive sooner than you expect. On GitHub, Git LFS rejects any single file larger than 5 GB, a platform ceiling raised from an older 2 GB cap and not a property of the Git LFS protocol itself (GitHub Docs); other hosts set their own. The free tier covers one gigabyte of LFS storage and one gigabyte of bandwidth per month before it bills or blocks, and files committed outside LFS hit a hard wall at 100 MiB. None of these are large numbers next to a real dataset.

A fourth limit is conceptual rather than numerical: no single tool covers every case. The file-pointer family is simple but bloats on large, frequently edited binaries. Object-store overlays scale on a different principle. Table formats version efficiently but only for tabular data. Treating any one family as the universal answer is the most common architectural mistake, and it sets up the scaling wall.

When the Dataset Outgrows the Repository

The bloat problem has an obvious-sounding fix: stop copying data when you branch. That instinct is correct, and it is exactly what the petabyte-scale tools are built around. Following it also exposes a harder limit that no amount of clever bookkeeping can erase.

Why does data versioning struggle with petabyte-scale datasets?

The branching problem, at least, is solved elegantly. lakeFS sits as a Git-like layer over object storage and makes branch creation a metadata-only operation: a new branch is a label pointing at existing objects, not a copy of their bytes, so you can branch a multi-terabyte or even petabyte dataset in seconds (lakeFS Docs). Its committed metadata is a content-addressed Merkle tree of table-range files kept in the object store itself, with a separate key-value store tracking references and uncommitted state. New bytes are written only when data actually changes, copy-on-write. This is bookkeeping, not compression.

But branching cheaply is not the same as storing cheaply. Every retained version’s bytes still sit physically in the object store. Versioning changes how you reference history; it does not make the history weigh less. Keep a hundred snapshots of a growing dataset and you pay for a hundred snapshots, branch-in-seconds or not. Petabyte scale is the metadata win, not a storage miracle.

Which returns us to deduplication, the real lever on the bill. The whole-file approach is the wasteful one. Content-defined chunking takes the opposite tack: it splits a file along boundaries determined by the content itself rather than at fixed offsets, so that inserting or changing a few bytes shifts only the affected chunks and leaves every other chunk with an unchanged fingerprint. Only the changed chunks are stored or transferred. Hugging Face’s Xet, built on this idea, reports transfer reductions above 90% in some cases (Hugging Face). That is the structural answer to the opening anomaly: deduplicate below the level of the whole file.

Table formats occupy a narrower lane. Delta Lake and Apache Iceberg version data by writing immutable snapshots over Parquet and let you query a table “as of” an earlier timestamp or snapshot id, time travel with ACID guarantees. The mechanism is efficient, but it is scoped to tabular lakehouse data. It will not version your image corpus or your model checkpoints, which is precisely why the three families remain distinct tools rather than one.

Three families of data versioning tools and where each one hits its storage or merge limit — File-pointer, object-store overlay, and table format: three versioning families, three different scaling limits.

Versions and security notes (as of June 2026):
Git LFS 3.7.1 is the current release and ships as a security update; the maintainers recommend updating all clients (Git LFS GitHub releases).
DVC changed stewardship in November 2025, when lakeFS acquired it from Iterative.ai; it remains fully open source under the same license (DVC Blog), not an abandoned project.
lakeFS 0.106.1, Delta Lake 4.1.0, and Apache Iceberg 1.10.1 (table-format spec v3) are the current stable versions referenced here.

What the Bookkeeping Predicts About Your Storage Bill

Once you see versioning as content-addressed bookkeeping, the failure modes stop being surprises and start being predictions.

If you version large binaries that change often with a whole-file tool, expect storage to grow roughly with dataset size multiplied by revision count, not with the size of your edits.
If you branch a dataset on an object-store overlay and the bill barely moves, that is the metadata-only design working as intended, right up until you retain many divergent versions, at which point the physical bytes, not the branches, drive the cost.
If your data is purely tabular, a table format will version it far more cheaply than any file-pointer tool, because it tracks snapshot-level changes instead of opaque blobs.
If two collaborators edit the same binary on separate branches, do not expect a merge; expect to choose one version and lose the other’s work.

Rule of thumb: match the tool to the shape of the data, file-pointer for occasional large files, object-store overlay for scale and cheap branching, table format for tabular data, and chunk-level deduplication when the same large files change repeatedly.

When it breaks: the model fails hardest when one tool is stretched across every data type at once, versioning frequently-edited multi-gigabyte binaries with whole-file deduplication, where each minor revision silently stores a full new copy and the repository grows without bound.

The Data Says

Data versioning borrows Git’s vocabulary but not its physics. The tools that scale do so by versioning metadata cheaply, never by making the underlying bytes cost less, and the single largest lever on storage is whether deduplication happens at the level of the whole file or the individual chunk. Choose the family that matches your data’s shape, because no tool escapes the arithmetic of storing what it keeps.

Aha Moments

MAX

Mona is describing an architecture decision that most teams make by accident. The mistake is reaching for whatever tool is closest to the keyboard, usually the file-pointer one, and discovering the storage bill months later when it is expensive to change. The fix is upstream: decide, before the first commit, which family fits each kind of artifact. Binaries that rarely change, tabular data, model checkpoints that churn every run, these are different problems with different right answers. Write that mapping into the project’s data contract the same way you would specify an API. When the constraint is named in advance, the bloat Mona describes never gets a chance to compound, because nothing frequently-edited ever lands in a whole-file store to begin with.

DAN

Max is right that it is a decision, and the market has already decided which way it leans. The direction is unmistakable: away from whole-file copying and toward chunk-level deduplication and object-store overlays. The consolidation tells the story by itself, with the established players folding the smaller pointer-file tools into platforms built for scale. That is not a tidy detail, it is a signal about where the spend is going. Teams standardizing on file-pointer tooling today are optimizing for the workflow that the rest of the field is already migrating off. You either pick the architecture that survives the next jump in dataset size, or you rebuild your data layer under deadline pressure when the bill forces your hand.

ALAN

Dan frames it as surviving the next jump in scale. I would ask what we are choosing to keep forever when we make versioning this cheap. Every retained snapshot preserves not only the dataset but every mistake inside it, the mislabeled records, the biased sampling, the records a person once asked to have deleted. Cheap branching quietly turns into permanent memory, and permanent memory collides with the right to be forgotten and with our duty not to relitigate someone’s data without consent. Reproducibility is a genuine good; so is the ability to actually erase something. When immutable history and deletion obligations point in opposite directions, which one does your versioning strategy honor, and who decided that on the team’s behalf?

AI-assisted content, human-reviewed. Images AI-generated. Editorial Standards · Our Editors