MONA explainer 7 min read

What Is an Ablation Study in Machine Learning? How Removing One Part Shows What It Contributes

Mona holds a stopwatch and studies a row of boxes with one removed beside two bars, under the hint "remove one part"

What is an ablation study in machine learning?

An ablation study in machine learning removes or replaces one part of a system and measures how the result changes.

It measures how much the result depends on that part, in that system and task. Researchers use ablation studies to justify that an addition improves performance.

An ablation study as a race. Lap times are illustrative.

An ablation study in machine learning belongs to model evaluation: checking how well a model performs a task. First, the full system must work and have a measured result; then the study tests a proposed explanation for that result. The Ablation Study page covers the wider subject.

A model builder can then decide whether to keep, remove or simplify a part.

How does removing a part show what it contributes?

A component is a part of a model or its training, such as one of the network’s internal units, or a training objective, the task used to teach it. An ablation experiment usually changes one component at a time, keeping the other conditions the same.

  1. Measure the full system. The baseline is the full system, measured first, against which you compare each changed version. Score it on an evaluation set: examples kept out of training and used to assess each version in the same way. A score, or metric, is the resulting number, such as accuracy, the share of answers that are correct.
  2. Remove one component. The variant is the system with that component removed or replaced. Keep the remaining conditions the same.
  3. Retrain the variant. Usually, retraining means training it again from the start, so the remaining parts can adapt without the removed component. Some studies instead switch the part off in an already trained model (see below).
  4. Evaluate and compare. Use the same evaluation set and metric for the baseline and variant. Good practice is to train and evaluate each more than once: run-to-run variation is the change in score between training runs of the same system, caused in part by random starting values and data order.

The comparison can produce three results: the part is load-bearing if the score drops clearly; not needed for that result if the change is no larger than the variation; or harmful to that result if the score rises clearly without it. Each verdict applies only to the system and task tested.

An ablation study compares the baseline with a variant, usually after retraining.

What do the three outcomes look like?

Load-bearing for that result. In the BERT language model study (BERT-BASE, development-set scores), removing next sentence prediction, a training task that asks whether one sentence follows another, reduced accuracy on QNLI, a task that asks whether a sentence answers a question, from 88.4% to 84.9%, while the other tasks dropped much less.

The paper does not report repeated runs for this comparison.

Not needed for the measured result. In Michel and colleagues’ study, researchers switched off individual attention heads, parts of a Transformer translation model, each of which can attend to different words in the input, in an already trained model.

The model was a pretrained Transformer with the original large architecture, trained for English-to-French translation.

Of its 96 encoder self-attention heads, only 8 changed the translation score significantly when switched off, by a statistical test of whether a change is larger than chance differences between the evaluation sentences: the other heads were not needed for that result with the rest of the trained model in place.

Harmful to the measured result. In the same test, 4 of those 8 heads raised the translation score when switched off. Removing those heads improved the measured result, though this does not show what would happen if the model were trained without them. That test compared evaluation sentences of one trained model, not repeated training runs.

How much change is enough?

A small difference can come from the training run rather than the removed part. How large a difference counts depends on how far the score moves between training runs: train the same system several times, see how far its scores spread, and treat only a larger difference as a finding.

If you only wanted to know what an ablation study is and how to read one, you can stop here.

Remove, replace or switch off?

Some systems cannot work with a part completely removed. Replace it with a simpler part instead, then interpret the result as a comparison between those choices, not as the effect of having no part at all. The guide to ablation experiment design explains how to choose a baseline and what to change.

Switching off means removing a part from an already trained model without training it again. It asks how much that particular trained model relies on the part.

Retraining from the start asks how well the system can do without it after the other parts have had a chance to adapt. Use retraining for the latter question; the two results need not agree.

For example, Meyes and colleagues found that an image-classification network recovered most of its lost accuracy after further training following damage, showing that the layers left free to train could adapt. Their recovery training began with the damaged trained network, rather than starting over.

What can one-at-a-time testing miss?

An interaction occurs when the effect of two parts together differs from the sum of their effects separately. Meyes and colleagues found that removing two units together in a small image-classification network hurt performance more than adding up the effects of removing each unit alone. Individual removals would not reveal the full combined effect.

A factorial design tests combinations of choices, such as keeping or removing each part. Testing every combination of k parts with two choices each takes 2^k configurations, so the number doubles with each additional part before repeated runs are counted. The explanation of interaction effects and scaling limits examines what this leaves out.

When it breaks: A one-at-a-time ablation analysis cannot rule out interactions it did not test. Treat a small individual effect as evidence about that particular comparison, not proof that the part can always be removed safely.

How is ablation different from tuning?

Tuning searches for settings that improve performance; ablation asks what a component contributes. Give every variant the same tuning effort, or a score difference may reflect a better search for settings rather than the removed part. Melis and colleagues showed that a series of published language-model improvements came from better tuning, as reported by Lipton and Steinhardt.

Cheat sheet

The article on one page: open it full size, save it or share it.

Ablation study cheat sheet: four steps, three results, how much change counts, and what one-at-a-time testing misses
PNG

AI-assisted content, human-reviewed. Images AI-generated. Editorial Standards · Our Editors