MMLU Benchmark

MMLU is the benchmark most teams cite first when comparing model capability, which makes its cracks — a documented label-error rate, a ceiling most frontier models now sit under — everyone’s problem, not just researchers’. It sits inside the model evaluation theme as the field’s most-referenced knowledge test, one entry point among several this topic routes you through in order. Reading it in sequence matters more than reading MMLU headlines in isolation, because the score means something different depending on whether you’re choosing a model, tracking the market, or auditing what optimizing for it actually rewards.

Reading MMLU in the right order: from the exam to its ethics

Start with what MMLU actually tests across 57 academic subjects — it explains what “57 subjects” means in practice and why the exam looks the way it does. Before trusting any score, read MMLU’s 6.5% label-error rate and its score saturation: it is the prerequisite for reading every later MMLU number honestly, because part of what looks like model improvement is just noise in the answer key.

When you need to run the evaluation yourself, the MMLU evaluation and model-selection guide walks through the configuration choices — shot count, prompt format, which test version — that change the number you get. For where the market stands right now, GPT-5’s 92.5% and MMLU-Pro’s rise tracks how tightly frontier models now cluster and why the field is migrating to a harder successor. Close with the ethics of optimizing for the MMLU number — what chasing this specific score has cost in contamination, and whose knowledge gets tested at all.

MMLU versus its successor, its failure mode, and its rival rankings

Three things get folded into “the MMLU score” that are worth pulling apart.

• MMLU is not MMLU-Pro. The original exam is four-choice and saturated above roughly 88% — most frontier models now cluster within a handful of points of each other. MMLU-Pro answers that saturation with ten-way questions and harder items, which is why a 2026 model comparison that still leans on the original exam is comparing models in the part of the range where MMLU can no longer tell them apart.
• MMLU’s error rate is not the general contamination problem. A roughly 6.5% label-error rate is a property of this one answer key; benchmark contamination is the broader failure where a model has simply seen the test questions during training, on any benchmark. A model can clear a clean MMLU run and still be contaminated elsewhere, or lose points on a correct answer MMLU happens to have mislabeled.
• MMLU is not a live ranking. The exam grades a fixed answer key once; ELO rating for LLMs re-ranks models continuously from human preference votes. A model can lead on one and lag on the other, because they measure recall of settled facts versus what people prefer in an open conversation.

Questions worth asking before you cite an MMLU score

Q: Should I trust an MMLU number I found on a vendor’s product page? A: Not without its evaluation spec — scores shift with shot count, prompt format, and which test version was run, so an unqualified percentage is unverifiable. Run your own locked evaluation before treating a marketing number as fact.

Q: Why do two models with almost identical MMLU scores sometimes behave very differently in production? A: Because frontier scores now cluster within a few points of each other, MMLU has stopped separating leaders from the pack — a near-tie on the exam says nothing about a near-tie on your workload.

Q: Does MMLU’s contamination and bias problem still matter if I only treat the score as one input among several? A: Less, but not zero — contamination and cultural framing distort the number itself, so even a heavily discounted MMLU score is still built on a compromised measurement, not a neutral one you’re merely weighting down.

Part of the model evaluation theme · closest neighbour: benchmark contamination. New to evaluation from a software background? Start with the story: Model Evaluation for Developers: What Maps and What Misleads.