MONA explainer 15 min read

What Is an Eval Harness? How Fixed Tests Make LLM and Agent Scores Comparable

Mona points to a diagram of an evaluation harness: model input, a model, a dataset and metrics lead to evaluation results

What is an eval harness?

An eval harness (also written evaluation harness) is the software that runs an AI model or agent through a fixed set of tests, grades every answer by the same rule and keeps the conditions the same, so that two scores can be compared.

It lets you compare two models, or two versions of an agent, without comparing two different setups. In our test, one model answered the same 114 questions in five different setups, five ways of asking and grading, and scored from 0 % to 63.2 %. When we repeated each setup 10 times, it gave the same score every time.

Where an eval harness fits, and how the same model scores from 0.0 % to 63.2 % in five setups. bestaiweb.ai test, 1 November 2026, lm-evaluation-harness 0.4.13, Qwen2.5-1.5B-Instruct.

Before you trust an AI model, you need to know how good its answers are. That work is called model evaluation.

It has a few kinds of tools: basic scores such as precision and recall, benchmarks (fixed tests with known answers) that measure what a model can do, and human raters or another model acting as a judge for open answers.

An eval harness belongs to the benchmark part. A benchmark is only a list of questions and answers: something has to put each question to the model, grade the answer and add the grades up into a score. That software is the eval harness.

Before it comes the choice of which benchmark to run. After it comes the question whether the score can be trusted, for example whether the test questions leaked into the model’s training data. Our model evaluation theme shows the whole map.

One small AI model, the same 114 questions from MMLU (a multiple-choice test), five setups (for example with or without solved example questions in the prompt) and five scores: 63 %, 61 %, 61 %, 40 % and 0 %. We changed only how we asked and graded.

Is a score a fact about the model, or about how it was tested? That is the question an eval harness exists to settle.

What an eval harness does

Step by step, the eval harness puts each question to the model, collects the answer, grades it and adds up the results. Within model evaluation, it is the part that makes a benchmark score repeatable.

Its job is to keep the test the same, so that a difference in score points to the model and not to the test. EleutherAI’s lm-evaluation-harness is an open-source harness for language models, and the one we used.

What stays fixed and what you swap

A harness fixes four things (our grouping) and swaps only the model or agent under test.

Fixed partIn plain words
The tasksThe same questions for every model
How a task is shownThe prompt: the exact text the model reads. It may include a few solved example questions before the real one (few-shot examples)
How the answer is gradedThe rule that turns the model’s answer into right or wrong
The conditionsWhether the prompt is wrapped in the model’s chat format, the markers chat models use to show who is speaking (step 4 shows it), and the version of the harness

Change the tasks, the prompt, the grading or the chat format and the score can move, even when the benchmark keeps its name.

An eval harness fixes the tasks, the prompt format, the scoring and the conditions, and swaps only the model or agent. Our grouping, based on Anthropic's definition and the lm-evaluation-harness docs.

Follow one question through the harness

Take a real MMLU question: “The study of older adults and aging is referred to as …”, with the options A. Gerontology, B. Geropsychiatry, C. Geriatrics and D. Gero-education. The right answer is A.

Our test ran every question in five setups, five ways of asking and grading: 1. the default; 2. five solved examples first; 3. the answer options hidden; 4. the model writes its answer; 5. setup 4 wrapped in the chat format. We follow this one question through all five.

1. The harness writes the prompt (setups 1 and 2)

The harness first builds the prompt, the exact text the model reads. In the first setup, the harness default, it looks like this:

The following are multiple choice questions (with answers) about human aging.

The study of older adults and aging is referred to as
A. Gerontology
B. Geropsychiatry
C. Geriatrics
D. Gero-education
Answer:

That text is all the model has. Drop the word “Answer:” and you have changed the test. In the second setup, five solved example questions came first, each ending in a line such as “Answer: C”. These are few-shot examples: plain text in the prompt, with no retraining of the model.

2. The harness grades the answer (setups 1, 3 and 4)

The harness now needs a verdict. One way: a language model works by predicting what text comes next and gives every candidate a probability, so the harness reads how likely the model finds “A”, “B”, “C” and “D” after “Answer:” and takes the most likely.

Here the model gave A a probability of about 98 % and C, the next best, about 2 %, so the pick is A, which is correct. This needs access to those probabilities, which closed models behind an API do not always give.

The other way: the model writes its answer and the harness compares it with the correct letter. Our model wrote “A”, which is correct. With the options shown and without the chat format, both ways (our setups 1 and 4) gave the same verdict on all 114 questions.

A third way (setup 3) hides the list: the prompt shows only the question and “Answer:”, and the harness checks how likely the model finds each option’s text, such as “Gerontology” or “Geriatrics”, as the next words. Here “Gerontology” was still the most likely, but over all 114 questions this setup scored 21 points lower than setups 1 and 4.

3. The harness adds up the grades

The score is the share of questions graded correct, called accuracy: 70 of 114 right is 61.4 %. Other tasks use other metrics, such as precision, recall and F1, counted from a confusion matrix of right and wrong answers; this article needs only accuracy.

4. The same question in the chat format (setup 5)

In the fifth setup the harness wrapped the prompt in the model’s chat format, special markers that say who is speaking. The same question then looked like this:

<|im_start|>system
The following are multiple choice questions (with answers) about human aging.

<|im_end|>
<|im_start|>user
The study of older adults and aging is referred to as
A. Gerontology
B. Geropsychiatry
C. Geriatrics
D. Gero-education
Answer:<|im_end|>
<|im_start|>assistant

The model answered “A. Gerontology”, which is right. But the grading rule of this setup, the same as in setup 4, compares the whole first line of the answer with the single letter “A”, so the answer was graded wrong.

Wrapped like a chat message, the question got answers like in a chat, never a bare letter. We only saw that this happened; the test does not show why.

Of the 114 answers, 52 began with the right letter and the option text, 25 with a wrong letter, and 37 with a sentence instead of a letter, such as “To determine the correct answer …” or “The correct answer is D”. All 114 were graded wrong.

A score of 0 % does not mean the model failed every question. The grading rule did not fit how the model now answers.

Our test: what each setup shows

We ran lm-evaluation-harness 0.4.13 on 1 November 2026 with Qwen2.5-1.5B-Instruct, a small open model with 1.5 billion parameters, small enough to run on the laptop we used. The questions were the first two of each of MMLU’s 57 subjects, 114 in all, the same in every setup. Each setup ran 10 times, 50 runs in total.

Setups 2 and 4 change one thing against setup 1. Setup 3 changes the task format: the prompt shows only the question, with a slightly different instruction line, and the harness scores the answer text. Setup 5 changes one thing against setup 4: the chat format.

Changing one part at a time and watching the score shows which change moved it. Used on the parts of a model, the same one-change-at-a-time logic is called an ablation study.

SetupWhat the harness doesScoreWhat it shows
1Picks the answer letter the model finds most likely; no examples in the prompt (the harness default; published MMLU scores often use 5 examples)61.4 % (70 of 114)The starting point that the other four are compared with
2As setup 1, with 5 solved example questions in the prompt63.2 % (72)The total moved by 2 questions, but 20 changed verdict: 11 right only with examples, 9 only without. Close totals can hide different right answers
3The prompt shows only the question, without the A–D list. The harness then puts each option’s text, such as “Gerontology” or “Geriatrics”, after “Answer:” and picks the one the model finds most likely40.4 % (46)Changing the task format cost 21 points; we cannot say which of its changes did it
4The model writes its answer; the harness checks that the first line equals the correct letter61.4 % (70)Another grading method reached the same 70 right answers, so a new method does not always change a score
5As setup 4, with the prompt in the model’s chat format0.0 % (0)A grading rule that does not fit the shape of the answers can wipe out a score
Same model, same questions, five harness setups, 10 runs each. bestaiweb.ai test, 1 November 2026, lm-evaluation-harness 0.4.13, Qwen2.5-1.5B-Instruct.

A harness can pick example questions at random, and agents behave differently from run to run. Our setups used fixed examples and no agent, but we still repeated each setup 10 times to check that its score does not change by chance. If it did not stay the same, one score would tell us little about the setup.

Every setup gave one and the same score in all 10 of its runs, so on this machine and with these software versions a fixed setup gave a repeatable score. In our reading, that is the precondition for comparing two models on one setup.

Across the five setups the same model on the same questions ranged from 0.0 % to 63.2 %. In our test, the setup alone moved the score by 63 points.

Read our numbers as a demonstration of the mechanism, not as this model’s MMLU score. We used one small model, 114 questions and one MacBook Air, with the model running on Apple’s graphics chip, where the harness README warns that results may contain errors. We did not test other machines.

In our reading, this does not change the lesson about setups, but our exact percentages may not match a full MMLU run or another computer.

The harness’s docs also say that the --limit option, which we used to take two questions per subject, is meant for testing, not for a full score. Setup 5’s zero shows that this grading rule does not fit this model’s chat-format answers; we did not test a task graded for chat answers.

If you only wanted to know what an eval harness is and why one model can get different scores, you can stop here. The rest shows the same problem in published results, how to read a score someone else reports, what changes for AI agents, and what a harness cannot fix.

The same gap outside our lab

In June 2023 Hugging Face ran a set of models on MMLU through three implementations: the code of Stanford’s HELM, another evaluation framework; the EleutherAI harness (as of January 2023); and the original MMLU code.

For LLaMA-65B the scores were 0.637 with HELM, 0.488 with the EleutherAI harness and 0.636 with the original code. That is a gap of almost 15 points (our subtraction) for one model. Hugging Face wrote that such implementations give “widely different numbers”.

The three did not ask or grade the same way. The prompts differed in small ways, such as a “Question:” prefix.

The original compared the probabilities of the four answer letters, HELM compared the text the model generated with the expected answer, and the harness compared the probability of the letter followed by its text. Our setups 1, 4 and 3 use similar methods, not copies: setup 3, for example, scores the option text without the letter.

It is the same pattern as in our test: the model was the same in all three runs, and the setup was not. Hugging Face concluded that such numbers are “not at all comparable even if they’re both labelled as ‘MMLU score’”.

How to read a score someone else published

Find the setup behind the number. Five questions cover it:

  • Which harness, and which version? Releases fix bugs. Version 0.4.13 fixed a bug in a filter, a small piece of code that pulls the answer out of the model’s text. The bug scored some correct answers as wrong, and the release notes say such fixes may shift previously reported numbers. Compare only within one version.
  • How many solved examples were in the prompt? In our test, five examples changed the verdict on 20 of 114 questions.
  • How were answers graded? The rule decides what counts as correct, and a rigid one produced our 0 %.
  • Was the chat format used? The same grading rule gave 61.4 % without it and 0.0 % with it.
  • Are the prompts and the evaluation code published? The harness developers’ paper “Lessons from the Trenches” (Biderman et al.) asks authors to release the exact evaluation code whenever possible, and to report the prompts in the publication. Without them, the other four are often hard to answer.

Read some of the model’s actual answers before you trust a total. Only that showed us why setup 5 scored 0 %. To run a benchmark yourself, our guide to benchmarking LLMs with lm-evaluation-harness, HELM and OpenCompass walks through it step by step.

The same idea for AI agents

An agent is an AI system that takes steps on its own, such as calling tools or running commands. An eval harness for agents follows the same idea: fixed tasks, a fixed way of grading and a clean starting point for every attempt. Agent behaviour varies between runs, so each task is run several times.

A grading rule can hide a good agent, as it did in our setup 5. Anthropic reports that Opus 4.5, an Anthropic model, first scored 42 % on the agent benchmark CORE-Bench until a researcher found multiple issues, among them a grader that penalized “96.12” when it expected “96.124991…”.

After the bugs were fixed and the agent got a less constrained scaffold, the system around a model that lets it act as an agent, the score jumped to 95 %.

The computer the agent runs on counts too. In a separate post on infrastructure noise, Anthropic reports that on Terminal-Bench 2.0, an agentic coding benchmark, the setups with the most and the least computing resources differed by 6 points, with the same Claude model, harness and tasks.

In Anthropic’s words: “Two agents with different resource budgets and time limits aren’t taking the same test.”

What a harness can’t fix

A harness keeps the measurement constant. It does not make the measurement valid.

It grades against the dataset’s answer key. Researchers who re-checked MMLU questions (Gema et al.) estimate that 6.49 % of them contain errors, and found errors in 57 % of the Virology questions they analysed. The errors include a wrong key, several correct answers and unclear questions. A wrong key gives a wrong grade whatever model you run.

It also cannot tell you whether the test questions leaked into the model’s training data, so the model may have seen the answers before. The harness’s docs describe a check for this, but it is off by default and our runs did not use it. The problem has its own page on benchmark contamination.

When it breaks: a harness reports a score for the benchmark you gave it. It cannot check that the benchmark is a valid stand-in for the ability you care about.

Anthropic defines saturation as an agent passing all of the solvable tasks, so no room for improvement is left. As a benchmark nears saturation, Anthropic notes, progress slows because only the hardest tasks remain, and large capability improvements can appear as small increases in scores.

Its January 2026 post says scores on SWE-bench Verified, a coding benchmark built from real GitHub issues, rose from 30 % to above 80 % and were nearing saturation. In our reading, a small gap between two top models on such a benchmark may therefore hide a larger difference in ability.

Cheat sheet

The article on one page: open it full size, save it or share it.

Eval harness cheat sheet: what a harness fixes, how to check a score you did not run, agent eval rules, a 0 % to 63.2 % test
PNG

AI-assisted content, human-reviewed. Images AI-generated. Editorial Standards · Our Editors