How to Benchmark a Local LLM's Accuracy with lm-evaluation-harness

How do you benchmark a local LLM?
To benchmark a local LLM, a language model that runs on your own computer, install lm-evaluation-harness, a free program that puts a fixed set of test questions to a model, and run one command naming your model and a test. It prints the share of questions answered correctly.
Use it to compare two local models. In our laptop test on 114 of the 14,042 questions of MMLU, a multiple-choice test, one small model scored 61.4 % whether we changed how many questions went to the model at once or ran it on the processor instead of the graphics chip.
When you benchmark a local LLM, a model that runs on your own computer, you do one step of model evaluation, the work of measuring how good a model’s answers are.
You put a benchmark, a fixed test with known answers, to the model and get a score. Before this step you pick the model and the test; after it you compare scores and decide which model to keep.
A teammate asks whether the open model you downloaded beats last month’s. You write a quick script that asks both models twenty questions. Numbers come out, but nobody can repeat your script, so nobody knows what they mean.
To benchmark a local LLM in a way others can repeat, you run a shared test with a shared tool. That tool is an evaluation harness. This guide uses one, and the wider field is Evaluation Harness.
Of all that model evaluation can measure, this guide measures accuracy: how often the model gives the right answer. It does not measure speed, such as words per second or memory use.
What you need, and what each part does
You need Python 3.10 or newer and pip, Python’s installer for packages. The model comes from Hugging Face, a website that hosts open models, or from a folder on your disk. The examples use a small one, Qwen/Qwen2.5-1.5B-Instruct, so a laptop can run it.
A harness is software that puts a benchmark to a model with the same prompts, grading rule and conditions every time. We use lm-evaluation-harness, published by EleutherAI, a research group. Its command is lm_eval.
Why a quick test of your own gives a number nobody can compare
The same model on the same benchmark can get different scores from different tools. Hugging Face compared three implementations of MMLU, a multiple-choice test: lm-evaluation-harness, HELM (an evaluation framework from Stanford) and the original MMLU code.
A model called LLaMA-65B scored 0.488 (48.8 % right) in the harness version of January 2023, 0.637 in HELM and 0.636 in the original.
Hugging Face tied such gaps to small details, such as the exact prompt and the tokenization (how text is cut into pieces, called tokens, for the model). The gaps even changed the ranking of models.
So a score belongs to a setup as well as to a model, and a harness lets you write the setup down; Steps 1 to 4 do that. Our explainer on why the same model gets different scores goes through the causes.
Step 1: Install the harness with the Hugging Face extra
Create a virtual environment first (a separate folder of Python packages for this project), then run:
pip install "lm_eval[hf]"
The part in square brackets is an extra, an optional bundle of add-on packages. The hf extra brings the packages that run Hugging Face models; a bare pip install lm-eval leaves them out. To check the install, run lm_eval ls tasks. It prints the tests the harness knows.
Step 2: Check the setup on a few questions first
Before a long run, ask a few questions and read what the model was asked and what it answered. Finding a wrong setup after two minutes costs less than after two hours. Here is the command we ran, without the three settings Step 3 adds:
lm_eval --model hf \
--model_args pretrained=Qwen/Qwen2.5-1.5B-Instruct \
--tasks mmlu --device mps --batch_size 4 \
--limit 2 --log_samples --output_path results
--model hfpicks the backend, the code that loads and runs your model.hfstands for Hugging Face.pretrained=...names the model: a Hugging Face name, orpretrained=/path/to/modelfor a folder on your disk.--tasks mmlupicks the test, a task: one benchmark the harness knows by name. MMLU has questions from 57 subjects, andmmluruns all of them as subtasks and combines them.--device mpssays where to run:mpsis the graphics chip (GPU) of an Apple silicon Mac,cuda(the default) is an NVIDIA GPU,cpuis the processor.--batch_size 4sends four questions to the model at once. The default is 1.--limit 2keeps only the first 2 questions of each subject, 114 questions in all.--log_samples --output_path resultssaves every question and answer in the folderresults.
This is a test run, not a full benchmark. --limit 2 takes 114 of the 14,042 questions in the full test, and the docs mark it “For testing only”. Its score is not an MMLU score for the model, so do not report it as one.
Every score from our own test comes from such a run: enough to compare settings on one machine.
When the run ends, the harness prints a table. The row acc is the accuracy as a fraction: ours printed 0.614, which is 70 right answers out of 114. The ± number is the standard error, the harness’s estimate of how much the score would move with a different sample of questions: ours was ±0.0447, about 4.5 points.
To get each answer, the harness takes a log-likelihood for each of the four choices, a number that says how likely the model finds that choice, and picks the likeliest.
Now open the folder results. In the subfolder named after the model, results_<time>.json holds the scores and the settings of the run, including the harness version. Each samples_mmlu_<subject>_<time>.jsonl file has one line per question: the prompt the model saw, the right answer and a log-likelihood per choice.
Check that the prompt holds the question and its choices; if it does not, check the task name and the chat format in your command before the long run. The harness developers’ paper advises reviewing a small batch like this before testing at scale.
Step 3: Run the full benchmark
Drop --limit and pin three more things:
lm_eval --model hf \
--model_args pretrained=Qwen/Qwen2.5-1.5B-Instruct,revision=989aa7980e4cf806f80c7fef2b1adb7bc71aa306,dtype=float16 \
--tasks mmlu --num_fewshot 0 \
--device mps --batch_size 4 \
--output_path results
revision=...fixes the exact version of the model files, so a later update cannot change your result. Without it the harness takesmain, the newest. Copy the hash, the code that names a version, from the model’s repository on Hugging Face. The results file records it.dtype=float16sets the number format of the model’s weights, the numbers it learned in training. Float16 stores each in 16 bits, half of float32’s 32: less memory, less exact. Withoutdtypethe harness usesauto, the model’s own setting.--num_fewshot 0puts no solved example questions (few-shot examples) in the prompt before the real question. Each task has its own default, so we set it ourselves.
This command runs the instruct model, one trained to follow instructions, without the chat format: the markers that show who is speaking in a chat (Step 4 says when to add them). That is a valid choice for one number, as long as you write it down.
Our batch-4 test runs used exactly this command with --limit 2 added. We never ran it without --limit, so we have no full MMLU score. In our reading, a full run prints the same table as Step 2, with acc over all the questions.
A full run has 123 times as many questions as our 114, so it takes much longer than the about two minutes each of our 114-question runs took. Add --use_cache <path> (a name for the cache file) and an interrupted run skips the questions it has finished.
That is a complete first benchmark. If you only wanted a number for one model, you can stop here. The rest is for numbers you will compare or publish.
Step 4: Write down what the number depends on
A score means something only next to the settings that produced it. A change in some of them can move it, so write them all next to the number.
| Setting | In plain words | What to write down |
|---|---|---|
| Harness version | Which release of lm-evaluation-harness ran. | The version. The results file saves it. |
| Model and revision | The exact model files. | The pretrained and revision values. |
| Test and examples | The task, and how many solved example questions the prompt holds. | --tasks and --num_fewshot. |
| Chat format | Whether the prompt is wrapped in markers that show who is speaking. | With or without --apply_chat_template. |
| Batch size, device, number format | How the model is run on your machine. | --batch_size, --device and dtype. |
| Question limit | Whether --limit cut the test short. | None for a number you report. |
| Which score | The accuracy measure the task reports. | acc or acc_norm. |
Harness version. The v0.4.13 release notes list fixes “that may shift previously reported numbers”. In one, test questions leaked into the few-shot examples, so the model saw part of the test in its prompt.
Chat format. Instruct models are trained on text in a chat template (markers for who speaks and where a turn ends); the harness adds none by default.
The docs of Hugging Face’s Open LLM Leaderboard, a public ranking of open models retired in March 2025, said to add --apply_chat_template. It changes the prompt, so it can change the score, by an amount we did not test; our test ran without it.
Which score. Some tasks report two numbers, acc and acc_norm. The harness ranks choices by a log-likelihood that adds up over the tokens of each choice, so it can favour short choices. acc_norm adjusts each choice’s score for its length, so a long right answer is not ranked down for being long. mmlu prints only acc, so for MMLU you record acc.
Accuracy also hides which mistakes the model makes. If one kind matters more, a confusion matrix, a table of the right answers against the answers the model gave, counts each kind, and precision, recall and F1, three scores built from that table, summarise them.
A high score can also mean the model saw the test during training: benchmark contamination. The harness’s own decontamination tool, which looks for test questions inside the training text, needs files built from the training data, which you rarely have for a downloaded model. So treat a high score on a famous test as a claim to check.
Our test: batch size and processor did not change the score
The docs name two traps in the settings of Step 3. Hugging Face’s docs say results vary slightly with batch size, because shorter questions get filler (padding) so a batch has equal length.
The README warns that the Apple graphics backend is early-stage, so correctness issues may exist, and suggests checking that cpu and mps agree. We tested both on 3 November 2026 with the 114-question run.
| Item | Value |
|---|---|
| Harness | lm-evaluation-harness 0.4.13, Hugging Face backend |
| Model | Qwen/Qwen2.5-1.5B-Instruct, revision 989aa7980e4cf806f80c7fef2b1adb7bc71aa306 |
| Hardware | MacBook Air, Apple M3, macOS 15.7.9, PyTorch 2.15.0, Python 3.12.12, transformers 5.22.1 |
| Task | mmlu, --num_fewshot 0, the first 2 test questions of each of the 57 subjects (--limit 2) = 114 questions |
We compared four setups. Batch 1, 4 and 16 ran on the Apple graphics chip with float16. The processor setup used the cpu, float32 and batch 4, so it changes device and number format together.
| Setup | Correct (of 114) | Score | Same in every run |
|---|---|---|---|
Batch 1 (Apple GPU, mps) | 70 | 61.4 % | yes (10/10) |
Batch 4 (Apple GPU, mps) | 70 | 61.4 % | yes (10/10) |
Batch 16 (Apple GPU, mps) | 70 | 61.4 % | yes (10/10) |
Processor (cpu, float32, batch 4) | 70 | 61.4 % | yes (3/3) |
The scores are the same, but the numbers underneath moved a little: 114 questions with four choices give 456 log-likelihoods.
A change of 0.01 means the model finds that choice about 1 % more or less likely. We compared the first run of every setup with the first run of batch 4, the setting of the Step 2 command; these first runs were the only ones that saved every question and answer.
- Batch 1: every question was still right or wrong as before. 33 of the 456 log-likelihoods (about 7 %) moved by more than 0.01.
- Batch 16: every question stayed as before, and none of the 456 moved by more than 0.01 (the largest change was 0.0000005).
- Processor: all 456 changed a little, and 136 of them (about 30 %) moved by more than 0.01. For one question the model picked a different choice, but it was wrong both times.
So the numbers underneath moved, as the docs say they might, but the score did not: on this model and these questions no shift turned a wrong answer right or the reverse.
That does not prove batch size never changes a score: a question the model barely decides could flip, and on 114 questions one flip moves the score 0.9 points.
Each setup gave one score in all its runs (33 in all), so on this machine the result repeats.
The limits: one model, 114 questions, one machine. The 61.4 % is the score of a test run and not a full MMLU score. We did not test the chat template, other backends or other models.
Mistakes that make a benchmark number untrustworthy
| Mistake | Why it fails | Fix |
|---|---|---|
| You compare your score with one from a paper or a leaderboard. | The other run may use another number of examples, chat format or task version. | Run the other model yourself, with your command. |
| You leave device and batch size unwritten. | The defaults (cuda, batch size 1) may not match your machine, and results vary slightly with batch size. | Pass both and write them next to the score. |
| You run an instruct model without deciding on the chat template. | The harness warns only if the model arguments contain “inst” or “chat”; in all 33 of our runs it printed that the model “appears to be an instruct or chat variant but chat template is not applied”, and the run went on. | Decide with or without, and write it down. |
| You pick the higher of two close scores. | On a small test, one question moves the score by about a point. | As a rule of thumb, treat a gap smaller than the standard error (about 4.5 points in ours) as too close to call. |
Compare two models by changing only the model name
To choose between two local models, or between a model and its compressed version (a version of it that takes less memory), run the same command twice and change pretrained and its revision. Keep everything else identical. Then compare the two acc rows against their standard errors.
Other ways to run a local model
Steps 1 to 4 use the Hugging Face backend. Any other backend is one more setting to write down.
- vLLM is a separate program for running models fast:
pip install "lm_eval[vllm]", then--model vllm. The README says it “occasionally differs in output from Huggingface”, so run a small slice on both. - A local server that copies OpenAI’s API works with
--model local-completionsandbase_urlset to its/v1/completionsaddress. Chat endpoints cannot rank answer choices: the harness code says they do not give the log-likelihoods of the prompt text, so tests like MMLU need a completion endpoint, one that continues a text prompt. - GGUF files, a single-file model format, load through the Hugging Face backend with
gguf_file=. Addtokenizer=too (it points to the tokenizer, the part that cuts text into tokens), or the load can take hours or hang.
The v0.4.13 docs name neither Ollama nor LM Studio, and we did not test them.
If you need another harness
HELM, Stanford’s evaluation framework, measures more than accuracy, such as efficiency and bias. It entered maintenance mode on 1 June 2026: no new features and no new leaderboard evaluations. That page lists lm-evaluation-harness as an alternative.
OpenCompass is another evaluation platform. It supports local open-source models and API models. For a local model’s accuracy, start with lm-evaluation-harness.
If MMLU is not the test you need, see our guide to benchmarking on MMLU-Pro, GPQA and SWE-bench, three other tests.
A prompt that writes your benchmark command and record
Use this prompt in Claude Code or Cursor. Fill in the brackets.
I want to benchmark the accuracy of a local LLM with lm-evaluation-harness.
MY SETUP:
- Model: [Hugging Face name or local folder, plus the revision if it is on the Hub]
- Computer: [e.g. MacBook with Apple M3]
- Question: [e.g. does the compressed version lose accuracy against the original?]
- Tests: [task names from `lm_eval ls tasks`, e.g. mmlu]
- Instruct (chat) model? [yes / no]
WRITE:
1. The install command, with the extra for my backend.
2. A small check command with --limit and --log_samples, so I can read what the model was asked and answered. State that its number is not reportable.
3. The full command without --limit: model_args with revision and dtype, an explicit --num_fewshot, --device and --batch_size, and --output_path.
4. A results record: harness version, model and revision, tests, number of examples, chat template yes or no, batch size, device, number format, which score, standard error.
RULES:
- Check every flag against `lm_eval run --help`. Do not invent flags.
- If my model is an instruct model, ask me whether to use --apply_chat_template and explain it.
Cheat sheet
The article on one page: open it full size, save it or share it.

AI-assisted content, human-reviewed. Images AI-generated. Editorial Standards · Our Editors