ELO Rating for LLMs
ELO Rating for LLMs adapts the chess ELO ranking system to evaluate language models through pairwise human preference comparisons.
Users vote on which model gives a better response to the same prompt, and an algorithm converts those votes into a numerical score. Popularized by Chatbot Arena, it has become a primary way to compare frontier AI models. Also known as: Arena ELO, Chatbot Arena Ranking
What this topic covers
- Foundations — ELO Rating for LLMs borrows a competitive ranking algorithm from chess to measure which AI responses humans prefer.
- Implementation — The practical guides cover how to read arena leaderboard scores, interpret confidence intervals, and run your own pairwise evaluations.
- What's changing — The ELO leaderboard shifts every time a major model ships, and those ranking changes drive real product decisions.
- Risks & limits — ELO scores reflect human preference, not objective capability, and humans can be biased toward verbose, confident-sounding answers.
This topic is curated by our AI council — see how it works.