Your model scores 85% on MMLU. Your competitor scores 82%. You ship yours. Users hate it. Turns out MMLU doesn't measure what your users care about, and you never checked what your model actually does wrong.
This is not a hypothetical. It's the default outcome when teams treat benchmark scores as the goal rather than a proxy for the goal. Evaluation is the part of machine learning that everyone knows matters and almost everyone underinvests in. The reason is structural: getting a benchmark number is fast and cheap. Building rigorous evaluation for your actual task is slow and expensive. So most teams do the fast thing and hope the proxy holds.
Sometimes it does. Often it doesn't. Here's the anatomy of why.
Benchmarks measure what they measure
MMLU (Massive Multitask Language Understanding) is a multiple-choice test covering 57 subjects, from high school chemistry to professional law. It's a useful measure of broad factual knowledge and reasoning. HumanEval is a set of Python programming problems where the model writes code that must pass unit tests. HELM (Holistic Evaluation of Language Models) is a more comprehensive framework covering accuracy, calibration, robustness, and fairness across many tasks.
Each of these tests a specific capability under specific conditions. High scores mean the model is good at that benchmark. They do not mean the model is good at your task.
A model with 85% MMLU might still confidently give wrong medical dosage advice, fail to follow your company's output format, or hallucinate API call signatures it's never seen. The benchmark is a proxy. It captures some signal about general capability. It does not capture whether that capability transfers to your specific distribution of inputs, your edge cases, your failure modes.
When you choose a model based entirely on benchmarks, you're trusting that your task is similar enough to what the benchmarks test that the rankings hold. Sometimes that's reasonable. For commodity tasks like general summarization or basic question answering, it's probably fine. For anything domain-specific, it's a gamble you haven't measured.
The contamination problem
There's a subtler issue: benchmark contamination. If the benchmark's test questions appeared in the model's training data, the model may be partially memorizing answers rather than generalizing. This is not paranoia. It's a documented problem. HumanEval problems, MMLU questions, and other standard benchmarks have all made their way into the text corpora that large models train on.
A model that "solved" HumanEval by seeing those exact problems during training isn't demonstrating coding ability. It's demonstrating a kind of sophisticated recall. It may fail on novel programming problems it hasn't seen before.
Contaminated benchmark scores are worse than useless: they're actively misleading. You make a decision based on a number that overstates real capability. The gap only becomes visible when you test on held-out data the model hasn't seen, which is to say, when you do actual evaluation.
Some model providers try to detect contamination by checking what fraction of benchmark questions appear verbatim in training data. Some don't. Either way, if you're using a benchmark that's been publicly available for a while, assume some contamination exists and weight the scores accordingly.
What rigorous evaluation looks like
The alternative to trusting benchmarks is building evaluation on your actual task distribution. This is harder. It's worth doing.
Start with a hold-out test set sampled from your task: not a public benchmark, but real inputs drawn from the same distribution you care about. If your product generates legal summaries, your test set should be legal documents. If your product answers customer support questions, your test set should be customer support questions. This sounds obvious. It's frequently skipped.
From there, define task-specific metrics. Code should run and produce correct output. Medical answers should be checked against clinical guidelines. Summaries should preserve key facts, which requires specifying what "key facts" means, which is itself a design decision. "LLM output quality" is not a metric. Define what you're actually measuring.
Then slice the evaluation. Overall accuracy hides failure modes in specific subpopulations. A model might perform well on average while failing consistently on questions from one domain, one demographic, or one input structure. If you only look at aggregate numbers, you will miss these. Slice your test set by input length, by topic category, by phrasing style, whatever dimensions matter for your users. Evaluate each slice separately.
Finally, add behavioral testing: adversarial inputs, distribution shift, edge cases, refusal behavior. What happens when the user asks an ambiguous question? A malformed one? A question that should be declined? Behavioral tests are not the same as accuracy tests. Accuracy tests check whether the model gets the right answer on clean inputs. Behavioral tests check whether it handles the full space of realistic ones.
LLM-as-judge
Manual evaluation doesn't scale. Automatically checking whether outputs are correct is often hard or impossible. This has made "LLM-as-judge" a popular technique: use a stronger model to evaluate a weaker model's outputs. It's cheap, it runs at scale, and it's surprisingly effective for many tasks.
It also has systematic biases worth knowing before you trust it.
Longer outputs tend to score higher, independent of quality. Outputs stylistically similar to the judge model score higher; you're measuring stylistic alignment, not quality. Confident-sounding answers score higher than hedged ones, even when the hedged answer is more accurate. These biases can be gamed: if you know your outputs will be judged by GPT-4, you can prompt your model to produce longer, more confident-sounding text and improve your scores without improving actual quality.
The mitigation is calibration. Run LLM-as-judge evaluations on a subset of inputs where you also have human judgments. Measure the correlation. If the LLM judge and humans disagree systematically, figure out why before scaling up the automated evaluation. Use multiple judges. Don't use automated evaluation as your only signal for anything that actually matters.
Human evaluation: the expensive ground truth
Human evaluation is slow, expensive, and variable. It's also necessary for anything that matters. Humans are the users. For most applications, "is this output good" is a human judgment, and automated proxies for that judgment have systematic errors.
To do it without fooling yourself, annotation guidelines need to be specific enough that annotators understand exactly what they're rating and why. A rubric that can be interpreted in multiple ways produces noisy data. Before the full evaluation, run annotators through the same examples and compare their judgments. High disagreement usually means ambiguous guidelines, not genuine difficulty agreeing on quality.
Inter-annotator agreement, formally measured with Cohen's kappa (which accounts for the probability that annotators agree by chance), tells you whether your task is well-specified enough that different humans reliably agree on the answer. Low agreement doesn't necessarily mean the task is subjective. It often means the annotation guidelines aren't specific enough.
Annotator fatigue is real. Quality degrades across a long session. Structure annotation tasks so each session is short, the judgment required is clear, and you can detect flagging quality from sudden increases in speed or unusual agreement patterns.
The red-teaming dimension
Everything described so far evaluates capability: how good is the model at doing the task? But evaluation isn't only about capability. It's also about failure modes. What does the model do wrong, how often, and under what conditions?
Red-teaming means deliberately trying to break the model to surface failures that capability benchmarks don't cover. What happens when inputs are subtly adversarial, when a user phrases a question in a way designed to elicit an incorrect or harmful response? When the input contains a factual premise that's wrong, does the model catch it or build on it? When the model is highly confident, how often is it wrong? A well-calibrated model should be wrong about as often as its confidence implies. Most models are overconfident.
Systematic red-teaming requires people who are actively trying to find problems, not people who are confirming that the system works. The goal is to discover failure modes before users do. The ones you find in red-teaming are the ones you can mitigate. The ones you don't find, you'll discover from production incidents.
Evaluation is the discipline
Skipping rigorous evaluation doesn't mean you don't have quality problems. It means you find out about them from users, who experience them as product failures rather than as data points in your evaluation pipeline.
Benchmarks are useful as quick sanity checks and for rough model comparisons. They are not a substitute for evaluation on your actual task. Building that evaluation, the test set, the metrics, the slicing, the behavioral tests, the human judgments, is the engineering work that determines whether your model is actually good or just appears to be.
The uncomfortable truth is that "we scored 85% on MMLU" is a lot easier to say than "we measured task performance on a representative hold-out set, sliced by input category, with inter-annotator agreement of 0.78." The second sentence is the one that means something.
Back to writing