Evaluating AI Systems

Opening Question

This book has now covered how these systems are built and how they act — reading, predicting, retrieving, reasoning, reaching into the world. But how does anyone actually know, in a rigorous way, whether a given system is good: better than another, safe enough to deploy, actually doing what it’s supposed to?

Real-World Story

A hiring manager screens every programming candidate using one fixed coding-quiz platform. At first, the score genuinely tracks who’s a strong programmer. But once candidates learn that this specific quiz is what gets you hired, they start studying that quiz’s question patterns specifically — not becoming better programmers in general, just better at that one test. A year later, the manager notices something strange: quiz scores are as high as ever, but on-the-job performance hasn’t improved to match. The test didn’t stop working because it was badly designed. It stopped working because it became the target, and once people optimize directly for a measurement, the measurement quietly stops tracking the thing it was built to approximate.

That’s the whole difficulty of evaluating an AI system, in miniature. Any fixed test is a proxy for the capability you actually care about, not the capability itself — and the moment that proxy becomes an explicit target, the gap between “scores well” and “is actually good” can widen without anyone noticing from the score alone.

Worked Example

Two models are compared on a public benchmark of grade-school math word problems. Model A scores 92%; Model B scores 89%. On the surface, Model A looks better at math reasoning. Look closer, though, and several of the benchmark’s exact questions turn out to appear, word for word, in Model A’s training data (Chapter 9) — the benchmark was scraped from a public source before that model’s training cutoff. Model A may not be reasoning through those specific problems at all; it may simply be recalling a memorized answer to a question it has effectively already seen, the training-is-memorization risk Chapter 9 already warned about.

The benchmark score alone can’t distinguish “genuinely better at general math reasoning” from “happened to have this specific test’s answers in its training data.” Only checking for this kind of overlap — known as contamination — shows the score can no longer be trusted as clean evidence of generalization, and once it’s found, the 92%-versus-89% comparison stops meaning what it appeared to mean. And even setting contamination aside, a 3-point gap on its own doesn’t say much without knowing how many questions the benchmark actually contains, how much the score moves across repeated runs, and whether both models were evaluated under identical conditions — a small difference can easily sit inside ordinary run-to-run noise.

Core Intuition

Evaluation is the general problem of measuring whether an AI system — not just the underlying model in isolation, but the model together with whatever retrieval (Chapters 17–18), tools, and orchestration (Chapters 21–22) surround it — actually performs its intended task well under realistic conditions, in a way trustworthy enough to compare versions, catch regressions, and decide whether it’s ready to deploy. A model can score well on its own while the complete system still fails a user, if something surrounding it is broken — retrieval returning the wrong passage, a tool interface behaving unsafely — so evaluation has to reach the whole thing, not stop at the model at its core. “How good is this system” also isn’t one single question — it splits into several distinct ones: raw capability on the task, factual reliability, faithfulness (is an answer actually grounded in its cited source, as distinct from merely being true), calibration (does the model’s stated confidence match its real accuracy), robustness to unusual or adversarial input, fairness across different users, latency and cost, and security, among others. Benchmarks, human judgment, and using another model as a judge are three different tools for approximating “how good is this,” each useful, and each with its own specific blind spots a reader needs to know about before trusting a number produced by any one of them alone.

Technical Explanation

Benchmarks are fixed sets of test questions or tasks, most often with a single known correct answer that can be scored automatically — fast, cheap, and repeatable, which is exactly why they’re used for quick comparison between models. (Some benchmarks instead cover open-ended tasks — summarization, creative writing, preference comparisons — where there’s no single correct answer, and scoring relies on a rubric or comparative judgment instead of automatic matching.) They carry two specific failure modes worth naming directly. The first is contamination: the exact test questions leaking into a model’s training data (Chapter 9), inflating its score without reflecting genuine improvement on the underlying skill. The second is the dynamic the opening story illustrated: once a specific benchmark becomes the explicit target model developers train and tune against, a high score can increasingly reflect narrow, benchmark-specific skill rather than the broader capability the benchmark was originally built to stand in for.

Human evaluation — people directly rating or comparing model outputs — is the same basic mechanism Chapter 19 already introduced for training (human feedback, RLHF), now applied to measurement instead of adjustment. It can track real-world usefulness more faithfully than an automated benchmark, since a person is judging the actual thing you care about rather than a fixed proxy for it — but that depends heavily on the task, the rubric, and rater expertise and consistency, and it’s slow and expensive to run at scale besides.

LLM-as-judge evaluation uses a separate model — often a larger or otherwise trusted one — to score or compare outputs against a rubric, instead of a human doing it directly. It’s dramatically cheaper and faster than exhaustive human evaluation, and — with a strong judge model and a well-specified rubric — its judgments can correlate reasonably well with what human raters would say in many settings. But a judge model inherits its own blind spots: it can hallucinate a justification for a wrong verdict the same way Chapter 15 described for any other generated claim, and it can carry specific, documented biases — favoring whichever answer it happens to see first or second (position bias), shifting its verdict based on small wording changes in the rubric (prompt sensitivity), favoring longer answers regardless of actual quality (verbosity preference), rating outputs from its own model family more favorably (self-preference bias), and being more or less reliable depending on the kind of task being judged — a judge is one more imperfect measurement tool, not a substitute for the thing it’s approximating.

Common Misconceptions

“A high benchmark score means a model is definitely better at real-world use of that skill.”

Why it’s wrong: Contamination and Goodhart’s-law-style overfitting to a benchmark’s specific format can both inflate a score without any corresponding improvement in the broader, real-world capability the benchmark was meant to approximate.

Correct intuition: A benchmark score is a proxy for a capability, not the capability itself — worth checking for contamination and format-specific overfitting before treating a comparison as settled.

Analogy: The hiring manager’s quiz score stopped meaning “good programmer” the moment candidates started optimizing for the quiz specifically, even though the number on the screen looked exactly the same as before.

“Using an LLM to judge another LLM’s outputs is just as trustworthy as having a human expert review it directly.”

Why it’s wrong: A judge model can hallucinate a confident-sounding justification for a wrong verdict, the same generation mechanism covered in Chapter 15, and can be specifically gamed by outputs styled to appeal to its particular known preferences rather than genuinely being better.

Correct intuition: LLM-as-judge is a useful, cheap proxy that often correlates with human judgment at scale — valuable in combination with human evaluation and diverse benchmarks, not a guaranteed, interchangeable substitute for either.

Analogy: A trusted colleague’s quick read of a report is genuinely useful and often right — but it isn’t the same guarantee as a full independent audit, and treating it as one invites exactly the errors an audit exists to catch.

“If a new model version scores higher on the overall benchmark, it’s strictly better at everything the old model was good at.”

Why it’s wrong: An aggregate score averages across many different sub-skills. A genuine regression on one specific capability can be entirely masked by improvement on others, even while the single headline number goes up.

Correct intuition: One aggregate score can hide capability-specific regressions — meaningful evaluation tracks performance across many distinct skills, not just a single combined figure.

Analogy: A student’s overall exam average can rise even while their specific grasp of one topic quietly gets worse, if they improved enough elsewhere to cover for it.

Practical Implications

This is why AI providers publish tables of many separate named benchmarks rather than one single score — no single number is trustworthy enough to stand alone, for exactly the reasons this chapter covered. It’s also why practitioners are consistently urged to build evaluation sets specific to their own actual use case rather than relying solely on general public benchmarks, which may not reflect their domain and may already be contaminated or specifically optimized against by model developers. And it’s the direct reason “LLM-as-judge” pipelines have become a common piece of real product evaluation workflows: not because they’re perfectly reliable, but because they make frequent, cheap evaluation possible at a scale exhaustive human review never could reach.

It’s also why a serious evaluation plan for a real product tests more than the model’s raw answers: whether retrieval (Chapters 17–18) actually surfaces the right passage, whether a tool-using agent (Chapters 21–22) picks the right tool and recovers sensibly when a step fails, and how the complete pipeline performs end to end — not just a benchmark score for the model sitting at its core, which can look excellent while the surrounding system still lets users down.

A reported score is also only as trustworthy as what stands behind it. Worth checking before treating any comparison as settled: how many test cases the score is actually based on (sample size), whether it was measured once or across repeated runs (with non-deterministic decoding, generation is often stochastic, so a single run isn’t necessarily representative — though a deterministic decoding setup run against a fixed harness can reproduce the same result every time), how much it moves run to run (variance), whether that variance is small enough to trust a given gap (an uncertainty estimate, not a gut feeling), whether small changes in prompt wording or output formatting shift the score (prompt sensitivity, a source of variation distinct from the model’s own randomness), whether the evaluation infrastructure itself introduces variation between runs (hardware, batching, and kernel differences can all cause this, independent of decoding settings), how much independent human raters agree with each other on the same cases, and separately, how consistent an LLM judge’s own verdicts are across repeated passes over the same cases (evaluator agreement covers both, but they’re not the same failure mode), and whether a handful of severe failures are being averaged away by many minor successes (failure severity) — an aggregate percentage can look reassuring while hiding exactly the kind of failure that matters most in production.

Key Takeaway

A benchmark score is a proxy for real-world capability, not the capability itself — and the more a score becomes an explicit target to optimize against, the more it risks measuring how well a system games that specific test rather than how good it actually is.

What to Remember

Evaluation is the general problem of measuring whether a complete AI system — the model plus retrieval, tools, and orchestration around it, not just the model in isolation — actually performs well under realistic conditions.
Benchmarks are fast and repeatable but vulnerable to contamination (test questions leaking into training data) and Goodhart’s-law-style overfitting once they become an explicit optimization target.
Human evaluation can track real-world usefulness more faithfully, but that depends on the task, rubric, and rater expertise and consistency — and it’s slow and expensive besides.
LLM-as-judge evaluation is cheap and scalable, correlates reasonably with human judgment, but inherits its own blind spots — including hallucination and susceptibility to being gamed.
No single evaluation method is sufficient alone; each is an imperfect proxy with different specific failure modes.
A single aggregate score can mask capability-specific regressions even while it goes up overall.
Practitioners are urged to build domain-specific evaluation sets rather than relying solely on general public benchmarks.

The Next Obvious Question

Evaluation tells you how a system performs on a fixed set of test cases before deployment. But once a system is actually live, handling real, unpredictable user requests every day, how do you know it’s still working correctly?