Learning From Examples

Opening Question

How does a neural network actually learn the right values for its billions of adjustable parameters?

Real-World Story

Think about learning to throw free throws in basketball. Nobody hands you a formula for the correct arm angle, wrist snap, and knee bend. Instead, you throw, you miss — long, short, left, right — and each miss carries information: “too much force,” “pulled left.” You adjust, slightly, in the direction that seems to correct that specific error, and you throw again. After thousands of repetitions, your form converges on something that reliably works, even though at no point did anyone tell you the exact correct angles. The correction signal was never “here is the right answer” — it was always just “here’s how wrong that one was, and roughly which way to adjust.”

Training a neural network is this same loop, industrialized: try, measure the error precisely, nudge the parameters that contributed to that error a tiny bit in the direction that would have reduced it, and repeat — not thousands of times, but billions.

Worked Example

Imagine training on the same single example five times in a row. First pass: the model’s prediction is wildly off; loss is high; every parameter that contributed to that error gets a relatively large nudge. Second pass, same example: the prediction is a little closer; loss is lower; the nudges shrink accordingly. By the fifth pass, the model’s prediction on this specific example is nearly exact, and further nudges from it are tiny.

Now multiply this by billions of different examples, each contributing its own small pull in its own direction — and these pulls don’t all point the same way. A nudge that helps this specific example can, in the same training run, make a different example slightly worse; loss on any one example doesn’t reliably drop on every single pass through training the way it does in this isolated five-repeat demo. The net result, across enough examples, is a set of parameters that works reasonably well across all of them at once — not perfectly tuned to any single example, but reliably good on the whole enormous collection.

Core Intuition

Loss is a single number that measures how much probability a network’s prediction assigned to the one token that actually came next — high probability on the right answer means low loss (“very close to correct”), low probability on the right answer means high loss (“very wrong”). Loss is what makes the free-throw analogy precise: instead of a vague sense of “too far left,” the network gets a precise, quantified measure of its error on every single example it sees.

Training is the repeated process of showing the network an example, computing its loss, and then adjusting the parameters that contributed to that loss a tiny amount in whichever direction would have reduced it, had it been applied beforehand. This happens not once, but across enormous datasets, over and over.

Learning is the name for the overall outcome of this repeated process: the gradual, emergent discovery of useful patterns — including the very embeddings from Chapter 5 — purely as a side effect of parameters being nudged, over billions of examples, in whatever direction reduces error. Nobody designs the patterns the network ends up using; they emerge because they happen to reduce loss more reliably than the alternatives.

A second grounding, from a completely different domain: adjusting a recipe by taste rather than a fixed formula. You taste the soup (measure the error), decide it needs a bit more salt and a little less acid (the direction and rough size of the adjustment), make the change, and taste again. Repeat enough times, across enough soups, and you converge on something that reliably tastes right — without ever writing down “the” formula for correctness in advance. Training a neural network is this same taste-adjust-repeat loop, automated and run billions of times instead of a few dozen.

Technical Explanation

For language models, training works like this: take a real passage of text, and at every position in it, ask the network to predict the token that comes next — so a single passage produces many predictions at once, one for each position, not just one. The loss at each position compares the network’s predicted probability distribution (from Chapter 6) against the token that actually came next in the real text — a prediction that puts high probability on the correct token produces low loss; a prediction that confidently predicts the wrong token produces high loss.

After computing the loss for an example, a procedure calculates, for every parameter the loss actually depends on, which direction of adjustment would have reduced it, and by how much — for a language model this is typically most of the network’s parameters on any given example, but not literally all of them; in architectures that keep separate input and output embedding tables, an embedding table entry for a token that didn’t appear anywhere in this example, for instance, gets no nudge at all this round. (Some architectures instead tie the input and output embedding tables together, in which case even an unused token’s row can still be nudged through the output side.) Each of those parameters is then nudged a small step in its own improving direction. This whole cycle (predict, measure loss, nudge the affected parameters) is repeated for an enormous number of examples, often the same examples multiple times, until the network’s predictions reliably carry low loss across a huge, varied body of text — not because anyone told it the rules of language, but because parameters that produce low loss on real text were kept and reinforced, over and over, while directions that increased loss were avoided.

Common Misconceptions

“Training makes the model memorize its training examples, like a database of stored sentences.”

Why it’s wrong: The training process specifically rewards parameters that predict well across an enormous, varied set of examples — patterns that generalize tend to reduce loss more reliably than memorized specifics, especially given far more distinct examples than parameters could ever store verbatim.

Correct intuition: A well-trained model has learned generalizable patterns of language, not a searchable copy of its training text — though under some conditions (very repeated text, insufficient data relative to model size) some memorization can still occur, a nuance Chapter 15 returns to.

Analogy: A student who does thousands of practice problems learns the underlying method, not a lookup table of exactly those problems — though a student who saw the same ten problems a thousand times might indeed just memorize those ten.

“A lower loss always means the model has gotten more generally ‘intelligent.’”

Why it’s wrong: Loss measures how well the model fits its specific training objective (next-token prediction on its training data) — improving at that objective correlates with many useful capabilities, but it’s a specific, measurable quantity, not a direct measure of general intelligence.

Correct intuition: Treat loss as a precise fitness score for one specific task, useful and informative, but distinct from the broader, harder question of evaluating a system’s overall capability — which Chapter 26 addresses directly.

Analogy: A student’s score on one specific exam is real, useful information about their preparation for that exam — but it isn’t the same thing as a complete measure of their intelligence.

Practical Implications

This is the machinery behind phrases like “training run,” “loss curve,” and “epoch” that show up constantly in AI engineering writing — a loss curve is literally a plot of this error measurement getting smaller over the course of training, and watching it is one of the most basic ways practitioners monitor whether training is working. It’s also why training a large model requires so much computation: billions of parameters, each needing its own small adjustment, repeated across enormous datasets, again and again.

Key Takeaway

Training is a loop — predict, measure loss, nudge whichever parameters contributed to it — repeated billions of times; learning is the name for whatever pattern emerges from it.

What to Remember

Loss is a number measuring how much probability the model assigned to the token that actually came next.
Training is the repeated loop of predicting, measuring loss, and nudging whichever parameters contributed to it toward reducing it.
Learning is the name for the useful patterns that emerge from this loop — nobody designs them directly.
A language model’s training examples are simply real text, with the model asked to predict the next token at every position and each prediction compared against what actually came next.
Well-trained models generalize rather than memorize, though memorization can occur under specific conditions (Chapter 15).
Loss measures fit to a specific objective, not general intelligence — a distinction Chapter 26 (Evaluating AI Systems) revisits.

The Next Obvious Question

If throwing more examples and more parameters at this training loop keeps improving performance, is there a predictable pattern to how much better it gets — and are there limits?