The Attention Revolution

Opening Question

How does a model decide which earlier words matter most when predicting the next one?

Real-World Story

Read this sentence: “The trophy doesn’t fit in the brown suitcase because it is too big.” Now read this one: “The trophy doesn’t fit in the brown suitcase because it is too small.” Only one word changed — “big” became “small” — but “it” now refers to something different in each sentence. In the first, “it” is the trophy. In the second, “it” is the suitcase.

Nothing about the word “it” itself changed. What changed is which earlier word in the sentence you leaned on to resolve it — and you did this instantly, without noticing you’d done anything at all. To understand “it,” you didn’t process the sentence strictly left to right, giving every word equal weight. You reached back and gave “big” or “small” outsized importance, almost ignoring words like “the” and “in” entirely. This selective reaching-back, weighing some earlier words far more than others depending on the specific case at hand, is exactly the ability a model needs — and exactly what was missing from every idea covered so far in this book.

Worked Example

Trace what has to happen, mechanically, once a model has read all the way through “big”: “The trophy doesn’t fit in the brown suitcase because it is too big.” At this point, every earlier token — “it,” “trophy,” “suitcase,” and “big” itself — is available to weigh against each other. Continuing the passage from here (predicting whatever comes next, or answering a question about what “it” means), the model can weigh “trophy” heavily and “suitcase” lightly, because “big” — now sitting in the available context — makes that the sensible reading. Swap “big” for “small” instead, and that same position now weighs “suitcase” heavily instead — one word changes, and the entire weighting shifts accordingly, computed fresh each time.

Here’s the detail worth catching: the model could not have resolved this any earlier. While it was still processing the token “it” itself — before “big” or “small” had appeared at all — nothing yet distinguished this sentence from its opposite twin; both were still fully ambiguous at that exact point in the sequence. A model generating text one token at a time can only weigh itself against tokens that already came before it — never ones that haven’t appeared yet. “It” gets resolved only in hindsight, once later tokens are available to attend back across everything, including “it,” that came before them.

Note what does not happen here: the model doesn’t reach back and rewrite whatever representation it originally computed at the position of “it.” That position’s calculation is done, and done for good, the moment the model moves on. What actually happens is that a later position — “big,” or anything generated after it — builds its own new representation, one that draws on “it,” “trophy,” and “suitcase” as available earlier context and, in doing so, effectively resolves the reference at that later point. The resolution moves forward with the sequence; it never travels backward into it.

Core Intuition

Attention is a mechanism that lets a model, while processing any given token, weigh every earlier token in the sequence — itself included — by how relevant it is right now, never looking ahead to tokens that haven’t been generated yet — a direct, computable version of Chapter 4’s “context”: instead of just saying “surrounding tokens disambiguate meaning,” attention is the machinery that actually computes, numerically, how much each earlier token matters for this particular word at this particular moment. This book, being about generative models like the ones behind ChatGPT, focuses specifically on this causal form of attention; other forms (bidirectional attention, or attention across two different sequences) exist and are used elsewhere, but aren’t this chapter’s concern.

That “earlier, never later” restriction has a name: causal masking. It isn’t a limitation of what attention could technically do — comparing a token against ones that come after it is no harder, mechanically, than comparing it against ones that come before. It’s a deliberate rule, enforced specifically because next-token prediction (Chapter 6) requires it: if a model’s calculation at a given position could see the very token it’s about to predict, or anything generated after it, prediction would collapse into copying an answer already sitting in view, not a genuine forecast from what’s actually known so far.

Positional encoding solves a different problem attention creates on its own. Attention weighs tokens by relevance, treating the sentence as a collection of tokens influencing each other — but by itself, that process has no built-in sense of order. “The dog bit the man” and “the man bit the dog” contain the exact same tokens; only their arrangement differs. Positional encoding tags each token with information about its position in the sequence, so the model can tell not just which tokens are present, but where each one sits relative to the others.

Technical Explanation

Formally, for each token being processed, attention computes three things: a query (what this token is looking for), and, for every token in the sequence, a key (what that token has to offer) and a value (the actual content it contributes if selected). “Looking for,” “has to offer,” and “contributes” are a useful way to talk about what these three quantities do, not literal roles written into the numbers — each is really just a learned numerical vector, produced by a simple transformation of the token’s representation, that happens to behave this way once trained. In the first transformer block, these are derived from the token representations that began as embeddings (Chapter 5); in every later block, they’re derived instead from the richer representations the previous block already produced (Chapter 12 covers this stacking) — not repeatedly from the original embedding. Comparing the current token’s query against its own key and every earlier token’s key produces a relevance score for each pair — comparisons against later tokens are blocked outright, assigned no weight regardless of how relevant they might otherwise seem. This blocking is the causal mask: applied uniformly across the whole sequence, so every position’s calculation can still be performed in parallel during training, while guaranteeing that no token’s result ever actually depends on information from tokens generated after it. The remaining, unblocked scores are converted into weights that sum to one; the token’s updated representation is a weighted combination of its own value and every earlier token’s value, using those weights — heavily favoring highly relevant tokens, barely including irrelevant ones.

This is precisely the revision process Chapter 5 flagged but deferred: a token’s embedding is a general-purpose starting point, and attention is the mechanism that revises it into a context-specific representation, different for “bank” in “river bank” than for “bank” in “investment bank.” A transformer typically runs several of these query/key/value comparisons side by side for the same sequence, called attention heads — each one free to specialize in a different kind of relationship (one head might specialize in exactly the pronoun-resolution pattern from this chapter’s story; another might track which words modify which; another, something else entirely, discovered through training, not assigned by a designer). Their results are combined before moving on to the next stage of processing (Chapter 12) — so a single layer isn’t computing one relevance pattern, but several complementary ones at once.

Positional encoding works by attaching a position-dependent numerical signature to each token’s representation before this process runs, so that the query/key/value comparison can distinguish “the token two positions back” from “the token ten positions back,” not just recognize which words are present. The original transformer design generated this signature from a fixed mathematical pattern; later designs use other schemes, including position information that’s learned during training or encoded directly as part of how attention compares tokens rather than as a separate signature — the specific method varies across models and has evolved since 2017, but the underlying need (order has to be supplied somehow, since attention itself carries none) hasn’t changed.

Common Misconceptions

“Attention means the model reads one word at a time, left to right, like a person reading a sentence.”

Why it’s wrong: Attention lets a model weigh every earlier token against every other earlier token simultaneously — it isn’t a sequential scan, it’s a parallel weighing computed in one step, just restricted to tokens already available (see the next misconception).

Correct intuition: Every token can draw on every earlier token’s information in one step, with weights that shift depending on the specific sequence, not a fixed left-to-right reading order.

Analogy: A conductor doesn’t process an orchestra one instrument at a time in sequence — they attend to the whole ensemble at once, adjusting emphasis on different sections based on what the whole piece needs at that moment.

“Since attention lets a model weigh every token’s relevance, it can look at the entire sentence, including words that come later — the same way a human reading the whole sentence at once can.”

Why it’s wrong: For a generative model, attention is deliberately restricted, by the causal mask, to the current token and everything before it — never anything after. This isn’t a memory limitation; it’s a rule enforced because next-token prediction (Chapter 6) requires it.

Correct intuition: Attention is comprehensive across everything available so far, not comprehensive across the whole eventual passage — the Winograd sentence in this chapter’s story is only resolved in hindsight, once “big” or “small” has actually appeared.

Analogy: A detective solving a case in real time can only use clues discovered so far, not clues that will turn up tomorrow — even though the eventual case file will contain the whole story.

“Positional encoding gives the model an understanding of grammar and word order, the way a person learns syntax rules.”

Why it’s wrong: Positional encoding is just a numerical tag marking where a token sits in the sequence — it doesn’t encode any grammatical rule directly; whatever use the model makes of position (including anything resembling grammar) has to be learned during training (Chapter 9), the same way everything else is.

Correct intuition: Positional encoding supplies raw positional information; any grammatical pattern built from that information is learned, not given.

Analogy: Numbering the pages of a shuffled manuscript tells you the correct order — it doesn’t, by itself, tell you what the story means.

Practical Implications

Understanding attention explains a genuinely important practical fact: because standard attention computes a relevance score between every pair of earlier-and-current tokens, processing a full document or prompt at once costs much more than the sequence length itself would suggest — doubling the length of a document being processed roughly quadruples the attention computation for that pass. (Some newer architectures restructure this to reduce the cost, but the standard version described here is still the dominant baseline.) This is a real reason very long documents are expensive to process and why “context window” size (Chapter 16) is a meaningful engineering constraint, not an arbitrary limit. It also explains “attention visualization” tools some AI researchers use to inspect which earlier words a model weighted heavily when producing a given output.

Key Takeaway

Attention lets a model weigh its own and every earlier token’s relevance when interpreting or predicting from any given token — never later ones, by design — and positional encoding is what lets it also know where each token sits in the sequence.

What to Remember

Attention computes, for each token, a weight reflecting how relevant it itself and every earlier token is to it right now — a direct, numerical version of Chapter 4’s “context.”
This weighing happens in parallel across the whole sequence during training, not as a sequential left-to-right scan — but it is restricted to earlier tokens only.
Causal masking is what enforces “earlier only, never later” — a deliberate rule, not a technical limitation, required for next-token prediction to be meaningful.
Attention heads run several relevance computations side by side per layer, each free to specialize in a different kind of relationship.
Positional encoding tags each token with its position, since attention alone has no built-in sense of word order; the exact method varies across models.
Query, key, and value are the three per-token quantities attention compares and combines to produce a weighted, context-aware representation.
This is the mechanism Chapter 5 deferred: it’s what revises a token’s starting embedding into a context-specific one.
Standard attention’s cost grows faster than sequence length, which is a real, practical reason context windows (Chapter 16) have size limits.

The Next Obvious Question

Once a model can weigh relevance across an entire passage, how do you actually assemble that mechanism into a full working system that can be trained end to end?