Compressing Language

Opening Question

Why can a short sentence carry so much meaning, and how do computers exploit that?

Real-World Story

In the era of the telegram, you paid by the word. This produced a strange, clipped dialect: “ARRIVE TUESDAY STOP NEED CAR STOP” instead of “I will be arriving on Tuesday and I’ll need you to arrange a car for me.” Nobody taught telegram writers information theory, but they were all doing the same thing intuitively: cutting every word whose absence the reader could easily reconstruct from what remained. “I will be” is obvious from context; cutting it costs you nothing. “Tuesday” is the one piece of the message you truly cannot guess in advance; that word has to stay.

This is exactly the same instinct behind modern text abbreviations like “lol” or “brb” — they work only because both sides already share enough context to fill in the missing pieces. Compression isn’t about making a message shorter for its own sake. It’s about identifying which parts of a message are predictable — and therefore cheap to omit or shrink — and which parts genuinely carry new information and must be preserved in full.

Worked Example

Text messages compress the same way telegrams did. “I will be right back” becomes “brb” — “I” and “will” vanish entirely, while “be,” “right,” and “back” each survive as a single letter, because any reader can reconstruct the full phrase instantly from context and convention, even from that much compression.

Now compare a message where the same trick would fail: “brb, the store called about a recall on the blue sedan we bought.” Nobody compresses “recall on the blue sedan” the same way “I will be right back” got compressed to “brb” — because that clause is exactly the part the reader has no way to predict in advance. The predictable half of the sentence still shrinks; the unpredictable half doesn’t, and can’t.

Core Intuition

Compression is the general idea of representing information using fewer symbols by exploiting redundancy — the parts of a message that are predictable given what’s already been said or what’s commonly known. This directly follows from Chapter 2’s idea of information as “surprise”: predictable content carries little information, so it takes few symbols to represent; unpredictable content carries a lot of information and needs more.

That general idea shows up in two different, easily confused forms. A lossless code — like Morse’s single dot for “E” — never actually throws anything away: every letter is still there, just represented with fewer pulses when it’s common, and the original message can always be reconstructed exactly. Ellipsis like “brb” works differently: words are genuinely omitted, not just shortened, and recovering them depends on the reader correctly guessing what was left out from shared context and convention — a real gamble, not a guarantee, which is exactly why the “recall on the blue sedan” half of the earlier example can’t be compressed the same way. Both are compression in the loose, everyday sense of “exploiting predictability to say more with less,” but only the first is compression in the strict, fully-recoverable sense the rest of this chapter’s technical discussion depends on.

Context is the surrounding material that makes a piece of language predictable — or that resolves which of several possible meanings a word has. The word “bank” means something completely different in “I sat by the river bank” versus “I deposited a check at the bank.” Nothing about the word “bank” itself tells you which meaning is intended — only the tokens around it do. Context is what turns an ambiguous token into a specific meaning.

A second example makes the same point from a different angle: “this concert was sick” and “this flu is making me sick” use the identical word to mean nearly opposite things — one enthusiastic praise, one literal illness. No dictionary entry for “sick” resolves that on its own; only “concert” versus “flu,” sitting a few tokens away, does.

These two ideas are deeply connected, in the strict, lossless sense above: a system that assigns accurate probabilities to what comes next, given context, can in principle use those probabilities to build a reversible code that represents likely continuations with fewer symbols, on average, than unlikely ones — “predictable” and “compressible” are the same property viewed from two different angles, under a shared probability model, not a claim that predictable content can simply be deleted.

Technical Explanation

Formally, compression schemes exploit statistical redundancy in a sequence: symbols or sequences that occur frequently (and are therefore predictable) are assigned short representations, while rare, unpredictable ones are assigned longer representations — this is precisely the same principle behind Morse code’s letter lengths in Chapter 2, and behind the subword-merging tokenizer in Chapter 3. Tokenization exploits this same redundancy in a compression-like way: common multi-character sequences get represented as single reusable tokens, cutting down the number of steps the model has to process — though a tokenizer isn’t a complete information-theoretic compression scheme on its own; the vocabulary size, token-ID encoding, and sequence length all still matter for how much a given text actually shrinks.

This same logic is what lets a good language model act as a compressor in the strict sense too, not just a metaphorical one: since it assigns a probability to every possible next token, that probability can be turned into a reversible code — an approach called arithmetic coding — that represents likely continuations in fewer bits than unlikely ones, on average. The better the model’s predictions, the shorter the resulting code. That’s the precise version of “prediction and compression are the same thing”: a claim about codes built from probabilities, not a claim that a system is free to silently drop whatever it finds predictable.

Context, formally, is the span of surrounding tokens a system considers when interpreting or predicting a given token. A model doesn’t process “bank” in isolation — it processes the entire surrounding sequence, and uses that sequence to narrow down which of “bank”’s possible meanings (or which of many possible next tokens) is most probable in this particular instance. The richer and more relevant the available context, the sharper this narrowing becomes — a theme that will resurface, in a much more consequential form, when we cover context windows and memory in Part IV.

Common Misconceptions

“Compressing language means losing quality or meaning, like a blurry, low-resolution photo.”

Why it’s wrong: Lossless compression — Morse code, or the tokenizer from Chapter 3 — never actually removes anything; it represents predictable content more compactly, and the original is always fully recoverable. Ellipsis like “brb” is a different, lossier move: words really are dropped, and recovering them depends on the reader guessing right from context.

Correct intuition: Good lossless compression represents redundant, predictable content more compactly without discarding it. That exact recoverability is what distinguishes it from an editing choice like ellipsis, where content is genuinely gone unless the reader can reconstruct it correctly.

Analogy: Morse code’s short dot for “E” doesn’t erase any information about “E” — it’s still perfectly recoverable. Removing “I will be” from “I will be arriving Tuesday” is a different move: those words are actually gone, recoverable only because both sides share enough convention to guess them back correctly.

“Context just means ‘the general topic’ being discussed.”

Why it’s wrong: Context, in this technical sense, is the specific sequence of surrounding tokens — not a vague subject label — and it can resolve very fine-grained ambiguity, not just topic.

Correct intuition: Context is the exact material immediately around a token that a model actually conditions its interpretation on.

Analogy: Knowing the “topic” of a conversation is a river — but knowing the exact previous sentence is what tells you whether “bank” means the water’s edge or the building down the street.

Practical Implications

Once you understand compression-as-redundancy-removal, engineering discussions about “efficient tokenizers” and “context compression” techniques stop sounding like marketing buzzwords and start sounding like the same simple idea applied at different scales. And once you understand that context is what disambiguates meaning, you’ll immediately see why giving an AI system too little surrounding context produces vague or wrong answers — it isn’t being lazy, it genuinely lacks the disambiguating material a human would have used.

Key Takeaway

Context is the span of surrounding tokens a model uses to resolve ambiguity — and the same predictability that resolves meaning is what makes language compressible.

What to Remember

Compression exploits redundancy: predictable content is cheap to represent; unpredictable content must be preserved in full.
This is the same principle behind Morse code letter-lengths and behind subword tokenization — both compress by favoring common patterns.
Context is the specific surrounding sequence of tokens that disambiguates meaning, not a vague notion of “topic.”
A word like “bank” has no fixed meaning on its own — context selects among its possible meanings.
Predictability and compressibility are the same property in the strict, lossless sense: a system that predicts well can, in principle, use those probabilities to build a genuinely reversible code.
Insufficient context produces vague or wrong outputs because the disambiguating material simply isn’t available — not because of laziness.

The Next Obvious Question

If meaning depends so heavily on context and surrounding tokens, how can a computer represent “meaning” itself in a form it can actually compute with?