Scaling Laws

Opening Question

Is there a predictable relationship between how big a model is and how good it becomes — and does scaling ever stop paying off?

Real-World Story

Chapter 1 described three ingredients converging: more data, more compute, and a new architecture. What it didn’t explain is something researchers found genuinely surprising once they started deliberately training models of many different sizes and comparing results: if you plot a model’s loss against the amount of compute used to train it — with both axes marked off in multiples of ten rather than in fixed steps — the result isn’t a jagged, unpredictable scatter. It’s close to a straight line.

That’s a remarkable thing to discover about something as messy as language. It means you can train a series of small, cheap models, plot their results, and extrapolate the line forward to forecast — with useful accuracy, within a given model family and training setup — how good a model ten or a hundred times larger will be, before ever training it. Predictability at this scale, in a field this young, was not something researchers expected to find. “Within a given model family and training setup” is doing real work in that sentence: change the architecture, the tokenizer, the quality or composition of the training data, the training objective, or the optimizer, and the fitted curve doesn’t automatically carry over — each new combination needs its own small-scale check before its extrapolation can be trusted.

Worked Example

Train four models of increasing size — say, roughly ten times more compute each time — and plot their loss. The jump from the smallest to the second model might cut loss by a certain proportion; the jump from the second to the third cuts it by a similar proportion; and so on down the line.

Researchers use exactly this consistency to forecast forward: if the pattern holds across three data points, a fourth, far larger and far more expensive model can be predicted before it’s ever trained. This general approach — extrapolating from smaller, cheaper runs — is one of the key inputs AI labs use when deciding whether a massive training run is likely to be worth its cost, alongside cost, timeline, and risk, before committing hundreds of millions of dollars to find out.

Core Intuition

A scaling law, in this context, is an empirical finding: as you increase training data, parameters, and compute together, a language model’s loss (Chapter 9) decreases in a smooth, forecastable way — not a guarantee for any single architecture forever, but a remarkably consistent pattern that has held across many model families and years of research.

This predictability is exactly why AI labs invest so heavily in scale: it isn’t blind faith that “bigger is better” — it’s a measured, repeatable relationship, verified on cheaper, smaller models before committing to the enormous expense of training a very large one. It’s also why researchers discovered that data and parameters need to be scaled together in the right balance: a model with far more parameters than its training data can support ends up undertrained relative to its own size — bigger, but not correspondingly better, because it never saw enough examples to make good use of all those parameters.

This diminishing-returns shape is much older than machine learning. Economists have described exactly this curve for centuries: adding fertilizer to a field increases crop yield sharply at first, then less and less with each additional bag, until at some point adding more stops helping at all. Scaling a language model follows the same general pattern that shows up whenever you keep adding more of one input to a process that has other, non-scaling constraints in the background.

Technical Explanation

Scaling laws describe loss decreasing roughly as a power-law function of compute, parameters, and data — meaning each additional order of magnitude of scale buys a real, but shrinking, improvement in loss. The near-straight line described in the opening story is what a power-law relationship looks like once both axes are stretched into equal ratios (marked off ×10, ×100, ×1000, rather than +10, +20, +30) — it’s a way of revealing that underlying relationship, not a claim that loss falls in a straight line on an ordinary chart. Researchers also typically separate loss into a part that keeps shrinking with scale and a remaining irreducible component that scale alone doesn’t remove (sometimes called irreducible loss, and not necessarily small — its size depends on the inherent unpredictability of the data itself) — the power-law description applies to the shrinking part.

This relationship was first characterized clearly around 2020, and refined in 2022 by research (widely known by the name of the model that demonstrated it, “Chinchilla”) showing that many earlier large models had been undertrained: they had more parameters than their training data justified, and a smaller model trained on proportionally more data would have performed better for the same compute budget. This refined the industry’s understanding from “just make it bigger” to “grow parameters and data together, in the right ratio.”

It’s important to be precise about what scaling laws promise and don’t. They predict loss — performance on the specific next-token-prediction objective from Chapter 9 — not every downstream capability a reader might care about, and not without limit. Returns diminish: each further order of magnitude of scale costs dramatically more compute for a progressively smaller loss improvement, and practical ceilings exist too — the amount of high-quality training text available, the cost of the compute itself, and the energy required to produce it.

Common Misconceptions

“Scaling laws mean you can scale forever with no real limit.”

Why it’s wrong: Returns diminish predictably — each further order-of-magnitude increase in scale buys a progressively smaller improvement — and practical ceilings exist in available high-quality data, compute cost, and energy.

Correct intuition: Scaling laws describe a reliable but diminishing-returns curve, not an unlimited runway.

Analogy: Training for a marathon reliably improves your time the more you train — but the tenth month of training buys you far less improvement than the first, and eventually further training yields almost nothing extra.

“A bigger model is automatically the better choice for any given use case.”

Why it’s wrong: Scaling laws predict lower loss on the training objective, not that a bigger model is the right cost, speed, or capability tradeoff for every task — a smaller, well-matched model is often the better engineering choice.

Correct intuition: Treat “bigger, following the scaling curve” as one axis of a decision, not the whole decision — Part IV’s chapter on quantization and efficient models covers the other side of this tradeoff.

Analogy: A commercial airliner is a better choice than a small plane for a transatlantic flight, and a much worse choice for a quick hop between two nearby small towns — bigger isn’t better in some absolute sense, it’s better for a particular job.

Practical Implications

This is why AI lab announcements so often lead with parameter counts and training compute figures — they’re citing exactly the quantities scaling laws relate to loss. It also gives you a genuinely useful skeptical question to ask of any “bigger model” headline: is this claiming a scaling-law-predicted improvement in the underlying objective, or a claim about some other, separately-earned capability? The two are related but not identical, and conflating them is one of the more common ways AI announcements oversell.

One more scoping note: this chapter’s scaling laws are specifically about pretraining — data, parameters, and compute spent before a model is ever deployed. “Scaling” as an industry term has since grown to cover other axes too: spending more computation at inference time on a hard problem (Chapter 23), additional post-training after the base model is done (Chapter 19), and giving a model more relevant information to work with through retrieval (Chapter 18) all also improve results in a scale-like way, without being the specific pretraining relationship described here.

Key Takeaway

Scaling laws let researchers predict a large model’s loss from smaller experiments — a real, reliable, but diminishing-returns pattern, not an unlimited or unconditional guarantee.

What to Remember

Scaling laws are an empirical finding: loss falls in a smooth, forecastable way as data, parameters, and compute increase together — visible as a straight line once both axes are stretched into equal ratios (×10, ×100, ×1000), which is what reveals the underlying power-law relationship.
A remaining irreducible component of loss (“irreducible loss,” not necessarily small) doesn’t shrink with scale alone — the power-law description applies to the part that does.
This predictability lets researchers extrapolate from small, cheap experiments to justify training much larger, more expensive models.
Data and parameters must be scaled together in the right ratio — a model can be undertrained if it has more parameters than its data justifies.
Returns diminish with scale, and practical ceilings exist: available high-quality data, compute cost, and energy.
Scaling laws predict loss on the training objective specifically — not every downstream capability, and not an unconditional case for “bigger is always better.”

The Next Obvious Question

Now that a model can learn enormous numbers of parameters and improve predictably with scale, how does it actually decide which earlier words matter most when predicting the next one?