From Transformer to ChatGPT

Opening Question

The transformer architecture existed since 2017 — why did it take years before something like ChatGPT appeared?

Real-World Story

In the earliest days of using large pretrained models through raw developer tools, people learned a strange trick: if you wanted a good answer to a question, you didn’t just ask the question. You wrote “Q: What is the capital of France?” and then “A:” and let the model continue from there. Asked plainly, the same underlying model might just as easily continue your question with more questions — “What is the capital of Germany? What is the capital of Italy?” — because, statistically, that’s a perfectly plausible way for a list of quiz questions to continue.

The model wasn’t malfunctioning. It was doing exactly what Chapter 6 described: predicting the most statistically likely continuation given its training text. If quiz-style lists were common in that training text, continuing the list was a perfectly good prediction. Getting a direct, helpful answer instead of a plausible continuation of the pattern required something more than a well-trained, well-scaled transformer — it required deliberately teaching the model to prefer being helpful over merely being plausible.

Worked Example

Compare two ways a model might respond to “What’s the capital of France?” A model trained only on next-token prediction (Chapters 6 and 9), with no further adjustment, might continue with something like “What’s the capital of Germany? What’s the capital of Italy?” — a highly plausible continuation if its training data contained many such lists, even though it never actually answers the question. A model that has additionally been shaped to behave like a helpful assistant responds simply: “Paris.”

Both models share the identical underlying architecture (Chapters 11–12) and the identical core training objective (Chapter 9). The difference is entirely in what happened after that core training: one model was further adjusted, specifically and deliberately, to prefer direct, helpful answers over other statistically plausible continuations.

Core Intuition

Three more ingredients turn a raw, pretrained transformer into something like ChatGPT — none of which change the underlying architecture from Chapters 11–12.

Fine-tuning (previewed here, formally covered in Chapter 19): further training, using the same predict/measure-error/adjust loop from Chapter 9, on a smaller, deliberately curated set of examples showing the desired style of behavior — direct answers, instructions followed, questions addressed rather than continued.

Human feedback (previewed here, formally covered in Chapter 19): humans compare multiple candidate responses and indicate which they prefer; that preference signal is used to further adjust the model toward producing the kinds of responses people actually rate highly, not just the ones that are statistically unremarkable continuations of internet text.

Inference-time choices (Chapter 14): exactly how the model picks a specific token from its predicted probabilities also shapes how a response feels to use, independent of training.

These stages usually preserve the same basic transformer architecture and build on the same underlying knowledge the base model already has. But — as Chapter 19 covers in full — “usually preserve” isn’t “never change”: additional training can shift more than surface style, sometimes touching the model’s factual associations and specific capabilities too, not just how it’s been pointed toward using what it already knew.

These three ingredients explain the model itself — the specific gap this chapter’s story and worked example are pointing at. A deployed product like ChatGPT layers still more on top: a system prompt setting its behavior for that specific product, safety and moderation checks, sometimes retrieval or tool access (Chapters 17–18, 21). None of that changes the underlying model this chapter describes; it’s additional scaffolding built around it.

Technical Explanation

Precisely: pretraining is the massive, generic phase covered since Chapter 9 — learning to predict the next token across enormous, broad swaths of text, with no specific behavioral goal beyond raw predictive accuracy. Fine-tuning is additional training performed afterward, on a much smaller, deliberately chosen dataset, using the same broad gradient-based optimization machinery as Chapter 9 but now aimed at a specific target style of behavior rather than generic text continuation — though, as the next paragraph and Chapter 19 cover in full, what actually counts as “error” during this stage can itself look quite different from Chapter 9’s original loop.

A widely used technique for the human-feedback step is Reinforcement Learning from Human Feedback (RLHF): humans rank pairs of candidate model outputs by preference; that ranking data trains a separate model — a “reward model” — to predict which of two responses a human would prefer. The original language model is then further adjusted using reinforcement learning: it generates responses, the reward model scores them, and the language model is nudged toward whatever tends to earn higher scores — usually while being kept close to a reference version of itself, so it doesn’t drift into degenerate behavior that chases the reward signal without actually being helpful. That’s a meaningfully different training procedure from Chapter 9’s original loop, not just the same loop with a different number plugged in for “error.” (RLHF isn’t the only way to use preference data — Chapter 19 covers a newer alternative that skips the separate reward model entirely.) The same basic architecture usually keeps doing the predicting throughout this process — though some fine-tuning methods add small extra components rather than adjusting the whole thing, which Chapter 19 also covers — and what changes most directly, across all of this, is the training signal being optimized against.

Common Misconceptions

“ChatGPT is just a base model like GPT-3, exposed directly through a chat interface.”

Why it’s wrong: Substantial additional training — fine-tuning and human-feedback adjustment — happens after pretraining specifically to shape the raw model’s behavior into something helpful and conversational; the interface alone doesn’t produce that behavior.

Correct intuition: A “chat” or “instruct” model is a pretrained model that has been through additional, deliberate behavioral training — not the same artifact as the raw pretrained model wearing a different interface.

Analogy: A brilliant graduate with encyclopedic knowledge still needs coaching before becoming a good teacher — the knowledge alone doesn’t guarantee the specific, helpful communication style.

“The 2017 transformer paper alone explains why chatbots suddenly got good.”

Why it’s wrong: As Chapter 1 established, architecture, data, and compute together produced a powerful base model — but the specific, helpful, conversational behavior people associate with modern chatbots required this additional fine-tuning and human-feedback stage on top of that base.

Correct intuition: The architecture made a powerful predictor possible; alignment techniques are what made that predictor feel like a helpful assistant.

Analogy: A powerful engine doesn’t make a car street-legal or comfortable to drive — it still needs the rest of the car built around it.

Practical Implications

This is why AI providers distinguish “base model” from “chat” or “instruct” versions of what is, underneath, the same pretrained architecture — and why a base model, used directly, can behave in surprising, less-helpful ways compared to its fine-tuned counterpart. It’s also worth knowing that this gap between raw pretrained behavior and carefully trained-in helpfulness is part of what certain adversarial prompting techniques try to exploit — a topic Chapter 28 returns to under security and safety.

Key Takeaway

A helpful chatbot isn’t just a scaled-up transformer — it’s a pretrained model further shaped by fine-tuning and human feedback to prefer responses judged more helpful over merely statistically plausible continuations.

What to Remember

A raw pretrained transformer predicts statistically plausible continuations, which don’t automatically mean helpful or direct answers.
Fine-tuning reuses Chapter 9’s training loop on a smaller, curated dataset showing the desired style of behavior.
Human feedback (e.g. RLHF) trains a separate reward model on human preferences, then uses that reward model to further adjust the language model.
These stages usually preserve the same basic transformer architecture (Chapters 11–12), but they can substantially reshape behavior and may also alter particular capabilities, factual associations, or trainable components — not merely redirect knowledge that stays otherwise untouched.
“Base model” versus “chat”/“instruct” model is exactly this distinction: same architecture, different amount of post-pretraining shaping.

The Next Obvious Question

Once a model has all this training baked in, what actually happens, step by step, when it generates a response to your prompt?