Retrieval-Augmented Generation

Opening Question

Once relevant documents can be retrieved this way, how does a model actually combine that with its own generation process to produce a grounded answer?

Real-World Story

Remember Chapter 14’s exam analogy: years of studying (training) followed by a closed-book exam (inference), where the student can only apply what they’ve already learned, with nothing to consult in the moment. Now imagine a different kind of exam — open-book — where the student is handed the exact relevant pages moments before answering, and explicitly told: base your answer on these specific pages.

The student’s general training hasn’t changed at all. What’s changed is that, for this specific question, they’re no longer relying purely on memory — they have the actual source material in front of them, right when they need it. That’s the entire idea behind giving a model retrieved documents before it answers.

Worked Example

A customer-support chatbot is asked about a company’s return policy. Before generating any response, the system first retrieves (Chapter 17) the two or three passages from the company’s actual policy documents most relevant to the question — often using semantic search over a vector database, though keyword search, a direct database query, or even a web search could fill the same role. Those retrieved passages are then inserted directly into the model’s context window (Chapter 16), placed right alongside the user’s question.

Only now does the model generate its answer — with the actual, current policy text sitting directly in its input, not just whatever general sense of “typical return policies” it picked up during training. If the policy changed last week, the retrieved passage reflects that change, even though the model’s trained parameters know nothing about it.

Core Intuition

RAG (Retrieval-Augmented Generation) is a system design that combines retrieval (Chapter 17) with generation (Chapter 6): before generating a response, first retrieve passages relevant to the query from some external source of information, insert them into the context window (Chapter 16), and only then let the model generate its answer — now with that specific material directly available to draw on, rather than relying solely on whatever got baked into its parameters during training. Chapter 17’s semantic search over a vector database is the most common way to do the retrieving, but it’s an implementation choice, not part of RAG’s definition — the defining idea is generation conditioned on retrieved external information, whatever mechanism finds that information.

Technical Explanation

Worth knowing if you read the original research: the term “RAG” was first used for a more specific architecture, where the retriever and the generator were trained together as one system. What this chapter describes — retrieving passages and simply inserting them into an already-trained model’s context window, no training involved — is a broader, later usage of the same term, one that became the common, everyday pattern once models developed strong long-context reading ability. That broader usage is what “RAG” means in most engineering conversation today, and it’s this chapter’s focus, but it’s worth knowing the name originally pointed at something narrower.

In the simplest and most common deployment pattern today, RAG requires no update to the model’s parameters (Chapter 8) at query time — it’s entirely a matter of what gets placed in the context window before generation happens, performed fresh at every single query. (More elaborate RAG systems sometimes go further and separately fine-tune the retriever or the generator itself — closer to the original, jointly-trained sense of the term — but that’s an addition on top of the pattern described here, not a requirement of it.) This is precisely why the simplest form of RAG can incorporate information published after a model’s training cutoff, or private and proprietary documents the model never trained on at all — none of that requires retraining anything; it only requires the information being retrievable and insertable into the window at the moment it’s needed.

This also marks RAG’s real limits. It’s bounded by whatever is actually in the retrieval store (if the answer isn’t in any retrievable document, retrieval can’t surface it) and by the context window’s finite capacity (retrieving too many or too irrelevant passages crowds the window and can confuse rather than help). RAG substantially reduces, but does not eliminate, the hallucination risk from Chapter 15: the model can still misread or contradict retrieved text, blend it incorrectly with prior trained-in knowledge, or produce a fluent answer when the retrieval step itself failed to find the genuinely relevant passage. And retrieval succeeding at its own job — finding a passage that’s genuinely relevant — doesn’t mean the passage itself is correct: a retrieved document can be outdated, wrong, or simply unreliable, independent of whether the model goes on to use it faithfully.

Common Misconceptions

“RAG means the model has been retrained on the retrieved documents.”

Why it’s wrong: In the standard, simplest deployment pattern, no training or parameter update happens anywhere in this process — it’s entirely a context-window technique, performed freshly at each individual query, exactly like handing a student pages for one specific exam rather than sending them back to school. (More elaborate setups can add retriever or generator fine-tuning on top, but that’s an addition to the core pattern, not what makes something RAG.)

Correct intuition: RAG’s defining move changes what the model can see for this one answer, not what it permanently knows.

Analogy: Handing someone a reference sheet during a test doesn’t require re-teaching them the whole subject — the reference sheet is temporary, per-question help.

“RAG eliminates hallucination entirely, since the model now has real sources to work from.”

Why it’s wrong: The model can still misinterpret retrieved text, blend it incorrectly with unrelated trained-in knowledge, or hallucinate confidently when the retrieval step itself fails to find the genuinely relevant passage — grounding reduces the risk substantially, but doesn’t remove it.

Correct intuition: RAG narrows the gap between “plausible” and “true” (Chapter 15) by giving the model real material to draw from, but doesn’t guarantee the model uses that material correctly every time.

Analogy: An open-book exam reduces guessing, but a student can still misread the reference page or cite the wrong passage.

Practical Implications

This is exactly why companies rely heavily on RAG for internal knowledge-base assistants: updating the underlying documents is far cheaper and faster than retraining a model every time something changes. It also gives you a concrete, practical test for evaluating a “grounded AI” product’s claims: does it actually retrieve and cite specific real sources for its answers, or does it just generate fluent-sounding text without anything underneath it — the exact distinction this chapter and Chapter 15 together equip you to spot.

A real production RAG system carries its own checklist beyond getting retrieval and generation working at all: source provenance, so a citation can actually be traced back to where it came from; documents going stale, so the retrieval store itself needs a real update process, not just a one-time load; an index being deliberately poisoned with malicious or misleading content by whoever can write to it; access controls on the retrieval store itself, so a user can’t retrieve documents they aren’t supposed to see (a distinct concern from the model’s own behavior); indirect prompt injection carried in via retrieved content — untrusted text a retrieved document contains being treated as an instruction rather than as data (Chapter 28 covers this in depth); and evaluating retrieval quality — did the right passage even get found? — as its own separate question from evaluating whether the generated answer used it well.

Key Takeaway

RAG grounds a model’s answer in passages retrieved from an external source and placed directly in its context window at query time — expanding what it can draw on, in the simplest and most common pattern without changing a single trained parameter.

What to Remember

RAG combines retrieval (Chapter 17) with generation (Chapter 6): retrieve relevant passages from some external source first, then generate an answer using them.
Semantic search over a vector database is the most common retrieval mechanism, but RAG’s definition doesn’t require it — keyword search, direct database queries, or web search can fill the same role.
Like an open-book exam, in the simplest and most common pattern the model’s underlying training doesn’t change — it’s simply given specific material to work from for this one query.
RAG can incorporate information published after training, or private documents the model never saw, without any retraining.
RAG is bounded by what’s in the retrieval store and by the context window’s capacity, and it substantially reduces but doesn’t eliminate hallucination.
Updating a RAG system’s knowledge typically means updating its documents, not retraining the model — a major practical advantage.

The Next Obvious Question

If a model’s knowledge and behavior can be shaped after pretraining without retrieval, purely through additional training, how does that actually work — and how does a model get pointed toward being safe and aligned with the goals its designers set for it?