# Do you still need RAG when models have huge context and memory?

Date: 2026-03-29 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/need-rag-models-have-huge-context-memory/

**Answer:** For a small collection, often not: if everything fits comfortably in the context window, pasting it all in is simpler and can work well. For a large library, yes: thousands of documents rarely fit, every question would pay to resend them, and models still find facts less reliably in very long inputs than in short ones, so retrieving the few relevant passages remains cheaper and usually more accurate.

**RAG**, short for retrieval-augmented generation, is a way of answering questions from your own documents. Instead of giving the model everything, your system first searches the documents, picks the few passages most likely to help, and adds only those to the prompt. The model then writes its answer from that small, relevant selection.

RAG became popular when models could read only a few thousand words at a time. Now the **context window**, the amount of text a model can take in at once, has grown enormously: GPT-5.4 accepts up to a million tokens through the API, and chat apps add **memory** features that recall details from earlier conversations. So it is fair to ask whether the search step still earns its place. The honest answer depends on how much material you have.

## When "just paste it in" works

Anthropic, whose Claude models support long contexts, gives a plain rule of thumb in its guide to contextual retrieval: if your knowledge base is under about 200,000 tokens, roughly 500 pages, you can often put all of it in the prompt and skip RAG. **Prompt caching**, which lets a provider reuse an identical block of input across requests at a lower price, makes that cheaper and faster. A product manual, a policy handbook or a semester's course notes often fall in this range.

Memory is a different tool. It helps an assistant recall details about you and your earlier chats, which is useful, but it is no substitute for searching a library of documents.

## A worked example: a 2,000-document library

Imagine an organization with 2,000 internal documents, such as reports, procedures and meeting notes, averaging an illustrative 1,500 tokens each. That is about 3 million tokens. Staff ask about 500 questions a month. Here is how the two approaches compare, using GPT-5.4's list price of $2.50 per million input tokens and $15 per million output tokens, with about 500 tokens of output per answer.

| | Long context only | RAG |
|---|---|---|
| Does the library fit? | No: 3M tokens exceeds the 1M window, so you must still choose what to include | Yes: the search index holds everything |
| Input per question | Up to about 900,000 tokens (a filled window) | About 8,000 tokens (ten retrieved passages of 800 tokens) |
| Input cost per question | About $2.25, or about $0.23 at the cached-input price of $0.25 per million | About $0.02 |
| Monthly cost, 500 questions, with output | About $1,129, or about $116 if fully cached | About $14 |
| Reliability of finding the fact | Falls as inputs get longer | Depends on search quality; measurable and improvable |

These figures are calculated from list prices and illustrative sizes, and they leave out the one-time cost of building the search index. Caching helps the long-context column a great deal, but only if the same huge block is sent unchanged each time, and the library still doesn't fit.

## What happens to accuracy in very long inputs

Fitting text into the window is not the same as using it well. OpenAI's own published long-context results for GPT-5.4 show the pattern. On its MRCR test, which asks the model to find several matching items hidden in a long input, GPT-5.4 scored 97.3% on inputs of 4,000 to 8,000 tokens, 79.3% at 128,000 to 256,000 tokens, and 36.6% at 512,000 tokens to one million. On another test, Graphwalks, its score fell to 21.4% in the longest range.

Earlier research found a related weakness. The 2023 paper "Lost in the Middle" showed that models did best when the useful information sat near the start or end of a long input and worst when it was buried in the middle, even for models built for long contexts.

RAG has its own failure mode: the search can miss the right passage, and then the model never sees it. But that failure can be measured and reduced. Anthropic reported that adding a short explanatory context to each chunk and combining meaning-based search with keyword search cut the share of relevant chunks missing from the top 20 results from 5.7% to 2.9%, and to 1.9% with a reranking step.

> **Try it:** Pick one long document you know well and write five questions whose answers sit in different places, including the middle. Ask a chatbot each question with the whole document pasted in, then again with only the relevant page pasted in. Note which answers change.

## When is retrieval still the better choice?

Retrieval is the better choice when your material is larger than a comfortable context window, which a library of thousands of documents almost always is. It also wins when you answer many questions a day, because each question then pays for a few thousand tokens instead of hundreds of thousands, and when documents change often, since updating a search index is quicker than rebuilding a giant prompt.

It is also the better choice when you need to show where an answer came from. A RAG system knows which passages it retrieved, so it can cite them, and you can check whether a wrong answer came from bad search or bad reading. That makes problems easier to fix.

Long context is the simpler choice for small, stable collections and for tasks that need the whole picture at once, such as summarizing one long report or comparing two contracts end to end. Many teams combine the two: retrieval narrows the library to the 20 or 30 most relevant documents, and a long context window lets the model read those in full rather than in small fragments.

## Sources

- Introducing GPT-5.4: https://openai.com/index/introducing-gpt-5-4/
- Introducing Contextual Retrieval: https://www.anthropic.com/news/contextual-retrieval
- Lost in the Middle: How Language Models Use Long Contexts: https://arxiv.org/abs/2307.03172v3