What is a one-million-token context window good for?

A model's context window is how much text it can hold in view at once: your instructions, any documents you paste in, the conversation so far and its own reply. It's measured in tokens, the small pieces of text (or of images, audio and code) that a model actually processes. Last week Google announced Gemini 1.5 Pro, with a standard window of 128,000 tokens. Some developers and business customers can now try a window of up to one million tokens in a private preview. For comparison, Anthropic's Claude 2.1 holds 200,000 tokens, about 500 pages.

What fits in a million tokens

Google gives these rough equivalents for one million tokens:

Google's demonstrations show what that size makes possible: asking questions about the 402-page transcripts of the Apollo 11 mission, picking out plot points and small details in a 44-minute silent Buster Keaton film, and reasoning over more than 100,000 lines of code. In research, Google tested windows as large as ten million tokens.

The practical gain is that you no longer have to cut your material into pieces and decide in advance which pieces matter. Good uses include:

• Contracts and reports: compare every clause across a stack of agreements, or find where two policy documents contradict each other.
• Codebases: ask how a feature works when the answer spans dozens of files.
• Recordings: search hours of meetings or lectures for the moment a topic came up.
• Learning from a manual: Google reports that, given a grammar book for Kalamang, which fewer than 200 people speak, the model learned to translate English into it about as well as a person studying the same book.

A worked "needle in a haystack" test

The standard way to check whether a model really uses its whole window is the needle-in-a-haystack test: hide one out-of-place sentence somewhere in a very long document and ask the model to find it. Google reports that Gemini 1.5 Pro located the planted text in 99% of trials, even with a million tokens around it.

You can run a small version yourself with any long-context model you have access to:

1. Take a long public-domain text, such as a novel from Project Gutenberg.
2. Insert one sentence that doesn't belong, about two-thirds of the way through: "The secret ingredient in Grandma Lucie's soup is smoked paprika."
3. Paste the whole text with this question:

Below is a long document. Answer the question using only the document. If you find the answer, quote the sentence it comes from.

<document>
[paste the full text here]
</document>

Question: What is the secret ingredient in Grandma Lucie's soup?

1. Repeat with the sentence near the start, the middle and the end, and note where, if anywhere, it fails.

How you ask matters. When an independent tester ran this kind of test on Claude 2.1, the model often replied that the document didn't say, because the hidden sentence looked out of place. Anthropic found that starting Claude's answer with "Here is the most relevant sentence in the context:" raised its score on that test from 27% to 98%. Asking for a quote, as in the prompt above, nudges models in the same direction.

Does a long context replace retrieval?

Not for most ongoing work, though it changes when you need it. Retrieval-augmented generation, or RAG, is the approach described in a 2020 paper by Lewis and colleagues: instead of giving the model everything, a search step pulls the few passages most relevant to each question from a large collection and passes only those to the model. The paper's authors pointed out that models store facts in their parameters but struggle to access them precisely, to show where an answer came from, or to update what they know. Retrieval addresses all three.

A long window shifts the balance in a few ways. For a single, self-contained set of documents that you'll question deeply once, such as one due-diligence file or one codebase, loading everything avoids the risk that the search step misses the passage you needed. Setup is simpler too: no index to build.

But cost and speed favour retrieval for repeated use. Models process every token you send, every time you ask. Sending a million tokens to answer each question means reading a library to look up one fact. Google itself tells early testers to expect longer waits with the one-million-token window, says it is still working to cut latency and computing needs, and plans pricing tiers that rise from the 128,000-token standard. Retrieval also scales past any window: a company wiki or years of email can be far larger than a million tokens, and it changes daily.

The likely pattern is a mix. Use retrieval to narrow a huge or changing collection down to the relevant documents, then use a long context to read those documents whole rather than in fragments. And whichever you use, ask for quotes so you can check the answer against the source.