Why does GPT-3 lose track in long documents?

Paste a long report into the GPT-3 Playground and ask for a summary, and one of two things tends to happen. Either the request is refused because it is too long, or you trim it until it fits and the summary quietly ignores everything you cut. The model has not forgotten anything. It never saw that text in the first place.

The reason is the context window: the maximum amount of text the model can take in during a single request. GPT-3 has no memory between requests and no way to scroll back. Whatever sits inside the window is its whole world for that moment.

How big is the window?

The window is measured in tokens, the word pieces the model reads. OpenAI's rule of thumb is that one token is about four characters of English, or three quarters of a word, so 100 tokens come to about 75 words and 1,500 words fill roughly 2,048 tokens.

That figure, 2,048 tokens, is the limit for most GPT-3 engines. OpenAI's documentation lists text-davinci-002, the newest and most capable model, as handling up to 4,000 tokens in a single request, about twice as much. Two details catch people out:

• The limit covers the prompt and the answer together. If your prompt uses 3,700 tokens on text-davinci-002, only about 300 remain for the reply.
• Tokens are not words. Languages other than English often need more tokens for the same amount of text. OpenAI notes that a short Spanish phrase can take one token for every two characters.

Some quick arithmetic shows the problem. A 30-page report at roughly 500 words per page is about 15,000 words, or around 20,000 tokens. That is five times the largest window available, before the model writes a single word.

Why "losing track" happens even when text fits

Because the window is shared, a prompt that only just fits causes its own trouble. The model writes until it reaches either its stopping point or the token ceiling, so a long document with a short allowance for the answer produces a reply that stops mid-sentence. And if you trim the document to make room, the model works only from the part you kept, with no hint that anything is missing. A good habit is to keep each request well under the ceiling and to decide in advance which parts of a document the model really needs.

A worked workflow: summarizing a 30-page report

The reliable answer is to stop asking the model to hold the whole document and give it one manageable piece at a time. OpenAI's own researchers used this approach last year to summarize entire novels: they split each book into sections, summarized each section, then summarized those summaries until one short account of the whole book remained.

Here is the same idea at desk scale:

1. Split the report at its headings into chunks of about 1,500 to 2,000 words each, which gives you around a dozen pieces.
2. Summarize each chunk with the same prompt, so the results are consistent.
3. Collect the chunk summaries in order in one document.
4. Summarize the summaries in a final request, which now fits easily.

The prompt for step 2 might look like this:

Below is section 4 of 12 from a report on regional public transit.
Summarize it in 5 bullet points for a busy city councillor. Keep every number and date exactly as written.

Section text:
"""
[paste section 4 here]
"""

Summary:

Telling the model which section it is reading, and who the summary is for, keeps the pieces aligned. Asking it to keep numbers exactly as written makes the summaries much easier to check against the source later.

How do you split a long text into chunks?

Split along the document's own structure whenever you can. Chapter breaks, headings and blank lines between sections are natural seams, and the novel-summarizing researchers aimed for exactly those boundaries. Cutting in the middle of a paragraph separates a claim from its explanation, and the model will summarize each half without the other.

Size each chunk to leave room. On a 4,000-token model, a chunk of about 2,000 words (roughly 2,700 tokens) plus your instruction still leaves space for a full summary. On a 2,048-token engine, aim for about 1,000 words. Each step should shrink the text a lot; the OpenAI book study aimed to cut it by five to ten times at every level.

Finally, give each chunk a little context. A section read on its own can be misunderstood. The book study gives a memorable example: read without the rest of the chapter, a passage in Pride and Prejudice in which a character is asked for her hand was summarized as a marriage proposal, when the full chapter makes clear the request was for a dance. The researchers eased this by passing the earlier summaries along with each new section. You can do the same by starting each prompt with two or three sentences that recap what came before.