# Why do AI models miss things in the middle of a long document?

Date: 2025-07-16 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/ai-models-miss-things-middle-long-document/

**Answer:** Language models do not read a long input evenly; research known as "Lost in the Middle" found they use information best when it sits near the start or the end, and noticeably worse when it is buried in the middle. You can see the effect yourself with a simple placement test, and you can reduce it by where you put your instructions and how you ask.

Models now accept very long inputs: a contract, a stack of meeting notes, a whole report. It is natural to assume that if a document fits in the **context window** (the amount of text a model can take in at once), the model reads every part of it with equal care. It does not. Where a fact sits in the input changes how likely the model is to use it.

The clearest evidence comes from a 2023 paper by Nelson F. Liu and six colleagues, known by the first half of its title, "Lost in the Middle." Its main finding still shapes the prompting advice model makers publish today.

## What the research found

The authors gave models two kinds of job that depend on finding one specific piece of information in a long input. In the first, a model received a question and a set of documents, only one of which held the answer. In the second, it received a long list of keys paired with values and had to return the value for one key. The researchers then moved the useful item around: first position, somewhere in the middle, last position.

Accuracy shifted a lot with that one change. Models tended to do best when the needed information came first or last, and worst when it was in the middle. If you plot accuracy against position, the line dips in the center and rises at both ends, like a shallow U. The dip appeared even in models built specifically for long inputs.

## Why position matters

A model processes text through **attention**, the mechanism that lets each word draw on other words in the input. In principle every token can reach every other one. In practice, how a model learns to spread that attention depends on its training, and nothing guarantees it will be even across a very long span. The paper measured the effect rather than pinning down a single cause.

A loose human comparison helps: after hearing a long list, people also tend to remember the first and last items best. The analogy is imperfect, but it is a useful reminder that "it was in the document" is not the same as "it was used."

Long-context behavior has improved since then. OpenAI, in its April 2025 prompting guide for GPT-4.1, reports strong results on simple retrieval across its full one-million-token window, while noting that performance can still drop when a task needs many items pulled out or careful reasoning across the whole input. So the middle problem is smaller for simple lookups than it once was, but it has not disappeared for real work.

## A worked placement test

You can check how your own model behaves in about fifteen minutes.

1. Take a long document you are allowed to share, or build one by pasting together twenty unrelated memos of a few paragraphs each.
2. Write one invented, checkable fact that appears nowhere else, such as: *"The staff parking code changes to 4417 on Friday."*
3. Make three copies of the document. Insert the fact in the first memo of copy A, the tenth memo of copy B, and the last memo of copy C.
4. In three fresh chats, paste one copy each, followed by the same question:

```prompt
Based only on the document above, what is the new staff parking code, and when does it change? If the document does not say, reply "Not stated."
```

5. Repeat each test two or three times and fill in a table like this one:

| Copy | Where the fact sits | Answer correct? (record yours) |
|---|---|---|
| A | First memo | |
| B | Tenth memo, the middle | |
| C | Last memo | |

To make the test harder, ask a question that needs two facts, one placed early and one in the middle, such as a date in one memo and a room number in another. Tasks that combine pieces tend to show the effect more clearly than single lookups.

> **Try it:** Run the three-copy test with a document from your own work, then repeat it with your question placed before the document instead of after it, and compare the two tables.

## How do you write prompts that work around it?

Put your instructions in two places. OpenAI's GPT-4.1 guide recommends placing instructions both before and after a long block of context, because that performed better in its testing than either position alone. If you only state them once, it found that above the context works better than below. A simple pattern: a short line at the top saying what the task is, the document, then the full question at the end.

Ask the model to find before it answers. Anthropic showed this with Claude 2.1 in December 2023: on a long-document test, simply starting Claude's reply with the line "Here is the most relevant sentence in the context:" raised its score from 27% to 98%. You can do the same in a chat by asking: "First, quote the passages that relate to my question, with their section headings. Then answer using only those quotes." The quoting step pulls the middle material forward, and it gives you something to check.

Reduce the middle. If one part of a document matters most, place it near the top or repeat the key passage at the end. For long reports, work in sections: ask for a short summary of each section first, then ask your real question against those summaries. For a set of files, ask about each file separately and combine the answers yourself.

Finally, verify what matters. Ask where in the document the answer came from, and look at that spot. If the model says "Not stated" for something you know is there, that is your sign to split the document or move the passage, not to trust the first reply.

## Sources

- Lost in the Middle: How Language Models Use Long Contexts: https://arxiv.org/abs/2307.03172v3
- GPT 4.1 Prompting Guide: https://cookbook.openai.com/examples/gpt4-1_prompting_guide
- Long context prompting for Claude 2.1: https://www.anthropic.com/news/claude-2-1-prompting