# How do models handle very long context today?

Date: 2026-06-12 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/models-handle-very-long-context-today/

**Answer:** Models read very long inputs by making attention cheaper to compute, not by changing what attention does. Techniques such as FlashAttention reorganize the work so it fits the GPU's fast memory, and Ring Attention spreads one long input across many chips, which is how context windows reached millions of tokens; recall still needs testing, because information in the middle of a long input is easier to miss.

A model's **context** is everything it can see at once: your instructions, the conversation so far, and any documents you paste in, all measured in **tokens**, the small chunks of text models read. Benchmarks in the 2022 FlashAttention paper still used inputs of 512 or 1,000 tokens. Research systems now handle millions, enough for a stack of contracts or an entire codebase in one request.

That growth did not come from a new kind of model. It came from engineering work that made the same core operation, attention, fit on real hardware. Understanding that work helps you judge what a big context window can and cannot do for you.

## Why long inputs are hard

In a Transformer, the architecture behind today's language models, **self-attention** lets every token look at every other token to decide what matters. That is what lets a model connect a pronoun on page 40 to a name on page 2. The cost is that the number of comparisons grows with the square of the input length. The FlashAttention paper puts it plainly: time and memory for self-attention are quadratic in sequence length.

A quick table shows what "quadratic" means. The figures are simple arithmetic, not measurements of any model.

| Input length (tokens) | Pairs of tokens to compare |
|---|---|
| 1,000 | 1 million |
| 10,000 | 100 million |
| 100,000 | 10 billion |
| 1,000,000 | 1 trillion |

Ten times more text means a hundred times more pairs. Earlier attempts to fix this used **approximate attention**, which skips some comparisons. The FlashAttention authors note that these methods gave up some quality and often did not run faster in practice.

## Technique 1: compute exact attention more cleverly

**FlashAttention**, published in 2022 by Tri Dao and colleagues, kept attention exact and attacked a different bottleneck: moving data. A GPU has a large but relatively slow main memory and a tiny, very fast memory on the chip itself. Standard attention shuffles large tables between the two again and again. FlashAttention splits the work into **tiles**, small blocks that fit in the fast memory, and finishes as much as possible there before writing anything back.

Think of a cook who walks to the pantry for each ingredient versus one who brings everything for one dish to the counter first. The recipe is the same; the walking is not. The paper reports that this made training several times faster on some tasks, for example about three times faster for GPT-2 with 1,000-token inputs, and that the longer context it allowed improved model quality.

## Technique 2: spread one input across many chips

Even efficient attention runs out of memory on one chip. **Ring Attention**, posted in 2023 by Hao Liu, Matei Zaharia and Pieter Abbeel, splits a long sequence into blocks and gives each block to a different device. The devices sit in a ring. Each one computes attention for its own block while passing pieces of its data, the **keys and values** that other tokens compare against, to its neighbor, so that sending and computing happen at the same time.

The result is that the maximum input grows with the number of devices. The authors report working with context sizes of millions of tokens, without approximating attention. Combined with tiled computation, techniques like this explain how a context window can grow far beyond what one GPU holds.

## A worked recall test

A large window tells you how much a model *accepts*, not how well it *uses* it. The 2023 study "Lost in the Middle" by Nelson Liu and colleagues found that models answered best when the needed information sat near the start or the end of the input, and worst when it was in the middle, even for models built for long contexts.

You can check this on the model you use. Take a long document you know well, such as a 60-page report, and hide three small, made-up facts in it: one on page 2, one on page 30, one on page 58. Then send:

```prompt
Answer from the document only. If the answer is not in the document, say so.
1. What colour is the delivery van mentioned in the report?
2. What is the name of the volunteer coordinator?
3. On what day of the week is the warehouse closed?
For each answer, quote the sentence you used.
```

Record the results in a grid like this one. The entries below are illustrative, to show the format.

| Fact position | Found? | Quote correct? |
|---|---|---|
| Start (page 2) | Yes | Yes |
| Middle (page 30) | No | — |
| End (page 58) | Yes | Yes |

Repeat with the facts moved to different places. Asking for a quote matters: it lets you see whether the model really found the sentence or guessed.

> **Try it:** Run the three-fact test twice on the same model, once with a 10-page document and once with a 60-page one. Note whether the middle fact survives the longer version.

## What are the trade-offs of long context?

**Cost and speed.** You pay for every token you send, in every request. Even with efficient attention, a very long input takes longer to process, so a 500-page upload is slower and more expensive than the three pages that actually answer your question.

**Uneven attention.** As the "Lost in the Middle" study showed, more context does not mean equal attention to all of it. A detail buried in the middle of a huge input is the one most likely to be missed, and the model may not tell you it missed it.

**Dilution.** Irrelevant material competes with the parts that matter. A focused input often produces a sharper answer than a complete one.

In practice, long context works best for tasks that genuinely need the whole picture, such as finding contradictions across a long contract or tracing how a function is used across a codebase. For questions with a specific answer, it is usually better to search first and send only the relevant passages, then ask the model to quote its evidence so you can check it.

## Sources

- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness: https://arxiv.org/abs/2205.14135v2
- Ring Attention with Blockwise Transformers for Near-Infinite Context: https://arxiv.org/abs/2310.01889
- Lost in the Middle: How Language Models Use Long Contexts: https://arxiv.org/abs/2307.03172v3