What comes after the Transformer?

The Transformer, introduced in 2017, is the design behind nearly every well-known chatbot. Its key part, attention, lets every word in a text look at every other word to decide what matters. That is powerful, and it is also expensive: double the length of the input and the work of comparing every pair of words roughly quadruples. A long document, a large codebase or an hour of audio becomes costly to process.

Researchers have spent years looking for designs that keep the Transformer's quality with less cost. As of mid-2026 the answer is not "something completely new" but a set of competing ideas, some already shipping in public models. Here are the three most discussed, what they change, and where they still fall short.

The cost being attacked

When a Transformer generates text, it keeps a KV cache, short for key-value cache: stored notes about every earlier token (a token is a word or part of a word) so it does not recompute them for each new one. That memory grows with every token. The team behind Jamba, a hybrid model described below, gives a concrete figure: Mixtral, a well-known Transformer model, needs about 32 GB for this cache at a context of 256,000 tokens. That is memory a graphics card cannot spend on anything else, and it is one reason long-context answers are slow and expensive.

Direction 1: state-space models

A state-space model (SSM) reads a sequence one step at a time and keeps a fixed-size summary, called the state, that it updates as it goes. Think of a reader who keeps a single page of notes instead of rereading the whole book before every sentence. Because the summary does not grow, the cost rises only in proportion to the length of the input, which researchers call linear scaling.

Earlier SSMs were fast but weaker on language. In December 2023, Albert Gu and Tri Dao published Mamba, which made the model's update rules depend on the token it is currently reading. That lets it decide, word by word, what to write into its notes and what to let fade. The authors report about five times the generation throughput of a Transformer and found that a 3-billion-parameter Mamba matched Transformers twice its size on their language tests.

The limit: a fixed-size summary must forget something. Tasks that depend on an exact detail from the prompt, or on copying a pattern shown in examples, are where that shows, as the follow-up section below explains.

Direction 2: hybrids

A hybrid keeps a small number of attention layers for precise lookup and uses SSM layers for the rest. Jamba, published in March 2024, uses one attention layer for every seven Mamba layers. Its authors report that this cut the KV cache to about 4 GB at 256,000 tokens, an eighth of the Mixtral figure, while scoring close to Mixtral and to Meta's Llama 2 70B on standard tests. They also found that the hybrid did better than either pure design in their small-scale experiments.

Direction 3: diffusion language models

The third idea changes how text is produced rather than how it is read. Standard models are autoregressive: they write one token after another, left to right. A diffusion model, the approach behind many image generators, starts from noise and refines the whole output over several steps. For text, the "noise" is hidden words. In February 2025 researchers Shen Nie, Chongxuan Li and colleagues presented LLaDA, an 8-billion-parameter model trained to fill in masked tokens. It performed about as well as autoregressive models the authors trained for comparison and was competitive with Meta's Llama 3 8B on learning from examples in the prompt. The limit so far is scale and evidence: published results are mostly at sizes far below the largest commercial models, and the benefit at those sizes is still an open question.

A worked comparison

The table summarizes the trade-offs. Figures come from each paper's own reports; the "good fit" column is a reasonable reading of those results, not a measurement.

	Transformer	State-space (Mamba)	Hybrid (Jamba)	Diffusion (LLaDA)
How it reads	Every token compares with every other	One pass, fixed-size state	Mostly one pass, a few attention layers	Transformer inside, but fills in many tokens per step
Memory for long input	Grows with length	Fixed	Small (8 times less cache in Jamba)	Not the focus of the paper
Known weak spot	Cost on very long inputs	Exact recall, learning from examples	More complex to design and tune	Tested mainly at modest size
Good fit	General use	Very long sequences: text, audio, genomics	Long documents at lower cost	Research for now

Try it: Pick a task you give an AI model, such as summarizing a long report or answering a question about one clause. Using the table, decide which design's weak spot would hurt that task most, and why.

What are state-space and hybrid models?

A state-space model is a sequence model that carries a running summary of everything it has read so far and updates that summary with each new token. It does not store every past token the way attention does, so it uses the same memory whether the input is a page or a book, and its cost grows in step with the length. Mamba's contribution was making that update selective, so the model chooses what to remember based on the content.

The weakness is the flip side of the strength. The Jamba team tested a pure Mamba model and found it often ignored the expected answer format when given examples. Asked to label film reviews as "Positive" or "Negative," it answered with phrases like "Very Good" or "3/10." The authors read this as difficulty with in-context learning, picking up a pattern from examples in the prompt, which attention handles well.

A hybrid model combines the two: mostly state-space layers for cheap, long reading, plus a few attention layers that can look back precisely. In Jamba's case, adding those few attention layers restored Transformer-like behavior on the format test while keeping most of the memory savings. That trade, a little attention for a lot of efficiency, is why hybrids are, for now, the most practical answer to what comes after the Transformer: not a replacement, but a Transformer with most of its expensive parts swapped out.