# What has interpretability research found inside language models?

Date: 2026-03-22 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/has-interpretability-research-found-inside-language-models/

**Answer:** Researchers have found that language models store recognizable concepts, called features, such as a particular landmark, a bug in code or flattery, and that these features connect into circuits that carry out steps like looking up a fact, planning a rhyme or doing mental arithmetic. Turning a feature up or down changes what the model says, which shows the features matter, but today's tools still explain only a small part of what a model does.

**Interpretability** is the research field that tries to explain how an AI model produces its answers by looking inside it, rather than only at what goes in and comes out. For language models this is hard. While a model works, its internal state is a long list of numbers, the activations of its artificial neurons, and none of those numbers comes with a label.

Over the past few years, researchers at Anthropic, OpenAI and other labs have built tools that turn some of those numbers into concepts a person can recognize. The results are real but partial: a growing collection of findings about what models represent and how some answers are assembled, alongside a clear sense of how much remains dark.

## Features: finding concepts among the numbers

The first obstacle is that a single neuron rarely means one thing. Anthropic describes each concept as spread across many neurons, with each neuron involved in many concepts. Reading neurons one at a time is like trying to understand a choir by listening to one singer who keeps switching songs.

The workaround is a method called **dictionary learning**, usually done with a **sparse autoencoder**: a second, simpler network trained to rewrite the model's activations as a combination of a few items from a much larger list. Each item that keeps meaning the same thing is called a **feature**.

In May 2024, Anthropic reported pulling millions of features out of Claude 3 Sonnet, read from a layer halfway through the model. Some matched concrete things, such as cities, people, chemical elements and programming syntax; a Golden Gate Bridge feature responded to the bridge's name in several languages and to photos of it. Others were abstract: bugs in code, discussions of gender bias, keeping secrets, sycophantic praise. A month later, a team of OpenAI researchers published a paper on scaling sparse autoencoders and trained one with 16 million features on GPT-4's activations, showing the method works on very large models too.

## Circuits: following a thought from question to answer

A list of concepts doesn't explain reasoning. In March 2025, Anthropic published work linking features into **circuits**, the pathways that turn the words going into a model into the words coming out, and studied Claude 3.5 Haiku on simple tasks. Here is a worked set of the experiments, each built the same way: find a feature, change it, and watch the answer.

| Experiment | What researchers changed | What happened | What it suggests |
|---|---|---|---|
| Golden Gate Bridge (Claude 3 Sonnet) | Turned the bridge feature up strongly | The model described itself as the bridge | Features cause behavior, not just accompany it |
| "Capital of the state where Dallas is" | Swapped the "Texas" concept for "California" | The answer changed from Austin to Sacramento | The model reasons in steps, not only by recalling answers |
| A rhyming couplet | Removed the planned rhyme word "rabbit" | The model wrote a line ending in a different rhyme | The model plans words ahead of writing them |
| Adding 36 and 59 | Traced the circuit without changes | One path estimated the size of the sum, another got the last digit | The model's method differs from the one it describes |

The last row is striking. Asked how it got 95, the model described carrying the one, the method taught in school, while its circuits showed something else. Anthropic also found cases where, given a wrong hint on a hard problem, the model wrote reasoning that worked backward to agree with the user. A model's own explanation of its thinking, in other words, is not a reliable view of its inner workings.

An analogy helps. Features are like the words in a model's private vocabulary; circuits are like the sentences it builds with them. The research so far has compiled a partial dictionary and diagrammed a few short sentences.

> **Try it:** Ask a chatbot "What is 36 + 59? Explain exactly how you worked it out." Compare its explanation with the parallel paths described in the table, and notice that a fluent explanation is not evidence of the process behind it.

## The limits researchers report

The researchers are frank about the gaps. Anthropic says the features it found are a small subset of what its model learned, and that finding them all with the same method would take more computing power than training the model did. Its circuit-tracing method captures only a fraction of the computation even on short prompts, may show artifacts of the tools rather than the model, and takes a few hours of human work to understand a prompt of a few dozen words. Most results come from simple, chosen examples, not from the long, messy tasks people give models every day.

## Does it make models safer?

Not yet directly, but it is building tools that could. Some findings already touch safety. Anthropic located features tied to scam emails, to behaviors such as manipulation, power-seeking and secrecy, and to flattery; when it strongly activated the scam-email feature, the model wrote a scam email it would normally refuse. Its circuit work found that refusing to answer is a default that recognizing a familiar name switches off, and that misfires of this switch can produce made-up answers, a concrete view of one source of hallucinations.

The hoped-for uses are monitoring a model's internal states for warning signs such as deception, steering it away from unwanted behavior, and auditing whether it is doing what it claims. Anthropic reports that its tools helped reveal the hidden goal of a test model that had been deliberately trained to pursue one and was reluctant to admit it.

For now, interpretability works best as an extra check alongside testing, training methods and human oversight, not as a guarantee. A model can be partly understood and still make mistakes, and the parts nobody has mapped yet are most of it.

## Sources

- Mapping the Mind of a Large Language Model: https://www.anthropic.com/research/mapping-mind-language-model
- Tracing the thoughts of a large language model: https://www.anthropic.com/research/tracing-thoughts-language-model
- Scaling and evaluating sparse autoencoders: https://arxiv.org/abs/2406.04093v1