What's new in Claude 2.1, and what can its 200,000-token window do?

Yesterday Anthropic released Claude 2.1, an update to the AI model behind its claude.ai chat service and its developer API. The headline is size: Claude can now take in about 200,000 tokens in one go, enough for a long novel or a stack of financial reports. Anthropic also says the new model is markedly more honest, and it gives developers two new controls, system prompts and tool use.

Claude 2.1 is available today. It powers claude.ai on both the free and the paid Pro plan, and developers can use it through Anthropic's API and its Console, the web dashboard where they manage their access. The full 200,000-token window in the chat app is reserved for Pro subscribers.

What changed

A token is a chunk of text, often a short word or part of a longer one. The context window is how much text a model can consider in a single request: your question, any documents you attach, and its answer. Anthropic gives examples of what now fits: a whole codebase (the full source code of a program), a company's S-1 (the long filing a company makes before going public), or a long classic such as The Iliad.

For comparison, The Verge points out that the costliest version of OpenAI's GPT-4 tops out at 32,000 tokens. Anthropic calls processing 200,000 tokens an industry first.

There is a trade-off. Anthropic notes that working through a message that long is demanding, and a task that might take a person hours can take Claude a few minutes. It expects that wait to shrink as the technology improves.

More honest answers

Language models sometimes hallucinate: they state things that sound right but are false. Anthropic says Claude 2.1 does this half as often as Claude 2.0.

To test it, the company built a large set of tricky factual questions aimed at known weak spots. It then scored answers with a rubric that separates a wrong claim from an honest admission of uncertainty. Its example: asked for the fifth most populous city in Bolivia, a model can either name the wrong city with confidence or say it isn't sure. Claude 2.1 was much more likely to choose the second.

Anthropic reports similar gains with long documents such as contracts, financial reports and technical specifications: 30% fewer incorrect answers, and three to four times fewer cases of wrongly concluding that a document supports a claim. That last point matters, because a confident "yes, the contract says so" that turns out to be false can be worse than no answer at all.

System prompts and tool use

A system prompt is a standing instruction given to the model before the conversation starts. It sets the role, tone or format, and the model keeps to it throughout. For example:

You are a careful assistant for a small accounting firm. Answer only from the documents provided. If the documents don't contain the answer, say so plainly. Keep answers under 150 words and use plain language for clients.

Tool use lets Claude work with other software. A developer describes a set of tools, such as a calculator, a database search, a web search service or an internal company system, and Claude decides which one a request needs and asks for it to be run. Anthropic's examples include translating a plain-language request into a structured call to another program, looking up product data to make recommendations, and doing complex arithmetic with a calculator instead of guessing. The feature is in early development, and Anthropic is asking developers for feedback.

Developers also get a new Workbench in the Console, a place to write and test prompts, keep several versions, and turn a working prompt into code for their apps.

Why it matters

For ordinary users, a bigger window means fewer workarounds. Instead of splitting a long report into pieces and asking about each, you can give Claude the whole thing and ask questions that span it, such as comparing the first chapter with the last.

For businesses, the honesty figures may matter even more. Companies hesitate to put AI in front of customers or into legal and financial work when it might invent facts. Anthropic is pitching Claude 2.1 directly at that concern, while saying that accuracy remains a top priority still being worked on.

The Verge sees tool use in particular as a step that brings Claude closer to what ChatGPT can do.

How can you get the most out of Claude 2.1's long context?

Start with a clear task. A long context lets Claude see everything, but it still works best when you say exactly what you want: "List every deadline in this contract with its clause number" works better than "What's important here?"

Ask Claude to point to where its answers come from, such as a section, page or quoted sentence, and spot-check a few. Anthropic's improvements reduce errors; they don't remove them. Giving Claude explicit permission to say "the document doesn't say" helps too.

Finally, be patient with very long inputs. A 500-page upload can take a few minutes to process. If you only need one chapter, sending just that chapter is faster and often gives a sharper answer.