What is Claude 3, and how do you choose between Opus, Sonnet and Haiku?

Yesterday, March 4, 2024, Anthropic announced Claude 3, the third generation of its AI assistant. Instead of one new model, it is a family of three, from smallest to largest: Haiku, Sonnet and Opus. Each step up is more capable, and slower and more expensive to run, so you can pick the balance of intelligence, speed and cost that suits a task.

Two models are available today. Sonnet now powers the free version of claude.ai, and Opus is available to Claude Pro subscribers. Both are also in Anthropic's API, which is now generally available, meaning developers can sign up and start right away, in 159 countries. Haiku is coming soon.

The three models side by side

A token is a small piece of text, often part of a word; one million tokens is roughly 750,000 words. "Input" is what you send to the model, "output" is what it writes back. Prices matter mostly to developers; on claude.ai you pay either nothing or the Pro subscription.

Speed differs too. Anthropic says Sonnet is about twice as fast as Claude 2 and 2.1 for most work, while Opus runs at about the same speed as those models but is much smarter. Haiku, it says, can read a dense research paper of about 10,000 tokens, charts included, in under three seconds.

What's new in Claude 3

It can see. For the first time, Claude can take images as input, as Tom's Guide points out. All three models can read photos, charts, graphs and technical diagrams. Anthropic notes that some business customers keep up to half of their knowledge in PDFs, flowcharts and slides, which earlier Claude models could not interpret.

Stronger benchmark results. On Anthropic's comparison, Opus outperforms other available models on most standard tests, including MMLU (undergraduate-level knowledge), GPQA (graduate-level reasoning) and GSM8K (grade-school math). Tom's Guide notes that the comparison is with OpenAI's GPT-4, which is about a year old, and Google's Gemini 1.0 Ultra. Anthropic also says that some newer models, such as Google's announced Gemini 1.5 Pro, are compared only in its technical model card.

Fewer unnecessary refusals. Earlier Claude models sometimes declined harmless requests that only looked risky. Anthropic says all three Claude 3 models refuse such borderline prompts much less often, while still recognizing real harm.

More accurate answers. On a large set of difficult factual questions, Anthropic reports that Opus gives correct answers about twice as often as Claude 2.1 and gives fewer wrong ones. It says the models are better at admitting when they do not know, and that citations pointing to exact sentences in your documents are coming soon.

Long memory for documents. Each model starts with a context window of 200,000 tokens, the amount of text it can consider at once, about a long novel. Anthropic says the models can accept more than one million tokens and may offer that to select customers. In a "needle in a haystack" test, where one planted sentence must be found in a huge collection of documents, Opus recalled it more than 99% of the time. In some runs, it even remarked that the sentence seemed to have been inserted artificially.

Better at following instructions. The models follow complex, multi-step instructions more reliably and produce structured formats such as JSON, which helps when Claude's answers feed into other software.

Safety

Anthropic says the Claude 3 family stays at AI Safety Level 2 (ASL-2) under its Responsible Scaling Policy, its framework that ties stricter safeguards to more dangerous capabilities. Its red-team evaluations found negligible potential for catastrophic risk at this stage. The company reports less bias than in previous models on a standard test called BBQ, and continues to train with Constitutional AI, its method of guiding a model with a written set of principles.

Why it matters

Claude 3 gives everyday users a stronger free model and, through Opus, a direct competitor to the paid tiers of other chatbots. Image input closes one of the main gaps between Claude and its rivals. For developers, three price points make it practical to use the cheapest model that does the job, and to keep the most capable one for hard problems.

Anthropic says it plans frequent updates to the family over the coming months, along with tool use (also called function calling), interactive coding and more agent-like abilities.

Here is a photo of a chart from my team's monthly report.
1. Describe what the chart measures and the time period.
2. List the three most important trends, with the numbers you read.
3. Tell me anything you are unsure about or cannot read clearly.

Which Claude 3 model should you start with?

Most people should start with Sonnet on free claude.ai. It handles everyday writing, summarizing, explaining and image questions well, and costs nothing. Use it on your real tasks for a week and note where it struggles.

Move to Opus, through Claude Pro, if your work involves long, complex reasoning: analyzing contracts or research, multi-step planning, difficult code or careful work across many documents. Compare its answers with Sonnet's on the same prompt before deciding it is worth the subscription.

If you are a developer, think in terms of cost per task. Prototype with Opus or Sonnet to see what quality is possible, then test whether Sonnet, or Haiku once it arrives, gives good enough results at a fraction of the price. Whichever model you use, ask it to point to the passage it relied on, and verify anything that matters.