What is Claude 3.5 Sonnet, and what are Artifacts?

Yesterday, Thursday, June 20, 2024, Anthropic released Claude 3.5 Sonnet, the first model in a new Claude 3.5 family. It is free to use at claude.ai and in the Claude app for iPhone, and developers can reach it through Anthropic's API and through Amazon Bedrock and Google Cloud's Vertex AI, two cloud platforms that host AI models.

The same day, Anthropic added a feature called Artifacts to claude.ai. Instead of burying a piece of code or a document in the middle of a long chat, Claude can now place it in its own window next to the conversation, where you can see the result and keep working on it.

A middle-sized model that beats the top one

Anthropic sorts its models by size. Haiku is the smallest and fastest, Sonnet is the middle option, and Opus is the largest and most capable. In March, Anthropic launched Claude 3 Opus as its most capable model. The surprise this week is that the new middle model outperforms the old top one.

Anthropic says Claude 3.5 Sonnet beats Claude 3 Opus and rival models on a wide range of tests, at the speed and price of Claude 3 Sonnet. It runs about twice as fast as Claude 3 Opus. Anthropic reports new high scores on three well-known benchmarks: GPQA, a set of hard graduate-level science questions; MMLU, a broad test of knowledge across many subjects; and HumanEval, which checks whether a model can write working computer code. The Verge, looking at Anthropic's charts, notes that the model outscored OpenAI's GPT-4o, Google's Gemini 1.5 Pro and an early version of Meta's largest Llama 3 model on seven of nine general benchmarks and four of five vision benchmarks, while reminding readers that companies choose which benchmarks to show.

Here are the main facts in one place:

A token is a small piece of text, often part of a word. The context window is how much text the model can consider at once, so 200,000 tokens is enough for a long report or a short book.

What it does better

Coding. Anthropic ran an internal test in which the model is given a real open-source software project and a plain-language request to fix a bug or add a feature. Claude 3.5 Sonnet solved 64% of the problems; Claude 3 Opus solved 38%. With the right tools, Anthropic says, the model can write, edit and run code on its own and work through errors. It is also good at code translation, rewriting programs from one language or style to another, which helps companies update old software.

Vision. It is Anthropic's best model yet at understanding images. The gains show most in visual reasoning, such as reading a chart and explaining what it shows, and in transcribing text from blurry or imperfect photos.

Writing. Anthropic says the model is better at grasping nuance, humor and complicated instructions, and writes in a more natural tone.

How Artifacts work

Until now, working with a chatbot on something substantial meant copying its output into another program to see it properly. Artifacts remove that step. When you ask Claude for something self-contained, such as a block of code, a document or a simple website design, it appears in a separate panel beside the chat. You can look at it, ask for changes in the conversation, and watch the panel update.

A quick example:

Make a simple one-page website for a neighborhood bake sale on Saturday at 10 a.m. Include a headline, a list of six baked goods with prices, and a short note that proceeds go to the school library. Use warm colors.

The chat explains what Claude did; the Artifacts panel shows the page itself. Follow up with "make the prices bigger" or "add a map placeholder," and the panel changes. The Verge gives another everyday case: if Claude drafts an email, you can edit it right in the Claude app.

Anthropic calls Artifacts a preview and describes it as a step from a conversational assistant toward a shared workspace. It says claude.ai will soon support teams, so groups and eventually whole organizations can gather their files, know-how and projects in a single shared space, with Claude acting as an on-demand teammate.

Safety, privacy and what comes next

Anthropic classifies models under its own scale of AI Safety Levels, and says Claude 3.5 Sonnet remains at level 2 (ASL-2) despite its gains. Before the release, the UK's AI Safety Institute tested the model and shared its results with its US counterpart. Anthropic also says it does not train its models on what users submit unless they explicitly allow it, and that so far it has not used customer or user data for training.

Anthropic plans to complete the 3.5 family with a smaller Haiku and a larger Opus later this year, and is exploring a Memory feature so Claude can remember a person's preferences when they ask it to.

Why it matters

Three months after Claude 3 Opus, Anthropic has a mid-priced model that it says matches or beats the best from OpenAI and Google. For people using the free version of claude.ai, that means a noticeably stronger assistant at no cost. For developers, top-tier results at a mid-tier price change the math of building an app. Artifacts matter as much as the model: they hint that chat assistants are turning into places to make and revise things, not just to ask questions.

How can you start using Claude 3.5 Sonnet and Artifacts today?

Go to claude.ai or open the Claude iOS app and sign in; Claude 3.5 Sonnet is available to free users, with higher usage limits on the paid Pro and Team plans. Start with a task where you can judge the result: a chart from a spreadsheet you know, a piece of code you can run, or a document you can check.

For Artifacts, ask for something self-contained and visual, such as a web page, a diagram or a formatted document, then refine it step by step in the chat. Developers can call the model through Anthropic's API, Amazon Bedrock or Vertex AI. Compare its answers and costs with the model you use now on a few real examples before switching.