What is Llama 3.1 405B, and what does a free frontier-sized model change?

Yesterday, Tuesday, July 23, 2024, Meta released Llama 3.1, a new set of AI language models. The headline is the largest one, Llama 3.1 405B, which has 405 billion parameters. Meta describes it as the first frontier-level open model, meaning one that can stand next to the best systems from OpenAI and Anthropic, and anyone can download it.

That is new. Until now, the most capable models have been closed: you could use them through a website or an API, but never hold them. With Llama 3.1, a company, a university lab or a curious developer can keep the whole model on its own servers, inspect it and train it further.

What Meta released

The release has three sizes, each with a base version and an instruction-tuned version for chat:

All three can take in 128,000 tokens at once. A token is a word or a piece of a word, and TechCrunch puts that window at roughly a 50-page book. The earlier Llama 3 8B and 70B, released in April, stopped at 8,000 tokens, so this is a big jump. The models work in eight languages: English, Spanish, French, German, Italian, Portuguese, Thai and Hindi. They handle text only; Meta's researchers say models that see images and hear speech are in development but not ready.

The weights, the file of learned numbers that makes up the model, are on llama.meta.com and Hugging Face. Cloud companies such as AWS, Microsoft Azure and Google Cloud offer the models on launch day, and US users can try 405B in Meta AI on WhatsApp and meta.ai.

Meta also shipped safety tools, Llama Guard 3 (which checks prompts and replies for harmful content in several languages) and Prompt Guard (which tries to catch prompt injections), and asked developers to comment on a draft called Llama Stack, a set of standard interfaces for building on Llama.

How it was built

A language model learns by reading enormous amounts of text and repeatedly guessing the next token, adjusting its parameters each time it is wrong. Meta trained 405B on more than 15 trillion tokens, using over 16,000 Nvidia H100 chips (the graphics processors used for AI training).

Meta made a deliberate choice for stability: a standard decoder-only transformer, the same basic design behind most chatbots, rather than a "mixture of experts," which splits a model into specialist parts. After the main training, it ran several rounds of refinement. In each round the model was shown good examples to imitate (supervised fine-tuning), its best answers were kept and its weaker ones dropped (rejection sampling), and it was nudged toward the replies people prefer (direct preference optimization). Much of that practice material was synthetic data, examples written by a model rather than by people.

The models are also trained to call three tools on their own: Brave Search for recent events, the Wolfram Alpha service for math and science, and a Python interpreter to test code.

To make 405B practical, Meta shrank the precision of its numbers from 16 bits to 8 bits, a step called quantization. Picture rounding every price in a catalog to the nearest dime: the catalog gets much lighter and the totals barely change. That lets the model run on a single server node, though it still needs serious hardware.

How good is it?

Meta says it tested the models on more than 150 benchmarks and with human reviewers, and that 405B holds its own against GPT-4, Claude 3.5 Sonnet and GPT-4o. TechCrunch, reading Meta's research paper, gives a more careful picture: human evaluators rated it on par with GPT-4, with mixed results against GPT-4o and Claude 3.5 Sonnet. It does better than GPT-4o at running code and making charts, is weaker overall in languages other than English, and falls behind Claude 3.5 Sonnet at coding and general reasoning.

Why it matters this week

The license changed in an important way. Developers may now use what Llama produces, including 405B's answers, to train other models. Meta even recommends 405B mainly for two jobs: generating synthetic data and distillation, where a large "teacher" model's answers are used to train a smaller, cheaper "student." Some limits remain: any app that has over 700 million users a month must ask Meta for a special license.

In an open letter published the same day, Mark Zuckerberg argued that open models spread the benefits of AI more widely and keep power from concentrating in a few hands. Meta reports more than 300 million downloads of Llama models so far.

Open questions remain. Meta has not said exactly where its training data came from, beyond web pages and public files, and TechCrunch notes that no model of this kind has solved the habit of stating false things with confidence.

How can you try Llama 3.1 yourself?

The easiest way is to chat with it. In the United States, Meta AI on WhatsApp and at meta.ai runs Llama 3.1, and Meta suggests testing 405B with a challenging math or coding question.

If you want to run it on your own computer, start with the 8B model. Tools that run models locally can load it on a recent laptop, and nothing you type leaves your machine. The 70B model needs a powerful workstation, and 405B needs a server, so most people reach those through a cloud provider instead.

Whatever route you choose, give the model a clear role and task, as you would with any assistant:

You are a patient math tutor. Solve this problem step by step, then check your answer by working backwards: A train leaves at 9:40 and arrives at 13:15. How long is the trip?

Because the weights are yours to keep, a developer can also fine-tune the 8B or 70B model on their own examples, or use 405B to create training data for a smaller model that fits a narrow task.