What is Llama 3.2, and how can an AI model see images or run on a phone?

Yesterday, Wednesday, September 25, 2024, Meta released Llama 3.2 at Meta Connect, its annual developer conference in Menlo Park, California. Mark Zuckerberg announced it on stage. Two months after Llama 3.1 brought a giant 405-billion-parameter model, this release goes in two other directions: models that can look at pictures, and models small enough to live on your phone.

Like earlier Llama models, these come as open weights: anyone can download the files that make up the model from llama.com or Hugging Face, run them on their own hardware and adapt them, under Meta's license.

What Meta released

Llama 3.2 has four sizes, in two groups:

A parameter is one of the internal numbers a model adjusts during training; more parameters usually means more knowledge but more computing power. "Edge devices" are machines at the edge of the network, such as phones, cars or smart cameras, as opposed to big data centers.

All four can work with 128,000 tokens at a time. TechCrunch, citing Meta, puts that at roughly 100,000 words. The small models are ready on launch day for chips from Qualcomm and MediaTek, the two largest makers of phone processors, and are tuned for Arm, whose designs run in nearly all mobile devices.

Meta also released new safety filters: Llama Guard 3 11B Vision, which checks prompts that combine text and images, and Llama Guard 3 1B, a compact filter cut down from 2,858 MB to 438 MB so it can sit alongside a small model. And it shipped the first official Llama Stack distributions, ready-made bundles that make it easier to run Llama on one computer, in the cloud, on company servers or on a device.

How the vision models see

The 11B and 90B models can describe a photo, read charts and graphs, write captions and point out where an object is in a picture. Meta's examples are practical: show it a sales chart and ask which month was best, or show it a trail map and ask where the hike gets steeper.

To build them, Meta did not start from scratch. It took its Llama 3.1 text models and attached an image encoder, a component that turns a picture into numbers a model can work with. Between the two, it trained adapter layers that pass what the encoder sees into the language model. During this training, Meta updated the encoder and adapters but left the language model's own weights untouched.

Think of hiring an interpreter for an expert. The expert keeps all their knowledge; the interpreter describes what is in the photo in terms the expert understands. Because the language part was not changed, the vision models are drop-in replacements for the matching Llama 3.1 text models: anything they could do with text, they still can.

How the small models got small

For the 1B and 3B models, Meta used two techniques on its larger models:

1. Pruning. It started from Llama 3.1 8B and removed parts of the network, keeping as much of its ability as possible, like trimming a long book while keeping the plot.
2. Distillation. It then trained the small models using the answers of Llama 3.1 8B and 70B as targets. The big models act as teachers; the small ones learn to imitate them.

Running a model on the device has two clear advantages, Meta says. Replies can feel instant, because nothing travels to a server and back. And your messages, calendar and notes stay on your phone. Meta's example is an assistant that summarizes your last ten messages, pulls out action items and uses a calendar tool to send follow-up invitations, all locally.

Why it matters this week

For developers, Llama 3.2 fills two gaps: an open model that understands images, and capable models small enough for apps that must work offline or protect private data. Meta compares its vision models with Claude 3 Haiku and GPT-4o mini on image tasks, and says the 3B model beats Google's Gemma 2 2.6B and Microsoft's Phi 3.5-mini on tasks such as following instructions, summarizing and using tools.

There is one notable exception. TechCrunch reports that the 11B and 90B vision models are not available in the European Union, where Meta points to an unpredictable regulatory environment; the 1B and 3B models launch there as everywhere else. As with earlier releases, platforms with more than 700 million monthly users need a special license from Meta.

How can you try Llama 3.2 today?

The quickest way to try the vision models is Meta AI, Meta's assistant in WhatsApp, Instagram and Facebook, which now uses them, outside the EU. You can share a photo and ask questions about it.

If you want to run a model yourself, start with Llama 3.2 3B or 1B. Meta's single-computer Llama Stack distribution works through Ollama, a free tool for downloading and running models on your own computer. The small models are a good fit for private tasks such as tidying notes:

Rewrite these meeting notes as a short list of action items, each with an owner and a due date if one is mentioned. Do not add anything that is not in the notes.

[paste notes here]

Developers can download all four models from llama.com or Hugging Face, or use them on cloud platforms such as AWS, Google Cloud and Microsoft Azure. For the vision models, test them on your own images, and check details like numbers on a chart, which any model can misread.