What is LLaMA, Meta's new language model for researchers?

While most of the attention in AI this month has gone to chatbots from OpenAI, Microsoft and Google, Meta took a different path yesterday, February 24, 2023. Its research division, Meta AI, released LLaMA, short for Large Language Model Meta AI. It is not a chatbot you can talk to. It is a set of models meant for researchers who want to study how large language models work and how to fix their problems.

Mark Zuckerberg announced the release in a Facebook post, and Meta frames it as part of its commitment to open science. The models come under a noncommercial license, and access is granted case by case.

What LLaMA is

A large language model is a program trained to predict the next word in a piece of text. Give it a sequence of words, and it predicts what comes next, again and again, to produce sentences. LLaMA works exactly this way. Its "size" is measured in parameters, the internal numbers the model adjusts during training; more parameters usually means more capacity, but also more computing power to train and run.

LLaMA is a foundation model: a general model trained on a large amount of unlabeled text, which others can then fine-tune (train a little further) for specific tasks such as answering questions or summarizing. Meta is releasing it in four sizes, along with a model card, a document describing how it was built.

Model	Size	What Meta and The Verge report
LLaMA 7B	7 billion parameters	Smallest version; trained on one trillion tokens
LLaMA 13B	13 billion parameters	Meta's paper claims it beats GPT-3 on most benchmarks; can run on a single Nvidia V100 data-center GPU
LLaMA 33B	33 billion parameters	Trained on 1.4 trillion tokens
LLaMA 65B	65 billion parameters	Trained on 1.4 trillion tokens; described as competitive with DeepMind's Chinchilla-70B and Google's PaLM-540B

A token is a piece of a word; models read text as tokens rather than whole words. Meta chose training text from the 20 most widely spoken languages, focusing on those written in Latin and Cyrillic alphabets.

How smaller models can compete

The key idea behind LLaMA is to train smaller models on much more text. GPT-3, which OpenAI described in a 2020 paper, has 175 billion parameters. If Meta's claims hold, LLaMA-13B outperforms it on most tests at less than a tenth of the size.

Why does that matter? A model's size drives the cost of running it. Every answer requires the computer to work through all those parameters. A model that fits on one graphics processor (GPU), the kind of chip used for AI, can be studied by a university lab. A model that needs a room full of them can only be studied by a handful of large companies.

An analogy: imagine two students preparing for an exam. One has an enormous memory but skimmed the textbook once. The other has a smaller memory but read the textbook many times, carefully. The second can do just as well, and is far easier to bring along to a study group. Meta argues that smaller models trained on more tokens are also easier to retrain and fine-tune.

Why it matters

Meta points to a real gap. Large language models are showing new abilities, from writing creative text to answering reading-comprehension questions, yet full research access to them is limited because they are so expensive to train and run. Outside researchers cannot easily examine why they work or tackle known problems such as bias, toxic output and hallucination, a model's habit of making up information.

By sharing LLaMA's code and publishing tests of the models' bias and toxicity, Meta hopes others will find new ways to reduce these problems. The company also says it wants academics, civil society, policymakers and industry to work together on guidelines for responsible large language models.

The Verge points out that Meta's earlier public efforts had a rough reception. Its BlenderBot chatbot was criticized for being weak, and Galactica, a model for writing scientific text, was taken offline after three days because it produced scientific nonsense. LLaMA's research-only release is a more cautious approach.

Try it: Using the table above, write two sentences explaining to a friend why a 13-billion-parameter model that matches a 175-billion-parameter one could matter for a small university lab. Then list one risk of making powerful models easier to run.

Who can get LLaMA, and what can researchers do with it?

Meta is not posting LLaMA for anyone to download freely. To prevent misuse, it is releasing the models under a license that allows research use but not commercial use. Access is decided case by case and is open to academic researchers, staff of government bodies and civil-society groups, and research labs in industry worldwide. The application link appears in Meta's research paper.

Researchers who are approved can run the models on their own hardware, test them, and fine-tune them for experiments, something not possible with models offered only through a company's website or paid API. Because LLaMA is a foundation model rather than a finished chatbot, it gives them a flexible base for studying many tasks.

For everyone else, there is nothing to install yet, and businesses cannot build products on it under this license. What is worth watching is what researchers report in the coming weeks: whether the benchmark claims hold up, and whether easier access to a capable model leads to progress on bias, toxicity and hallucination.