What is llama.cpp, and how can a large language model run on a laptop?

Until this week, running a language model in the same class as GPT-3 meant renting powerful servers. Developer Simon Willison notes that such models usually need several A100-class graphics chips, each costing $8,000 or more. Yesterday, March 10, 2023, that changed. Georgi Gerganov, an open-source developer in Sofia, Bulgaria, published llama.cpp on GitHub, and by this morning developers were generating text with Meta's LLaMA models on their own laptops.

Willison ran the 7-billion-parameter model on his laptop last night and the 13-billion-parameter one this morning, the version Meta claims is competitive with GPT-3. He calls it the "Stable Diffusion moment" for language models, comparing it to last August, when an image generator became something anyone could run at home.

What llama.cpp is

llama.cpp is an inference program. Inference means using a model that is already trained to produce output; it does not train anything. It runs LLaMA, the family of research models Meta released on February 24, from 7 to 65 billion parameters (the internal numbers a model learns).

The project's README, its main instruction page, lists the design:

• Plain C/C++ with no other software dependencies
• Apple silicon treated as a first-class target, using its Arm Neon instructions
• Mixed 16-bit and 32-bit number precision
• Support for 4-bit quantization
• Runs on the CPU, the computer's main processor, rather than a graphics card

Gerganov is open about how new it is. The README says it was "hacked in an evening" and that he is not sure it works correctly. Today's updates add a stronger caution: do not judge the LLaMA models from this implementation, which he describes as an educational project that will not be properly maintained.

He has done something similar before. Willison points out that Gerganov earlier released whisper.cpp, a C++ version of OpenAI's Whisper speech recognition model.

How it fits a big model on a laptop

The trick is quantization. A model's parameters are normally stored as 16-bit numbers. Quantization rounds each one to a coarser 4-bit value, so each parameter takes about a quarter of the space. Think of a photo saved with fewer colors: it is smaller and still recognizable, though some detail is lost. Willison reports the 7B model shrinks to about 4 GB on disk and the 13B model to just under 8 GB.

The README walks through the steps:

1. Download and build the code with make.
2. Place the original LLaMA weights (the trained parameters) in a models folder.
3. Run a Python script to convert them to the project's 16-bit ggml format.
4. Run the quantize tool to turn that file into a 4-bit one.
5. Start ./main with the model file and a prompt.

In the README's sample run on an Apple M1 Pro, the 7B model continues the prompt "Building a website can be done in 10 simple steps:" with a numbered list, at roughly 61 milliseconds per word piece (token), a usable speed. The text soon starts repeating itself, a reminder that this is a raw model, not a polished chatbot.

The project is moving quickly. On launch day only the 7B model was tested and quantized models ran only on Apple silicon. By today, the README covers all four sizes, which split into one to eight files, and adds AVX2 support for x86 processors, the type found in most Windows and Linux PCs, with Windows support under discussion.

Why it matters, and the catch

Running a model on your own machine means no internet connection, no usage fees, and no company in the middle. Willison argues that this lets developers explore how these systems work from the inside, as Stable Diffusion did for images.

There are two catches. First, access: Meta grants LLaMA weights case by case for noncommercial research. Willison reports that within days of the release, someone posted an unofficial BitTorrent link to the model files, so they are now circulating widely, but that does not change the license. Second, risk: he lists spam, romance scams, disinformation and automated radicalization as real harms, and notes that the limited controls companies could apply through their own websites do not exist when the model runs on your laptop.

Should you try running a language model on your own computer?

If you are comfortable with a terminal, have an Apple silicon Mac or a recent processor with enough memory, and have legitimate access to the LLaMA weights through Meta's research program, llama.cpp is a remarkable learning tool. You will see exactly what a raw language model does, without the safety filters and polish of a chatbot.

For most beginners, it is worth waiting a little. The project is days old, its author warns it may contain mistakes, and the setup involves compiling code and converting large files. The models also need practice to use well, because they are not tuned to follow instructions the way ChatGPT is.

Either way, the bigger lesson is clear already: capable language models no longer require a data center. Expect more tools that package this approach for ordinary users, and keep the same habits as with any AI tool: check facts, respect licenses, and think about how what you generate could be misused.