How do you run a language model on your own laptop with llama.cpp?

Twelve days after its first release, llama.cpp lets people run a GPT-3-class language model on an ordinary computer. It is a free, open-source program written in C and C++, with no extra software libraries required, and it runs entirely on the computer's main processor, the CPU. The project's stated goal is to run Meta's LLaMA model on a MacBook, and it now also supports Linux, Windows and Docker.

This guide walks through the steps as the project's README, its instruction page, describes them this week. One warning first: the code changes almost daily, and a large rewrite is being merged right now, so check the current README before you start.

Before you start: the model files and the licence

llama.cpp is only the engine. It does not include a model. The LLaMA weights, the billions of numbers the model learned during training, come from Meta, which released LLaMA on February 24 in four sizes: 7, 13, 33 and 65 billion parameters. Meta grants access case by case, for noncommercial research only, to academics, government, civil society groups and industry labs. The llama.cpp README states plainly that it will never distribute the model files itself.

Copies of the weights are circulating online without Meta's permission. Simon Willison, a developer who wrote about running LLaMA on his laptop, notes that an unofficial download link appeared within days of the release. Using those copies does not change the licence terms, so apply through Meta if you want to stay within them.

What your computer needs

Because the program loads the whole model into memory, you need as much free RAM as the model file is large, plus the same amount of disk space. The README lists these sizes (it labels Meta's 33B model as 30B):

Note that the conversion steps temporarily need room for the original files and the intermediate ones, so keep plenty of disk space free. The last column is a rough guide that leaves room for the operating system and your other apps. You also need Python, ideally version 3.9 or 3.10, since one required package does not yet install cleanly on 3.11.

The steps

Here is the sequence for the 7B model on a Mac or Linux machine, in the terminal:

1. Download and build. Clone the repository with git clone https://github.com/ggerganov/llama.cpp, move into the folder with cd llama.cpp, and type make.
2. Add the weights. Put Meta's original files in the models folder, so you have models/7B and the tokenizer.model file.
3. Install Python helpers. Run python3 -m pip install torch numpy sentencepiece.
4. Convert. Run python3 convert-pth-to-ggml.py models/7B/ 1 to turn the weights into the program's ggml format at 16-bit precision.
5. Quantize. Run python3 quantize.py 7B to shrink the file to 4 bits.
6. Run. Start the model with ./main -m ./models/7B/ggml-model-q4_0.bin -n 128. The -n value sets how many tokens, or word pieces, to generate.

Before step 4, the README suggests checking your downloaded files against the published checksums in its SHA256SUMS file, to confirm they are complete and genuine.

LLaMA is a raw model: it continues text rather than following instructions. So write prompts as the beginning of something you want completed:

A short guide for new gardeners. Step 1: Choose a sunny spot.
Step 2:

For a chat-like session, add -i for interactive mode and -r "User:", which hands control back to you whenever the model writes "User:". The README also shows an instruction mode that uses Alpaca, a model from Stanford researchers built on LLaMA.

What is quantization?

Quantization means storing each of a model's numbers with fewer bits, so the whole file takes less space. LLaMA's weights normally use 16 bits per number. llama.cpp rounds each one to a 4-bit value, which allows only 16 possible levels instead of tens of thousands. That is why the 7B model drops from 13 GB to 3.9 GB in the table above.

An everyday comparison is a photo saved with a smaller color palette. The image is much smaller and still easy to recognize, but fine shades are lost. In the same way, a quantized model keeps most of its knowledge and fluency, but its numbers are slightly less precise.

How much quality that costs is still an open question. The README says so directly in its list of limitations: the effect of quantization on the generated text is not yet known. For learning and experimenting, the trade is usually worth it, because without quantization even the smallest LLaMA model would need more memory than most laptops have.