What is Ollama, and how does it let you run AI models on your own Mac?

Yesterday, July 8, 2023, a small open-source project called Ollama published its first release on GitHub, the site where developers share code. Version 0.0.1 is labeled an early preview. Its promise is simple: type one short command, and a language model downloads to your Mac and starts answering you, with no cloud service involved and no internet connection needed once the model is on your disk.

Running a model locally has been possible since March, when the llama.cpp project showed that large models could fit on a laptop. But it usually meant finding model files, converting them, compiling code and remembering long command lines. Ollama wraps those steps into a tool that feels more like installing an app.

What was released

The release is the work of a small team led on GitHub by Jeffrey Morgan, whose account, jmorganca, hosts the project. The repository started on June 22, and Bruce MacDonald, Michael Yang, Eva Ho and Michael Chiang are among the people who have contributed code since. The code is under the MIT license, which lets anyone use, change and share it.

Here is what the first version offers, according to its README, the project's front-page description:

The first models

The project lists three example models, each a free, openly shared model packaged by a Hugging Face user known as TheBloke:

• Orca Mini, with 3 billion parameters, recommended as the fast starter model that runs well even without a graphics chip;
• Nous Hermes, with 13 billion parameters, a general-purpose model fine-tuned by Nous Research from Meta's LLaMA;
• Wizard Vicuna Uncensored, also 13 billion parameters, which its own description says has had its guardrails removed.

Parameters are the adjustable numbers inside a model; more parameters usually mean better answers but more memory. The README adds a warning worth reading: some models you can run with Ollama are licensed for research or non-commercial use only.

How it works

Ollama sits on top of llama.cpp, the program Georgi Gerganov released in March to run Meta's LLaMA models on ordinary computers. The key trick is quantization: storing each of the model's numbers with fewer bits, much as a photo can be saved at lower quality to take less space. Simon Willison reported in March that 4-bit quantization shrinks a 7-billion-parameter model to about 4 GB, small enough for a laptop.

Ollama adds three conveniences on top. It knows where to download each model. It runs a small server in the background so models stay ready between questions. And it fills in each model's prompt template, the special wording a particular model expects around your question, so you don't have to look it up.

Here is the whole first experience, step by step, as the README describes it:

1. Download Ollama for macOS and open it.
2. In the Terminal app, type ollama run orca.
3. Wait while the model downloads the first time.
4. Type a question at the prompt, such as "Write an email to my boss."
5. Ask a one-off question instead: ollama run vicuna "Why is the sky blue?"

For programmers, the local server listens on port 11434. A request to the address /api/generate with a model name and a prompt returns the model's answer, word by word.

Why it matters

Running a model on your own machine changes several things at once. Your prompts never leave your computer, which matters for private notes, medical questions or company documents. There is no per-word fee and no waitlist. And it works on a plane.

Ollama does not make local models as capable as the largest cloud models; a 3- or 13-billion-parameter model will make more mistakes than GPT-4. Its contribution is ease: it lowers the effort of trying open models from an afternoon of setup to a few minutes. For builders, the local API also means an app can be developed and tested against a model that costs nothing to call.

What do you need to try Ollama, and what are its limits?

For now you need a Mac with an Apple Silicon chip (M1 or newer), a few gigabytes of free disk space for each model, and some comfort with the Terminal. Windows users have to wait, though developers on Linux can try the Docker image or build the program from source. A 3-billion-parameter model like Orca Mini is the gentlest start; the 13-billion-parameter models give better answers but need more memory and patience.

Keep the limits in mind. This is a version 0.0.1 preview, so expect rough edges and frequent updates. Smaller models invent facts more often than large commercial ones, so check anything important. Read each model's license before using it for work. And be aware that "uncensored" models have had their safety training removed on purpose, so they will follow requests that other assistants refuse; decide whether that suits your use before you download one.