How do you run an AI model on your own computer in 2026?

Running a model locally means the AI works on your own machine: nothing you type leaves it, it keeps working offline, and there is no monthly bill. The setup has become routine. A free program downloads an open-weight model (one whose trained numbers are published for anyone to use), loads it into memory and gives you a chat window or a local API. The hard part is no longer installing anything. It is choosing a model that fits your hardware.

This guide updates the 2024 hardware guide and the Ollama or LM Studio comparison linked below with what has changed by October 2026: bigger default context windows, better small and mid-size models, and clearer advice from the tools themselves.

The three tools most people use

LM Studio is a desktop app with a model browser and a chat window, the easiest start for most people. As of October 2026 its documentation asks for an Apple Silicon Mac (M1 to M4) on macOS 14 or newer, or a Windows or Linux PC. It recommends 16 GB of RAM, and on Windows at least 4 GB of dedicated graphics memory. An 8 GB Mac still works if you stay with small models and short conversations.

Ollama runs in the background and is driven from a small app or the terminal (ollama run <model>). It suits people who want other programs, such as a coding assistant, to talk to a local model.

llama.cpp is the engine underneath many of these tools. You use it directly when you want full control over every setting.

All three read the same kind of file, usually GGUF, a format that stores the weights in compressed form together with the information needed to run them.

A worked example: what fits your machine

The rule of thumb is simple: the model file must fit in fast memory, with room to spare. On a PC, fast memory means the VRAM of your graphics card. On an Apple Silicon Mac, the processor and graphics share one pool of unified memory, and the system needs part of it for itself.

The llama.cpp project publishes measured sizes. A Llama 3.1 model with 8 billion parameters takes about 15 GiB in full 16-bit precision but only about 4.6 GiB in the popular Q4_K_M format; the 70-billion version shrinks from about 281 GB to 43 GB. From those figures you can estimate other sizes at the same 4-bit level:

The middle rows are estimates (parameters multiplied by about 0.6 bytes each), so check the actual file size on the model page before you download. Leave a few gigabytes free for the conversation, as the next section explains.

Steps for a first run in LM Studio:

1. Open the model search, type a model family you know, and choose a version marked Q4_K_M whose size fits the table above.
2. Load it and ask a question you know the answer to.
3. Watch the speed. If words appear slowly, try the next smaller model rather than a lower-quality file of the same one.

Context also needs memory

The context window is how much text the model can keep in view at once: your question, the earlier conversation and any pasted document. It lives in memory beside the model, so a long context can push a model that "fits" over the edge.

Ollama now sets the default automatically from your graphics memory: about 4,000 tokens below 24 GiB of VRAM, 32,000 between 24 and 48 GiB, and 256,000 above that. Its documentation suggests at least 64,000 tokens for agents, coding tools and web search. If you raise the setting on a smaller machine, expect to need a smaller model. The command ollama ps shows how a loaded model is split between graphics card and processor. Anything other than 100% GPU means part of the work has spilled to the slower CPU, and Ollama advises avoiding that.

What do quantization and VRAM mean for what you can run?

Quantization stores each weight with fewer bits. A model is trained with 16 or 32 bits per number; a quantized file might use about 8, 5 or 4.5. The file shrinks in proportion, and generation usually gets faster because less data has to move through memory. In llama.cpp's own test on the 8B model, the 4-bit file wrote about 72 tokens per second against 29 for the 16-bit original on the same machine.

The cost is a small loss of accuracy, which the llama.cpp developers measure with statistics such as perplexity (how surprised the model is by real text). Around 4 to 5 bits, most people notice little difference in everyday chat. Below about 3 bits, answers degrade faster. Two practical rules follow: a larger model at 4 bits usually beats a smaller model at 8 bits for the same memory, and you should download a file quantized directly from the original weights, since squeezing an already compressed file again can hurt quality badly.

VRAM is the ceiling. If the model plus its context fits in graphics memory (or in the share of unified memory macOS lets the GPU use), it runs at full speed. If it does not, the tool either refuses to load it or splits it with the main processor, and speed can drop several times over. So the question "can I run this model?" really means "does its quantized file, plus the context I need, fit in my fast memory?" Answer that with the file size on the download page, and you will rarely be surprised.