What hardware do you need to run AI models at home in 2024?

Running a language model on your own computer means your questions never leave the machine, there is no monthly fee, and it works without an internet connection. Free tools such as Ollama make the software side easy: one command downloads a model and opens a chat. The hardware question is where most people get stuck, and the answer is simpler than it looks. For local AI, memory matters more than almost anything else.

When a model answers you, it has to read through all of its parameters for every word it writes. Parameters are the numbers the model learned in training, and there are billions of them. If they all sit in fast memory close to the processor, replies come quickly. If they don't fit, the computer has to shuffle pieces in from slower memory or the disk, and the model slows to a crawl or refuses to load.

Memory is the number to check

There are two kinds of computers to think about.

• A PC with a graphics card. The card, or GPU, has its own memory, called VRAM (video memory). A model runs fastest when it fits entirely in VRAM. NVIDIA's GeForce RTX 4090, announced in 2022, has 24 GB; the RTX 4080 has 16 GB. The computer's ordinary RAM can hold whatever doesn't fit, but that part runs much more slowly.
• An Apple silicon Mac. Macs with M-series chips use unified memory, a single pool shared by the main processor and the graphics processor. So most of the Mac's memory can serve as model memory. Apple's new MacBook Pro line, announced on October 30, starts at 16 GB and goes up to 128 GB with the M4 Max chip.

Speed also depends on memory bandwidth, meaning how many gigabytes per second the chip can read. Apple quotes 120 GB/s for the base M4 and more than half a terabyte per second for the M4 Max. Higher bandwidth means more words per second from the same model.

How big are the models?

The models you download for home use are usually quantized: their parameters are stored with fewer bits each, which shrinks the file a lot at a small cost in quality. Here are download sizes from Ollama's model library, with its own memory guidance:

Ollama's README gives a simple rule: at least 8 GB of RAM for 7-billion-parameter models, 16 GB for 13B models and 32 GB for 33B models. A practical way to read the table is that the download size is the minimum memory the model needs. Add a few gigabytes for the conversation itself, which the model keeps in memory as it grows, and more for your operating system and other apps.

Two worked examples

A MacBook Air with 16 GB. Recent MacBook Air models with M2 and M3 chips now start at 16 GB. Llama 3.1 8B at 4.7 GB fits easily with room for a browser and a long conversation. Gemma 2 9B or Phi 3 Medium fit too. Gemma 2 27B, at 16 GB, would take the whole memory, leaving nothing for the system, so it isn't a realistic choice on this machine.

A gaming PC with an RTX 4090 (24 GB VRAM) and 64 GB of RAM. Everything up to Gemma 2 27B fits inside the card's memory and runs fast. Llama 3.1 70B, at 40 GB, does not fit in 24 GB. Ollama can still run it by placing part of the model in system RAM, but the part on the slower side holds everything back. Expect a noticeable drop in speed, illustratively from a fluent stream of text to a few words per second, depending on how much spills over.

Is a Mac with lots of memory enough?

For many people, yes. Because unified memory serves as model memory, a Mac with 64 or 128 GB can load models that would need several graphics cards in a PC. Apple says an M4 Max with 128 GB lets developers work with language models of nearly 200 billion parameters. Llama 3.1 70B, at 40 GB, fits on such a machine with plenty of room left.

The trade-off is speed and flexibility. A high-end NVIDIA card has more raw computing power, and much of the open-source AI software, especially for training and fine-tuning models, is written first for NVIDIA's CUDA platform. A Mac is excellent for running models and chatting with them, and less suited to training your own.

Remember also that memory on Apple silicon is fixed when you buy the machine; you can't add more later. If local AI is a main reason for the purchase, choose more memory over a faster chip with less. And for the very largest open models, like Llama 3.1 405B at 231 GB, no single laptop is enough: those are still best used through a cloud service.