How do you run Qwen3.5 locally?

When Alibaba released Qwen3.5 on February 16, its first open model was enormous: 397 billion parameters, far too big for a personal computer. Yesterday the Qwen team added three mid-sized members to the family, and one of them, Qwen3.5-35B-A3B, is the first that a well-equipped laptop or desktop can realistically run. It keeps the family's main features: it reads text and images, thinks before answering, supports 201 languages and dialects, and is free to use under the Apache 2.0 license.

The name tells you what makes it practical. It has 35 billion parameters in total (the adjustable numbers a model learns), but it is a mixture of experts: for each token, a router activates only about 3 billion of them ("A3B"). All 35 billion still have to sit in memory, but each word costs the computing effort of a much smaller model, so it answers at a comfortable pace on ordinary hardware.

What you need

Two things: a program that runs the model, and a compressed copy of the model that fits your memory.

The program is llama.cpp, a free, open-source engine for running language models on everyday hardware. It reads models in a file format called GGUF, and it can download them directly from Hugging Face, the main website for sharing models.

The compressed copy comes from quantization: storing each parameter with fewer bits (4 instead of 16, for example), which shrinks the file and the memory it needs at a small cost in quality. Unsloth, a team that adapts open models for home hardware, published GGUF versions of Qwen3.5-35B-A3B within hours of the release, in sizes from about 12 GB to about 25 GB.

Qwen3.5 uses a new architecture, so use a current build of llama.cpp. An older copy you installed months ago may not recognize the model.

A worked install, step by step

1. Install llama.cpp. On a Mac (or Linux with Homebrew), and on Windows:

brew install llama.cpp       # Mac or Linux with Homebrew
winget install llama.cpp     # Windows

If it is already installed, update it (brew upgrade llama.cpp on a Mac).

2. Download and start the model. The -hf option names the Hugging Face repository, and the part after the colon picks a quantization:

llama-cli -hf unsloth/Qwen3.5-35B-A3B-GGUF:Q4_K_M \
  --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.5 \
  -c 32768

The first run downloads about 21 GB, so start it on a fast connection. The sampling settings are the ones the Qwen model card recommends for thinking mode on general tasks. Temperature controls randomness, and presence penalty discourages the model from repeating itself. The -c 32768 sets a context window (how much text the model keeps in view) of 32,768 tokens to save memory. The model supports far more, and the card advises keeping at least 128K for long reasoning; raise it if you have memory to spare.

3. Ask something that rewards thinking.

A train leaves at 9:40 and arrives at 13:15, with a 25-minute stop on the way. What was the time actually spent moving? Show your reasoning briefly, then give the answer in one line.

The model writes its reasoning first, inside <think> tags, then the answer. Read the reasoning: it shows whether the model understood the question.

4. Prefer a browser window? Run the server instead:

llama-server -hf unsloth/Qwen3.5-35B-A3B-GGUF:Q4_K_M -c 32768

Then open http://localhost:8080 for a simple chat page. The same address also offers an OpenAI-compatible API, so other apps on your computer can use the model.

Which size fits your computer?

Memory decides. The model file must fit in RAM, or in a graphics card's memory, with room left for the context and for the rest of your system. Here are the Unsloth files for Qwen3.5-35B-A3B and a rough guide to the memory each needs:

The memory column is a rule of thumb, not a guarantee: download size plus several gigabytes for context and the operating system. Fewer bits mean smaller files and slightly weaker answers; 4-bit is the usual balance, and 2-bit is a last resort.

The other new models are a different scale. Qwen3.5-27B should need roughly 16 to 17 GB at 4 bits, and Qwen3.5-122B-A10B roughly 70 GB or more; both figures are estimates scaled from the 35B file, not published sizes. The 122B model is for workstations and high-memory Macs. The original 397B model needs server hardware.

A sensible path: on a 16 GB machine, try the 2-bit file and judge for yourself, or use the hosted version, which Alibaba offers through its Model Studio service as Qwen3.5-Flash. On 24 GB, start with the 3-bit file. With 32 GB or more, use a 4-bit file. If replies crawl or your computer starts swapping to disk, step down one size or lower the context. Whatever you choose, everything you type stays on your own machine.