What are small language models, and when are they enough?

When people talk about the "size" of a language model, they usually mean its number of parameters: the adjustable numbers a model tunes during training. The models behind ChatGPT are very large and run in data centers. A small language model, often shortened to SLM, has only a few billion parameters. That makes it cheap to run and quick to reply, and small enough to work on ordinary hardware, sometimes with no internet connection at all.

Two recent families show where this is going. In February, Google released Gemma, open models in 2-billion and 7-billion-parameter sizes built from the research behind Gemini, which Google says can run on a laptop or a workstation. On April 23, Microsoft introduced Phi-3. Its first member, Phi-3-mini, has 3.8 billion parameters and can be downloaded from Hugging Face or run through Ollama, a free program for running models on your own computer. Microsoft says two larger versions, Phi-3-small (7 billion) and Phi-3-medium (14 billion), are coming in the next few weeks.

Why small models got better

The Phi-3 technical report gives the credit to training data rather than size. The team used web pages filtered hard for quality, plus synthetic data, meaning text generated on purpose to teach the model. Phi-3-mini was trained on 3.3 trillion tokens (the word pieces models read) and scores 69% on MMLU, a broad multiple-choice test of knowledge and reasoning. The authors compare that with GPT-3.5 and with Mixtral 8x7B, a much larger open model. Benchmarks are only rough guides, but the trend is clear: a model you can run at home now handles many everyday language jobs.

The same report is honest about the weak spot. A small model has less room to store facts, so Phi-3-mini does poorly on TriviaQA, a test of general-knowledge questions, and Microsoft repeats this caution in its announcement. The authors suggest pairing the model with a search engine to fill the gap. They also note that it was built mostly for English.

When a small model is enough

A useful rule of thumb: small models are good at working with text you give them and weak at knowing things you don't. Microsoft's own list of good fits matches this. It names devices with limited computing power or no connection, situations where a fast reply matters most, and simpler tasks where cost is the main concern.

These verdicts are general guidance, not measured results. Your own task is the only test that counts.

There is also a privacy angle. When a model runs on your computer, the text you paste in is processed there instead of being sent to a company's servers. For drafts that include client or family details, that can matter as much as quality.

A worked test: small versus large

The fairest way to decide is to give the same real task to a small model and a large one and score both. Here is a prompt that works well for this kind of comparison:

Below is an email from a client. List every date, amount and action item it contains.
Answer as a table with three columns: Item, Detail, Who is responsible.
If something is not stated, write "not stated". Do not add anything that is not in the email.

[paste the email here]

1. Install Ollama, open a terminal and type ollama run phi3. The first run downloads Phi-3-mini; after that you can chat with it offline.
2. Paste the prompt with a real email from your work.
3. Send the same prompt to a large hosted model, such as GPT-4o in ChatGPT. Use an email without private details for this step, since it leaves your computer.
4. Score each answer in three columns: items found correctly, items missed, items invented.

A result might look like this (illustrative): the small model finds 9 of 10 items and invents none; the large one finds all 10. If 9 out of 10 is acceptable because you review the table anyway, the small model is enough for that job, and it costs nothing per message. If a single miss causes real trouble, keep the larger model.

Can a small model run on your phone?

Yes, at least in testing. The Phi-3 report describes compressing Phi-3-mini through quantization, which stores each parameter with fewer bits so the model needs less memory, at a small cost in accuracy. At 4 bits, the model takes about 1.8 GB of memory. The team ran it on an iPhone 14 with Apple's A16 Bionic chip, entirely on the device and offline, producing more than 12 tokens per second.

That is a research demonstration, not an app you can download today. For now, a small model on a phone usually arrives inside an app built by a developer. Microsoft says Phi-3-mini is tuned for ONNX Runtime, a tool that lets the same model run on regular processors, graphics chips and mobile hardware, which is the kind of groundwork such apps need.

When small models do show up on phones, expect the same trade-off described above. You gain privacy, offline use and speed. You still need to check any fact the model states from memory, and a large model in the cloud remains the better choice for hard, open-ended questions.