What is GPT-4o mini, and why does a small, cheap model matter?

Yesterday, Thursday, July 18, 2024, OpenAI released GPT-4o mini, a smaller and much cheaper member of the GPT-4o family it introduced in May. OpenAI calls it "our most cost-efficient small model." For people who use ChatGPT, the change is immediate: GPT-4o mini takes the place of GPT-3.5, the model that has quietly answered free users' questions since ChatGPT first appeared.

For developers, the headline is the price. A model that scores well on standard tests and costs pennies per million words changes which projects make financial sense. That is the real story behind this release.

What OpenAI released

GPT-4o mini is available in two places from launch day:

• In ChatGPT, for Free, Plus and Team users, in place of GPT-3.5. Enterprise customers get it next week.
• In the API, the service programmers use to connect OpenAI's models to their own software. It works in the Chat Completions API, the Assistants API and the Batch API.

The model reads text and images today. OpenAI says support for audio and video, both as input and output, will come later. It can consider up to 128,000 tokens at once (a token is a word or a piece of a word, so that is roughly a book's worth of text), write up to 16,000 tokens in a single reply, and knows about events up to October 2023. Fine-tuning, which lets a company train the model further on its own examples, is promised within days.

OpenAI has not said how large GPT-4o mini is. TechCrunch reports the company places it in the same class as other small models such as Meta's Llama 3 8B, Anthropic's Claude Haiku and Google's Gemini 1.5 Flash.

What it costs, and what it scores

Prices for AI models are quoted per million tokens, split between what you send (input) and what the model writes back (output). Here is how GPT-4o mini compares on OpenAI's own figures:

OpenAI says one million tokens is about 2,500 pages of a standard book. On its tests, GPT-4o mini beats Gemini Flash and Claude Haiku on all four measures above. OpenAI also points out that each token now costs 99% less than with text-davinci-003, a weaker model it offered in 2022.

Speed matters too. Artificial Analysis, an independent testing firm, told TechCrunch it measured a median of 202 tokens per second, more than twice as fast as GPT-4o and GPT-3.5 Turbo.

How a "mini" model works

Think of a large model and a small model as two cooks trained in the same kitchen. The head chef knows more dishes and handles unusual requests, but is expensive and slow to book. The line cook has learned the house recipes well and turns out the everyday orders quickly. Most orders are everyday orders.

A small model has fewer parameters, the internal numbers a model adjusts during training to store what it has learned. Fewer parameters means less computing power per answer, which is why it is cheaper and faster. The trade-off is depth: on hard or unusual problems, a larger model usually does better.

GPT-4o mini also shares GPT-4o's improved tokenizer, the component that splits text into tokens. Because it needs fewer tokens for many non-English languages, the same paragraph in, say, French or Hindi now costs less to process.

On safety, OpenAI says the model carries the same protections as GPT-4o and is the first to use a method it calls the instruction hierarchy. The idea is that the model learns to give the developer's instructions priority over text that tries to override them, which helps against jailbreaks (prompts designed to get around the rules) and prompt injections (hidden instructions planted in a web page or document the model reads).

Why it matters this week

For builders, low cost opens up patterns that were too expensive before. OpenAI names three: apps that make many model calls in a row or in parallel, apps that send a lot of context with each request (such as a whole code base or a long conversation history), and fast customer-support chat. Early partners Ramp and Superhuman reported that GPT-4o mini did much better than GPT-3.5 Turbo at pulling structured data out of receipts and drafting email replies from a thread.

Olivier Godement, who leads OpenAI's API product, framed it to TechCrunch as a question of reach: AI only gets to every corner of the world if the models become far more affordable.

For everyday ChatGPT users, the free tier simply gets a stronger default model. Questions that GPT-3.5 fumbled, especially math and code, should now go better, at no extra charge.

How can you start using GPT-4o mini today?

If you use ChatGPT on the Free, Plus or Team plan, you are already using it: GPT-4o mini replaces GPT-3.5 from launch day, with no setting to change. Give it clear instructions, the audience, the length and the format you want, and check facts that matter, as you would with any model.

If you write code, change the model name in an existing project to gpt-4o-mini and run your usual test examples. Because the price is so low, it is worth trying for jobs you had kept on GPT-3.5 Turbo, and for high-volume tasks such as sorting support messages, summarizing documents or extracting fields from forms. Here is a simple extraction prompt to test it with:

Read the receipt text below. Return the store name, the date, the total and the payment method, one per line. If a value is missing, write "not found".

[paste receipt text here]

Keep larger models for the hardest questions, and compare results on your own examples before you switch anything important. A small model that is right most of the time is a good deal only if you know when it is wrong.