What is the difference between GPT-3's Ada, Babbage, Curie and Davinci?

Anyone who opens the OpenAI Playground or reads the API pricing page meets four names in alphabetical order: Ada, Babbage, Curie and Davinci. They are not different products. They are four sizes of the same family of GPT-3 language models, and they let you choose how much capability you want to pay for.

The rule of thumb is simple. Moving up the alphabet buys more capability and costs more time and money. OpenAI's pricing page calls Ada the fastest model and Davinci the most powerful, and describes the four as a range of capabilities and price points.

The four models side by side

Here is how they compare, using OpenAI's published prices for the base models this month. Prices are per 1,000 tokens, where a token is a piece of a word; 1,000 tokens is about 750 English words.

You pay for both directions of each request: the tokens in your prompt plus the tokens in the reply. OpenAI's own example is a 10-token prompt and a 90-token answer from Davinci, which uses 100 tokens and costs $0.006.

The gap between the models is large. Curie costs a tenth of Davinci, and OpenAI notes that it can handle many of the same tasks faster. Ada costs one seventy-fifth of Davinci.

Base models and instruction models

Each name now covers two kinds of model. On January 20, OpenAI made its Instruct series, models tuned to follow written instructions, the recommended choice for natural language tasks. They have names like text-curie-001. The original base models, plain ada, babbage, curie and davinci, remain available, but OpenAI now recommends them mainly for fine-tuning (training a custom version on your own examples) and for a few specialized endpoints such as search and classification.

The newest member of the family arrived on March 15: text-davinci-002, which OpenAI calls the latest version of GPT-3 and which supports the new insert feature at the same price as earlier Davinci models.

In practice, if you are writing prompts by hand, start with the text- models. They respond to a plain request without needing a page of examples.

A worked test: the same task on two models

Prices on a table do not tell you whether a cheaper model is good enough for your job. The only way to know is to test. Here is a simple, repeatable comparison between Curie and Davinci for a common task: sorting customer messages.

Classify the customer message into one of these categories: Billing, Delivery, Product question, Complaint, Other. Answer with the category only.

Message: I ordered a lamp two weeks ago and the tracking page hasn't changed since Monday.
Category:

1. Collect 20 real messages and decide the correct category for each yourself before running anything.
2. Run the same prompt on text-curie-001 and text-davinci-001 with the temperature set to 0, so each model gives its most likely answer every time.
3. Score each model: how many of the 20 did it get right, and how many answers broke the format (for example, a sentence instead of a single word)?
4. Estimate the cost at your real volume. Say each request uses about 100 tokens and you expect 1,000 messages a month. That is 100,000 tokens, which at the listed per-model rates comes to about $6.00 on Davinci, $0.60 on Curie, $0.12 on Babbage and $0.08 on Ada.

If Curie scores 19 out of 20 and Davinci scores 20, the extra cost may not be worth it for this task. If Curie scores 14, it probably is. The numbers you get are what matter, because results depend heavily on the task and the prompt.

Which model should you pick for a given task?

Start with Davinci when you are exploring. It is the most forgiving model: if a prompt works anywhere, it is most likely to work there, so it tells you quickly whether a task is possible at all. Once your prompt works, test it on the smaller models and move down as far as quality allows.

As a rough guide, tasks that need careful reasoning, a precise reading of a complex instruction, or long, polished writing tend to call for Davinci. Many everyday jobs, such as summarizing a short text, classifying messages into a few clear categories or extracting a name and date, are worth trying on Curie first. Very simple, high-volume jobs, such as tagging large numbers of short items where speed matters, are where Babbage and Ada earn their place. OpenAI encourages developers to experiment and find the most efficient model for each application.

Price is not the only reason to step down. Smaller models answer faster, which matters in anything interactive, like a chat window or an autocomplete box. Smaller models can also be fine-tuned more cheaply, since training is billed at a share of each model's base price.

A final tip: keep your test set. When OpenAI adds or updates a model, as it did with text-davinci-002 this month, you can rerun the same 20 cases in a few minutes and see whether switching is worth it.