How do you choose an AI model for a task in mid-2024?

Three companies now offer models at the top level, and each sells them in more than one size. OpenAI has GPT-4o, released in May. Anthropic has the Claude 3 family from March: Haiku, Sonnet and Opus, from fastest to most capable. Google has Gemini 1.5 Pro and, since I/O in May, the lighter Gemini 1.5 Flash. All of them write well, read images and handle long documents. The differences show up in the details of your task.

The good news is that you don't need to find the single best model. You need the one that does your job well enough, at a price and with privacy terms you can live with. That is a question you can answer with a short, structured test.

Five questions to ask first

1. Quality. How hard is the task? Drafting an email is easy for any of them. Analyzing a long contract or writing tricky code is where the larger models earn their price.
2. Cost. Are you using a chat app with a monthly fee, or paying per use through an API, the programming interface developers use? API prices are counted in tokens, the word pieces models read and write.
3. Privacy. What will you paste in? Read each provider's data terms for the plan you use, and check whether your conversations can be used for training and whether you can turn that off.
4. Context. How much text must the model read in one go? This limit is the context window, measured in tokens.
5. Tools. Do you need to upload files, see images, browse the web or talk by voice?

A decision table for mid-2024

Figures come from each company's own announcements and will change. For privacy, there is no single answer per company: the terms for a free chat account, a business plan and the API can all differ.

A worked bake-off: three prompts, three models

Suppose you write a monthly newsletter for a regional tourism office and want to pick a model for it. Choose three prompts that reflect the real work, and use the same wording on each model.

Prompt 1, your everyday task:

You are an editor for a regional tourism newsletter read by families and retirees.
Rewrite the text below as a 150-word article with a headline, in a warm, plain style. Keep every date, place and price exactly as given.

[paste last month's raw notes]

Prompt 2, a fact you already know: "List the three largest lakes in our region, with their surface area." Choose a question whose answer you can check, so you can see which model invents numbers.

Prompt 3, a format test: "Turn this list of 12 events into a table with columns Date, Event, Place and Price, sorted by date. Don't add events."

Score each answer from 0 to 2 on accuracy, following instructions and tone, and note speed and cost. A filled-in sheet might look like this (illustrative scores):

Close totals are common. When that happens, cost, privacy terms and the tools you need usually decide.

Which tests should you run before you commit?

Test with your own material. Public benchmark scores tell you which model is strong on average; they do not tell you which one handles your documents, your vocabulary and your audience. Three to five prompts taken from real work are worth more than any leaderboard.

Include a trap. At least one prompt should have an answer you already know, preferably a specific number or name. Models differ in how often they fill gaps with confident guesses, and that is the failure you most need to see before you rely on one.

Test the limits you will hit. If you plan to feed it 80-page reports, test an 80-page report, and ask about a detail near the end. If you will paste in client information, read the privacy terms for the exact plan you will use before the first real document goes in.

Run each prompt more than once. Answers vary from one run to the next, so a single good reply can be luck. Two or three runs show you whether a model is consistently good or just occasionally impressive. Then repeat the test every few months, because the models, their prices and their terms are changing quickly.