Short answer
Pick three or four candidate models, one per role you need (a strong model for hard work, a mid-priced one for daily work, a small one for volume), then run the same ten to twenty of your own tasks through each and record quality, cost and time on one sheet. Choose per job, not per vendor, and keep the sheet so you can rerun it when something changes.
Follow-up question How often should you re-test?
In the past six weeks alone, new models have arrived from OpenAI, Anthropic, Meta, Google and Mistral. Each launch comes with charts showing it on top of something. No one can try them all on everything, and nobody needs to. What you need is a shortlist: a small set of models you have tested on your own work, with a clear note of which one you use for what.
The method below takes an afternoon. It turns "which AI is best?" into a question you can answer with your own numbers.
Step 1: shortlist by role, not by brand
Most people need models for three kinds of work, and each major lab now offers a tier for each:
| Role | What it does | Examples available today |
|---|---|---|
| Heavy | Long, open-ended or high-stakes work | Claude Opus 5.5, GPT-6 Astra, Gemini 4 Argon, Muse Wonder |
| Daily | Most writing, analysis and coding | Claude Sonnet 5.5, GPT-6.1 Sol |
| Volume | Sorting, extracting, short summaries at scale | Claude Haiku 5.5, GPT-6 Luna |
Pick one or two candidates per role that you can actually reach: through a subscription you already have, an API account, or a free trial. If you need to run a model on your own servers, add an open-weight candidate such as Mistral Large 4 once its weights are out. Four to six models in total is plenty.
Step 2: choose your test tasks
Take ten to twenty real tasks from the past month, spread across the roles. Write down, before you run anything, what a good answer must contain. If you have never built a test set, start with the guide to building an eval set from your own work, linked below; it covers rubrics and grading in more depth.
Step 3: run the bake-off
Run every task through every candidate, in a fresh conversation each time, with the same prompt. Record four things: a quality score from 1 to 5, the time to a usable answer, the cost, and the number of fixes you had to make.
Cost needs care. The price per token is only half of it, because models use very different amounts of text to finish the same job. Artificial Analysis, which runs its own test suite on many models, reported average costs per task such as $5.98 for Claude Opus 5.5, $5.46 for Claude Sonnet 5.5, $4.52 for Muse Wonder, $3.26 for GPT-6 Astra and $1.99 for Gemini 4 Argon, according to Latent Space. Some models with high per-token prices finish in fewer tokens. Your own sheet will show how that plays out on your tasks.
Here is a sheet for a single daily task, "summarize a 10-page report into five bullet points for a manager." The figures are illustrative, to show the format.
| Model | Quality (1–5) | Time | Cost per task | Fixes needed |
|---|---|---|---|---|
| Candidate A (heavy) | 5 | 70 s | $0.21 | 0 |
| Candidate B (daily) | 5 | 25 s | $0.06 | 0 |
| Candidate C (daily) | 4 | 30 s | $0.05 | 1 |
| Candidate D (volume) | 3 | 6 s | $0.01 | 2 |
On this one task, Candidate B wins: the same quality as the heavy model for less money and less waiting. That is the most common outcome of a bake-off, and it is why a shortlist pays for itself.
Try it: Pick three tasks you did this week and two models you can access. Run each task on both, fill in the four columns above, and decide which model you would use for each task if you had to choose today.
Step 4: make the comparison fair
A few habits keep a bake-off honest. Anthropic's research team recommends comparing models question by question, since they all answer the same set; looking at the difference on each task is more precise than comparing two averages. If answers vary from run to run, run each task two or three times and average the scores. And remember that a small set only detects large gaps. With fifteen tasks, a one-point difference on a few of them may be luck.
Treat outside rankings as a starting point. Artificial Analysis, for example, combines ten different tests into one index, and other testers weigh other skills, so the same model can sit higher on one list than another. Your sheet tests the one thing that matters to you.
Step 5: write down the decision
End with one line per role: "Heavy: model X. Daily: model Y. Volume: model Z." Add a note of the price you paid and the date. That page is your shortlist.
How often should you re-test?
Rerun the sheet whenever something on it changes. Three triggers matter most.
A new model or version in one of your roles. When a lab updates a model you rely on, or a rival releases one in the same tier, run your tasks again before switching. Previews deserve a second look too, since they can change while training continues.
A price change. Prices move quickly. Anthropic, for instance, cut the per-token price of its small model by 90% for shorter requests with Claude Haiku 5.5, and halved Sonnet 5.5's cache-read price the same day. A change like that can flip a decision without any change in quality.
A change in your own work. New kinds of tasks, longer documents or a new language deserve fresh test cases.
Between those events, a light quarterly rerun keeps the sheet honest. Because your prompts and scoring are already written down, a rerun takes an hour, not an afternoon, and it tells you whether to stay put or move.
Sources
- A statistical approach to model evaluations — Anthropic, November 19, 2024
- Meta Muse Wonder & Wonder Full: MSL's Claude 5.5-tier Frontier Model — Latent Space (AINews), October 13, 2026
- Introducing Claude Haiku 5.5 — Anthropic, October 7, 2026