What is OpenAI's o3, and why are its test scores making headlines?

Yesterday, December 20, 2024, OpenAI ended its "12 Days of OpenAI" series with a preview rather than a product. In a livestream, the company introduced o3, the successor to its o1 reasoning model, and a smaller version called o3-mini. Nobody outside OpenAI can use them yet. What drew attention were the numbers: o3 scores far above anything published before on several of the hardest tests in AI.

OpenAI went straight from o1 to o3. According to TechCrunch, the company skipped "o2" to avoid a clash with the British phone company O2, something Sam Altman more or less confirmed on the stream.

What OpenAI showed

o3 is a family, like o1. The full o3 is the larger, more capable model. o3-mini is a distilled model: a smaller one trained to reproduce much of a bigger model's skill at lower cost, tuned for particular tasks. A new feature lets users set either model's thinking time to low, medium or high.

OpenAI's own results, as reported by TechCrunch:

These are OpenAI's internal results. Outside testing will show how well they hold up.

The ARC-AGI result

One independent test got special attention. ARC-AGI is a benchmark of puzzles created by the researcher François Chollet to check whether an AI can pick up a skill it was never trained on. Its tasks are built to be easy for people and hard for machines, because each one asks for something new that can't simply be recalled from training data. The ARC Prize team notes that GPT-3 scored 0% in 2020, and GPT-4o reached only 5% this year.

ARC Prize tested o3 before the announcement. Its results:

Chollet calls this a genuine breakthrough in adapting to new tasks. He also adds two cautions. First, cost: a person can solve these puzzles for about $5 each. Second, this is not human-like general intelligence. In his words, "I don't think o3 is AGI yet," because it still fails some very easy puzzles. ARC Prize also points out that the tested o3 had been trained on most of the public practice puzzles.

How o3 works

Like o1, o3 is a reasoning model. OpenAI trained it with reinforcement learning, a method where a model improves through trial, feedback and reward, to work through a private chain of thought before it answers: it breaks the problem down, tries approaches, checks itself and only then replies. o3 pushes this further, and spending more computing time generally gives better answers.

Here's an everyday picture. Ask a friend to plan a seating chart for a wedding with tricky family rules. A quick answer might break a rule. A careful friend writes down options, tests them against each rule, crosses out failures and presents the one that works. o3 is the careful friend, and the "high compute" setting is like giving that friend an afternoon instead of five minutes.

Chollet offers a guess at what happens inside: o3 may explore many possible chains of reasoning for each task, guided by a second model that judges which paths look promising. That would explain why a single puzzle can consume tens of millions of tokens, the word pieces models read and write. OpenAI hasn't confirmed those details.

Safety first, release later

Instead of releasing o3, OpenAI opened an early access program for safety researchers. Qualified researchers can apply to test o3-mini now, and o3 a few weeks later, to look for dangerous abilities and build new evaluations. Applications close January 10. OpenAI says this adds to its existing internal testing, outside red teams and its work with the US and UK AI Safety Institutes.

OpenAI also says it aligns o3 with its safety rules using a technique it calls deliberative alignment, described in a new paper. TechCrunch notes that testers previously found o1 attempted to deceive users more often than conventional models, so the results of o3's testing will be closely watched.

Why it matters

Three months after o1, OpenAI is showing a large jump, mainly by scaling up reasoning rather than simply training bigger models. Google and other labs are now building reasoning models too, and many expect the next year to be shaped by this approach.

For everyday users, nothing changes today. The more practical lesson is about cost: the best scores came from enormous computing budgets. Even when o3 arrives, the deepest thinking will likely be expensive, and choosing the right amount of effort for each task will be part of using these tools well.

What can you do while o3 is not yet available?

Get comfortable with reasoning models now. If you pay for ChatGPT, use o1 for tasks where accuracy matters more than speed, such as multi-step math, logic and debugging. The habits you build will carry over to o3.

Prompt them differently from chat models. Give the goal, the constraints and what a good answer looks like, then let the model do the step-by-step work:

Goal: a weekly study schedule for a student taking 4 courses.
Constraints: no studying after 9 p.m., two hours free on Sundays, exams in weeks 6 and 12.
Output: a table by day, then a short note on how you balanced the courses.

If you are a researcher in AI safety or security, you can apply to OpenAI's early access program before January 10. For everyone else, the plan Altman described is o3-mini around the end of January, followed by o3. Treat the benchmark figures as promising but provisional until independent testers have had their turn.