What is GPT-4, and what can it do that GPT-3.5 can't?

Yesterday OpenAI announced GPT-4, the model that follows GPT-3.5, the one behind the free version of ChatGPT. It is available today in two places: inside ChatGPT for people who pay for ChatGPT Plus, with a limit on how many messages you can send, and to developers through an API (a way for programs to call the model) that has a waitlist.

The headline claim is easy to repeat and easy to misread. OpenAI says GPT-4 reaches human-level performance on a range of professional and academic tests, while also being less capable than people in many real situations. Both halves matter.

What is new in GPT-4

OpenAI calls GPT-4 a multimodal model: it can take images as well as text as input, and it answers in text. In the announcement it describes a photo of an odd phone charger panel by panel and explains why the picture is funny. Image input is not open to the public yet; OpenAI is starting with a single partner.

The other big change is quality on demanding work. OpenAI tested both models on exams written for people, without special training for them. On a simulated bar exam, the test lawyers take, GPT-4 scored around the top 10% of test takers; GPT-3.5 scored around the bottom 10%. OpenAI also reports that GPT-4 scored 40% higher than its latest GPT-3.5 on internal tests designed to catch false statements.

Developers get more room as well. The standard API version can handle 8,192 tokens (pieces of words) of text at once, and a version with 32,768 tokens, about 50 pages, is available to a limited group.

Where you will notice the difference

OpenAI is candid that in a casual chat the two models can be hard to tell apart. The gap shows when a task gets complex. Here is a rough guide based on what OpenAI describes:

Task	GPT-3.5	GPT-4
Quick question, short email	Fine	Often a similar result
Long instructions with many conditions	Often drops some	Follows them more closely
Multi-step reasoning, exam-style problems	Frequent mistakes	Noticeably stronger
Staying in a style or role you set	Drifts more	OpenAI reports better steerability
Images as input	No	Announced, not yet public

That third row is a good test. Try a prompt with several constraints at once:

Prompt

You are an experienced travel planner. Plan a 3-day trip to Montreal for two adults in late April with a total budget of 900 Canadian dollars, excluding the train. One of us is vegetarian and neither of us drives. Give a day-by-day table with morning, afternoon and evening, an estimated cost for each line, and a final total. If something doesn't fit the budget, say so instead of hiding it.

A weaker model tends to forget the vegetarian meal, the no-car rule or the running total. A model that tracks every condition is what OpenAI means by handling more nuanced instructions. Either way, check the prices yourself: the model does not know current prices.

What has not changed

OpenAI's own limitations section reads much like the one for earlier models. GPT-4 still invents facts and makes reasoning errors; OpenAI's phrase is that "it still is not fully reliable." Its knowledge mostly ends in September 2021, it does not learn from your conversations, and it can be confidently wrong, or too easily persuaded by a false statement you give it. OpenAI recommends extra care, such as human review, for anything high-stakes.

One feature to watch is what OpenAI calls steerability. Developers can now set the model's style and task in a "system" message, a standing instruction that sits above the conversation. One example in the announcement is a tutor that never gives the answer and only asks guiding questions. OpenAI says ChatGPT users will get this kind of control soon.

Try it: Take a prompt that GPT-3.5 handled poorly for you, one with several rules or steps. Write down which rules it missed. If you have access to GPT-4, run the identical prompt and compare the misses line by line.

Is ChatGPT Plus worth paying for to get GPT-4?

ChatGPT Plus costs $20 a month. Before GPT-4, the plan promised access when the free service is busy, faster responses and early access to new features; OpenAI says free ChatGPT stays available. Now Plus subscribers also get GPT-4, with a usage cap that OpenAI will adjust to demand. The company warns that capacity will be severely limited at first, so expect the cap to be tight in the coming weeks.

It is worth it if your work involves long, detailed instructions: drafting reports, structured planning, code, analysis or learning hard material where reasoning errors cost you time. In those cases the gap in instruction following is the main thing you are paying for.

It is probably not worth it yet if you mostly ask quick questions, write short messages or brainstorm. GPT-3.5 handles those well, and the GPT-4 message limit means you would not use it for everything anyway.

A sensible approach is to keep a short list of tasks where the free version let you down, subscribe for a month, and test those tasks. If GPT-4 fixes most of them, keep it. OpenAI also says it hopes to offer some free GPT-4 queries at some point, so waiting is a reasonable choice too.