How does Bard with Gemini Pro compare with ChatGPT?

Google gave Bard what it calls its biggest upgrade on December 6: in English, the chatbot now answers with a specially tuned version of Gemini Pro, the middle model of the new Gemini family. That changes the comparison many people made in the spring, when Bard often felt a step behind ChatGPT. The question now is less "which is better?" and more "better at what, and against which version of ChatGPT?"

The answer depends on matching like with like. Bard is free. ChatGPT comes in two forms: the free version, built on a GPT-3.5 model, and ChatGPT Plus, a $20-a-month subscription that adds GPT-4 with a cap on how many messages you can send. Comparing free Bard with paid GPT-4 is like comparing a good family car with a sports car; both are useful, but they are not the same purchase.

What Google is claiming

Google ran Gemini Pro through eight widely used benchmarks (standard test sets that let researchers compare models) before putting it in Bard. It reports that Gemini Pro came out ahead of GPT-3.5 on six of them, including MMLU, a broad exam covering dozens of school and professional subjects, and GSM8K, a set of grade-school math word problems. Google also says it tuned this version of Gemini Pro specifically for understanding, summarizing, reasoning, coding and planning, and that outside raters in blind tests preferred Bard over other free chatbots.

Two cautions apply. First, these are the company's own numbers; independent results take time to appear. Second, a benchmark measures a narrow skill under fixed conditions, while your daily use mixes many skills with your own phrasing. A small lead on a test does not guarantee a better answer to your question.

For now, Bard with Gemini Pro handles text prompts, and Google says other kinds of input will follow. English speakers in over 170 countries and territories can use it now, and Google says Europe and other languages will follow. The largest model, Gemini Ultra, is not in Bard yet; Google says it will arrive in a separate Bard Advanced experience in the first part of 2024, after more safety checks.

Where ChatGPT stands

On OpenAI's side, the main dividing line is GPT-3.5 versus GPT-4. When OpenAI introduced GPT-4 in March, it said the difference can be hard to notice in casual chat but shows up on complex tasks, where GPT-4 is more reliable, more creative and better at following detailed instructions. On a simulated bar exam, for example, GPT-4 landed around the top 10 percent of test takers, while GPT-3.5 landed around the bottom 10 percent. Both models can still state wrong facts with confidence, which OpenAI says plainly.

So a realistic picture for a beginner looks like this:

A side-by-side test you can run

Open Bard and ChatGPT in two browser tabs and paste the same prompt into each. Use tasks you actually care about; here are three that cover different skills.

Task 1 (summarizing): Summarize the text below in five bullet points for a busy manager. Keep every number exactly as written.
[paste a 400-600 word article or report]

Task 2 (planning): Plan a three-day visit to a city of your choice for two adults on a budget of $600 excluding travel. One of us uses a wheelchair. Give a day-by-day table and flag anything I should confirm before booking.

Task 3 (reasoning): A recipe for 6 people uses 450 g of flour and 3 eggs. I need to cook for 15. How much of each do I need? Show your working, and tell me how to handle half an egg.

Score each answer from 1 to 5 on four points: accuracy (check the numbers yourself), following every instruction, clarity, and how much you would need to edit. The scores below are an illustrative example of how a filled-in sheet might look, not real results.

Which tasks suit each one?

For quick summaries, rewrites and everyday questions, free Bard with Gemini Pro and free ChatGPT are close enough that personal preference and habit can decide. If Google's benchmark lead holds up, Bard may have an edge on reasoning and planning among free tools, so it is worth testing first for itineraries, step-by-step plans and simple math. Always check the arithmetic either way.

For long, complex instructions, careful writing in a specific style, or tasks where small errors matter, GPT-4 in ChatGPT Plus is still the stronger choice of the three today, based on OpenAI's published testing. It costs money and has a message cap, so it makes sense to save it for the work that needs it.

The most practical habit is to keep both free tools open for a week and send your real tasks to each. When two good models disagree, that disagreement is useful: it tells you exactly which facts to verify before you rely on either answer.