Why does GPT-3 make things up?

GPT-3 can answer many factual questions correctly, and the people testing it through OpenAI's API beta are often impressed. It can also state something false with complete assurance. Both behaviors come from the same source. GPT-3 is a language model: it learned to predict the next piece of text from a huge amount of writing, and it answers by continuing your prompt in the most plausible way. Sometimes the most plausible continuation is a fact it absorbed many times. Sometimes it is a guess that merely has the right shape.

There is no separate fact-checking step inside the model and no database it consults. It has no built-in signal that says "I am unsure", and it has no way of knowing what has happened since its training text was collected. OpenAI's model card for GPT-2, which works the same way, says such models do not tell fact from fiction, and that OpenAI does not support uses that need the text to be true.

What the GPT-3 paper says

The researchers who built GPT-3 are open about this. In their limitations section they note that long samples can repeat themselves, drift away from the topic, contradict themselves and sometimes include sentences that don't follow from what came before. On a quiz benchmark called TriviaQA, answered from memory with no reference material, GPT-3 scored 64.3% with no examples and 71.2% with a few, a strong result for this kind of test that still leaves roughly three questions in ten answered wrongly.

The paper also points out why false details appear in machine-written news. A model writing an article from a headline has no access to the actual facts behind it, or even to when it was written. So it supplies details that fit the style of a news story. Human readers in the paper's test could barely tell such articles from real ones, which is what makes invented details easy to miss.

A worked example: answers to impossible questions

In July, developer Kevin Lacker published a set of question-and-answer tests he ran with GPT-3. He started each session with a short prompt of correct sample questions and answers, then asked his own. Many replies were right. Then he asked questions with no valid answer:

Each answer follows the pattern of the prompt perfectly: a name where a name belongs, a number where a number belongs. The model completed the form of an answer without any check that a correct answer existed. As Lacker puts it, GPT-3 doesn't know how to say a question is nonsense, or how to say it doesn't know.

The same thing happens with requests for sources or life stories. Ask for "three studies that prove" a claim, or the biography of someone barely documented online, and the model will often produce something in the right format, with authors, titles, years or career highlights, whether or not real ones exist in its training text. The output looks like a citation because citations have a familiar pattern, and patterns are what the model learned.

How can you spot a confident wrong answer?

You usually can't tell from the writing itself. Fluency and confidence are exactly what the model is good at, and they appear whether the content is right or wrong. Instead, run three checks.

1. Check every specific. Names, dates, numbers, quotes and titles are where invented details hide. Look each one up in a source you trust, such as a library catalog, an encyclopedia or the original document. If a citation is given, search for the exact title. A study that cannot be found anywhere probably does not exist.

2. Ask again, differently. Rephrase the question, or run it several times at a normal temperature. A fact the model has seen often tends to come back the same way. An invented one often changes: a different year, a different author, a different middle name. Disagreement between runs is a strong warning sign.

3. Test whether the question has an answer at all. Ask yourself whether the premise is true before trusting the reply, as with a president in 1700. You can also build this into your prompt. Lacker noted that others got better results by including examples where the right answer is "unknown" or "that question doesn't make sense", which shows the model that declining is an acceptable continuation.

None of these checks makes GPT-3 a reliable reference. They make you a careful reader of it. For tasks where a wrong fact causes harm, such as health, money or law, treat its output as a first draft to verify, never as the answer itself.