# Why do chatbots invent citations?

Date: 2023-06-11 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/chatbots-invent-citations/

**Answer:** A chatbot like ChatGPT writes text by predicting likely words, not by looking references up in a library, so it can produce a citation with the right shape, plausible authors and a believable year that points to nothing. Asking the chatbot whether its own reference is real does not help; only checking an outside database does.

At the end of May, a New York lawsuit against the airline Avianca became the clearest lesson yet in what a chatbot citation is worth. Lawyers for the passenger filed a brief citing earlier court decisions to support their argument. The airline's lawyers could not find them. Judge Kevin Castel concluded that six of the cited cases appeared to be fake, complete with invented quotes, and called a hearing on possible sanctions.

The lawyer who did the research, Steven Schwartz, explained in a sworn statement that he had used ChatGPT. He had even tried to check: he asked the chatbot whether one case was real, and it said yes and claimed it could be found in the major legal databases Westlaw and LexisNexis. He said he had not known its content could be false.

## Why it happens

A large language model, the technology behind ChatGPT, is trained on huge amounts of text to predict which words are likely to come next. It does not keep a catalog of documents it can fetch, and unless you switch on a browsing feature, ChatGPT is not connected to the internet. OpenAI's own help pages warn that ChatGPT will occasionally make up facts, which the company calls hallucinating.

Citations are especially easy to fake because they follow a strict pattern. A court case looks like "Name v. Name, volume, reporter, page (court, year)." A journal article has authors, a title, a journal, a year, a volume and pages. The model has seen millions of these. When you ask for a source, it can produce text that fits the pattern perfectly, with surnames that sound right for the field and a journal that really exists, without any real article behind it. Sometimes it blends real pieces: one invented case in the Avianca brief appeared to borrow from a genuine airline case but gave the wrong date and other details.

This is also why asking "Is that real?" fails. The chatbot answers that question the same way it answered the first one, by producing likely text, and "Yes, it is real" is a likely reply.

## How often it happens

A study published in the journal Cureus on May 19 put numbers on the problem for medical writing. The authors asked ChatGPT (the GPT-3.5 version) to write 30 short medical papers with references, then searched for all 115 references in Medline, Google Scholar and a directory of open-access journals.

| Result | Share of the 115 references |
|---|---|
| Fabricated: no such article | 47% |
| Real but with wrong details | 46% |
| Real and accurate | 7% |

The most common error was a wrong PubMed identifier, the PMID number that labels each article in the main medical database; it was wrong in 93% of the papers. Wrong volumes, page numbers and years were next. This is one small study of one model version, but the direction is clear: most references were unusable as given.

## A worked three-step check

Suppose ChatGPT gives you this reference in an answer about sleep and memory. We invented it for this exercise: the authors and the article do not exist, even though the journal does, which is exactly how fabricated references tend to look.

> Martin, L., & Okafor, D. (2019). Sleep spindles and overnight memory consolidation in older adults. *Journal of Sleep Research*, 28(4), 412–420.

1. **Find it independently.** Search the exact title in quotation marks in Google Scholar or, for medicine, PubMed. Do not click a link the chatbot gave you; links can be invented too. If nothing with that title appears, treat the reference as fake.
2. **Match every detail.** If a similar article appears, compare authors, year, journal, volume and pages. A real paper with different authors or a different year is a warning sign that the chatbot stitched pieces together.
3. **Read the source.** Open the abstract or full text and confirm it actually says what the chatbot claimed. A real paper cited for a finding it does not contain is still a wrong citation.

> **Try it:** Ask ChatGPT for three peer-reviewed references on a topic you know, such as a hobby or your field. Run each through the three steps above and record how many pass all three.

## How do you check that a reference really exists?

Use the database built for that kind of source, not the chatbot and not a general web page that repeats it. For research articles, Google Scholar works across fields and PubMed covers medicine. For court cases, use an official court site or a legal database such as Westlaw or LexisNexis. For books, a library catalog or the publisher's site. If the reference includes a DOI, a permanent identifier for academic papers, paste it after doi.org and see whether it opens the article named.

Check every element, not just the title, and keep a note of where you confirmed each one. If you cannot find a reference after a reasonable search, do not cite it. Find a real source that supports the point, or drop the point.

A better habit is to reverse the order: find sources first, then use a chatbot to help you summarize or organize what you have actually read. Chat tools that search the web and link their sources make the first step easier, but their links still need the same three checks before anything goes into a report, a school paper or a court filing.

## Sources

- A lawyer used ChatGPT and now has to answer for its 'bogus' citations: https://www.theverge.com/2023/5/27/23739913/chatgpt-ai-lawsuit-avianca-airlines-chatbot-research
- High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content: https://doi.org/10.7759/cureus.39238
- ChatGPT General FAQ: https://help.openai.com/en/articles/6783457-chatgpt-general-faq