# How do you read AI-generated text critically?

Date: 2020-10-06 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/read-ai-generated-text-critically/

**Answer:** Read AI-generated text by asking how it was made as well as what it says: who wrote the prompt, how many attempts were made, what a person chose or edited, and whether the claims hold up. Fluent machine writing can hide heavy human steering and unchecked facts, so the story behind the text matters as much as the text.

Text written by language models such as OpenAI's GPT-3 is now appearing in newspapers, blogs and social media, sometimes clearly labeled and sometimes not. It can be smooth, persuasive and grammatically clean. Those qualities say little about how much a machine really did, how many tries it took, or whether the content is true.

Reading such text well means adding a second layer of attention. You still ask what the text claims and whether the claims are supported. But you also ask how it came to exist. A widely discussed example from last month shows why both layers matter.

## The worked example: the Guardian's GPT-3 op-ed

On September 8, the Guardian published an opinion piece under the headline "A robot wrote this entire article. Are you scared yet, human?" The essay, credited to GPT-3, argues that people have nothing to fear from artificial intelligence. It reads well and makes a lively case.

The important part is at the bottom. An editor's note explains how the piece was produced:

- The Guardian wrote the instructions, asking for a short op-ed of about 500 words, in simple language, on why humans need not fear AI.
- It also supplied a ready-made opening paragraph that set out the argument the essay should make.
- Liam Porr, a computer science student at UC Berkeley, fed these prompts to GPT-3.
- GPT-3 produced eight separate essays.
- The editors took the best parts of each, cut lines and paragraphs, and rearranged some of them.

The note adds that this took less time than editing many human-written op-eds. Read with that information, the piece looks different. People chose the topic, the position, the opening and the length. A machine produced eight drafts. People then picked and stitched together the strongest passages. The published essay is a real collaboration, but "a robot wrote this entire article" is not an accurate description of it.

The same day, Thomas Macaulay at The Next Web pointed out that the newspaper did not publish the eight original outputs or list its edits. Without them, readers cannot judge how good GPT-3's raw writing was, or how much of the final quality came from the editors' choices. Mozilla fellow Daniel Leufer said the eight unedited essays would have been the more interesting thing to see.

## A second case: when there is no label at all

The Guardian at least disclosed its method. In August, MIT Technology Review reported on a blog that Porr himself had run under a made-up name, with posts generated by GPT-3 through a graduate student who had access. He gave the model a headline and an introduction, generated several versions, and published one with little or no editing. One post reached the top of Hacker News, a popular technology forum, and some readers subscribed.

Porr told the magazine he picked productivity and self-help topics on purpose, because GPT-3 is good at pleasant language and weak at strict logic. Among dozens of commenters on his top post, only three or four suggested a machine might have written it, and other readers voted those comments down.

Taken together, the two cases teach complementary lessons. Labeled text can still mislead through its framing. Unlabeled text can pass unnoticed, especially in genres that reward a warm tone over careful argument.

## Reading habits that help

- **Read the fine print first.** Editor's notes, captions and disclaimers often contain the most important facts about how a text was made.
- **Separate style from substance.** Ask what the text actually claims, sentence by sentence. Fluent paragraphs can carry vague or circular reasoning.
- **Watch for selection.** If a showcase presents the best of several outputs, you are seeing the top of the range, not the typical result.

> **Try it:** Find the Guardian op-ed and read only the essay. Write one sentence on how impressive you find it. Then read the editor's note at the bottom and write a second sentence. Compare the two, and list which facts in the note changed your view.

## What questions should you ask about any text a machine wrote?

Here is a five-question checklist you can use on any text that a model may have produced, whether it is labeled or not.

1. **Who set the task?** Look for the prompt or instructions. In the Guardian case, the newspaper chose the topic, the position and even the opening paragraph. A model's "opinion" often reflects the prompt it was handed.
2. **How many attempts were made, and who chose?** One output out of eight is a curated result. Ask whether you are seeing a typical sample or the best one.
3. **What did people change?** Cutting, reordering and rewriting can turn rough drafts into a polished piece. If the edits are not disclosed, treat claims about the machine's ability with caution.
4. **Are the facts and sources real?** Language models produce plausible text, not checked text. Verify names, numbers, quotes and references against sources you trust before repeating them.
5. **Who benefits from how it is presented?** A dramatic headline draws clicks; a hidden machine author can fill a blog cheaply. Ask why the text is framed the way it is, and what the framing leaves out.

None of these questions assumes machine-written text is bad. A clearly described, well-edited collaboration can be useful and worth reading. The aim is to know what you are reading, so you can judge it on accurate terms.

## Sources

- A robot wrote this entire article. Are you scared yet, human?: https://www.theguardian.com/commentisfree/2020/sep/08/robot-wrote-this-article-gpt-3
- The Guardian's GPT-3-generated article is everything wrong with AI media hype: https://thenextweb.com/news/the-guardians-gpt-3-generated-article-is-everything-wrong-with-ai-media-hype
- A college kid created a fake, AI-generated blog. It reached #1 on Hacker News.: https://www.technologyreview.com/2020/08/14/1006780/ai-gpt-3-fake-blog-reached-top-of-hacker-news/