Short answer
GPT-2 is a new OpenAI language model with 1.5 billion parameters, trained to predict the next word on about 40 GB of web text, that can write coherent paragraphs on almost any topic you start it with. OpenAI is holding back the full model because it worries the text could be misused for fake news, impersonation or spam, so for now it is releasing only a much smaller version, sample code and a research paper.
Follow-up question What does a staged release mean for researchers and the public?
Yesterday the research lab OpenAI announced GPT-2, a language model that writes surprisingly convincing text. Give it a sentence or two, and it continues in the same style for paragraphs, keeping track of names and topics far better than earlier systems. It also scores at the top of several language tests without being trained for them.
The surprise is not only what GPT-2 can do but what OpenAI is choosing not to share. The lab is keeping the full model, its training data and its training code private. It is publishing a much smaller model, code to generate samples, and a technical paper, and calls the decision an experiment in responsible disclosure.
What GPT-2 is
A language model is a program trained to guess the next word in a piece of text. GPT-2 does only that, yet doing it well across millions of documents forces it to pick up grammar, facts, styles and some reasoning along the way.
Some numbers from the paper and the announcement:
| Detail | GPT-2 |
|---|---|
| Size | 1.5 billion parameters (the adjustable numbers inside the model) |
| Training text | About 40 GB from roughly 8 million web pages, called WebText |
| How pages were picked | Links shared on Reddit that received at least 3 "karma" points, a rough sign that people found them worthwhile |
| Design | A Transformer, the attention-based network design from Google's 2017 paper |
| Reading window | 1,024 tokens (words or parts of words) |
OpenAI describes GPT-2 as its first GPT model from last year made much bigger: over ten times the parameters, trained on over ten times the data. The research team trained four sizes, from 117 million to 1.5 billion parameters, and found that performance kept improving as size grew.
What it can do, and where it fails
OpenAI's showcase sample begins with a human-written prompt about scientists who discover a herd of unicorns in the Andes that speak English. GPT-2 continues with a news-style story, complete with an invented scientist and quotes. It is fluent and mostly consistent.
The lab is candid about weaknesses. Good samples take a few tries. On topics that appear often on the web, the model gives reasonable output about half the time; on technical or obscure subjects it does poorly. Failures include repeated text, odd jumps between topics and errors about how the world works, such as fires burning under water.
Without any extra training, GPT-2 also attempts reading comprehension, translation, question answering and summarization, though it still falls short of systems built for each task. The researchers call this the "zero-shot" setting: the model is tested on a task it was never trained for.
Why OpenAI is holding it back
OpenAI lists good uses it expects, including writing assistants, better dialogue systems, translation and speech recognition. It also lists worries: misleading news articles, impersonating people online, and automating fake or abusive posts and spam. In the announcement it argues that the public will need to grow more skeptical of text found online, much as "deep fake" videos call for skepticism about images.
The lab admits that others could reproduce its results. Its argument is that a limited release buys time for the research community to discuss what responsible publishing should look like, as fields such as biotechnology and cybersecurity already do. OpenAI also says it is unsure this is the right decision and promises to revisit the strategy in six months.
Try it: Write the first two sentences of a short, harmless news story about something unusual in your town. Then write, yourself, the next paragraph you think a text generator would add. Ask a friend which parts sound machine-made, and why.
What does a staged release mean for researchers and the public?
A staged release means sharing a powerful system in steps rather than all at once. Here, step one is the small model, the sampling code and the paper. OpenAI has not committed to dates for anything larger; it says it will discuss the strategy again in six months, after watching how the small model is used.
For researchers, the trade-off is real. The paper gives enough detail to understand the method, and the small model allows hands-on experiments, but no outside team can test the full 1.5-billion-parameter model's claims directly. Expect a lively argument in the coming weeks: caution about misuse on one side, the research tradition of open publication on the other. Either way, OpenAI has opened a public discussion about when, and how, AI research should be published.
For the public, the practical message is simple. Fluent writing is no longer proof that a person wrote it. As text generators improve, checking where a story comes from matters more than how polished it sounds. If you are curious, the small model and code are already available to experiment with, and they are a good way to see both the strengths and the strange mistakes of this new kind of system.
Sources
- Better Language Models and Their Implications — OpenAI, February 14, 2019
- Language Models are Unsupervised Multitask Learners — OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever), February 14, 2019