OpenAI released the full GPT-2: what changed?

Yesterday OpenAI released the largest version of GPT-2, its text-generating language model with 1.5 billion parameters. This is the last step of a "staged release" that started in February, when the lab announced GPT-2 but kept the full model private out of concern about misuse. Along with the model weights (the trained numbers that make the model work), OpenAI is publishing a detection model to help spot GPT-2 text and a "model card" that describes the model's limits.

For anyone curious about language models, this is a big moment. Until now, outsiders could only use the smaller versions. Now the same model OpenAI showed in February is available to download and run.

How the staged release unfolded

OpenAI released GPT-2 in four sizes over nine months. Note that the lab has corrected its original parameter counts, so the first two sizes have new names:

In August, along with the 774M model, OpenAI announced research partnerships with four groups: Cornell University on how people respond to generated disinformation, the Middlebury Institute's Center on Terrorism, Extremism, and Counterterrorism (CTEC) on extremist misuse, the University of Oregon on bias, and the University of Texas at Austin on detection. It said the full model would follow in a few months unless something changed its mind.

What OpenAI says it learned

The release post lists five findings:

• People find the text convincing. In a Cornell survey, readers rated the 1.5B model's text 6.91 out of 10 for credibility, slightly above the 774M model (6.72) and clearly above the 355M model (6.07). Since the jump from the previous release was small, OpenAI felt more comfortable publishing.
• It can be fine-tuned for misuse. CTEC showed that GPT-2 can be retrained to produce propaganda for extremist ideologies.
• Detection is hard. OpenAI's own detector, built on a model called RoBERTa, catches about 95 percent of text from the full GPT-2. The lab says that is not good enough on its own and should be paired with human judgment and other signals, such as where a text came from.
• No strong evidence of misuse so far. OpenAI acknowledges it cannot see every threat, and that determined groups could build similar models themselves.
• Bias needs standards. OpenAI probed GPT-2 for gender, race and religious biases, and argues the research community needs shared ways to study them.

OpenAI frames the whole process as a test case for future powerful models and says it is continuing the conversation through the Partnership on AI's work on responsible publication norms.

What it means for builders and learners

Readers find the full model's text a little more believable than the smaller ones, but it keeps the same weaknesses. OpenAI's repository warns that the models can be incoherent or inaccurate in subtle ways, that they likely carry the biases and errors of their web training data, and that samples are unfiltered and may be offensive. It recommends clearly labeling generated text as synthetic before sharing it widely.

How can you try GPT-2 yourself?

The simplest route is a website that runs the model for you. OpenAI's August update pointed to TalkToTransformer.com, a site built by Adam King that lets you type a prompt in your browser and see GPT-2 continue it. No installation is needed, which makes it a good first look.

If you are comfortable with a command line, you can run the model on your own computer from OpenAI's GitHub repository. The steps in the developer notes are:

git clone https://github.com/openai/gpt-2.git && cd gpt-2
pip3 install tensorflow==1.12.0
pip3 install -r requirements.txt
python3 download_model.py 1558M
python3 src/interactive_conditional_samples.py --top_k 40

The download step fetches the model by size name (124M, 355M, 774M or 1558M), so you can start small if your machine is modest. Run the sampling script with -- --help to list its options before your first run. The last command opens an interactive prompt: type a few sentences, press Enter, and the model writes a continuation. The --top_k setting limits each choice to the 40 most likely next tokens, which tends to keep text more coherent; a --temperature flag makes output more or less adventurous. A graphics card helps a lot with the larger models.

Whatever route you choose, treat the output as a draft written by a very fluent guesser. It does not check facts, and it will confidently invent names, numbers and quotes. That makes it a fine tool for exploring how language models work, and a poor source of information.