# What is InstructGPT, and why does it follow instructions better?

Date: 2022-01-28 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-instructgpt/

**Answer:** InstructGPT is a family of GPT-3 models that OpenAI fine-tuned with human feedback, and as of January 27, 2022 they are the default models in the OpenAI API. They follow instructions better because people showed and ranked good answers during training, instead of the model learning only to predict the next word of internet text.

Yesterday OpenAI announced that its InstructGPT models are now the standard language models behind its API. If you have used GPT-3 through the API, you have probably noticed its quirks: ask it to "explain the moon landing to a 6 year old" and it may answer with a list of similar requests, as if it were continuing a page of homework prompts. InstructGPT is built to do what the instruction asks instead.

The difference does not come from a bigger model. It comes from a different kind of training. OpenAI had people write example answers and judge the model's outputs, then used those judgments to adjust GPT-3. These models have been available in beta on the API for over a year; what changes now is that they become the default.

## Why GPT-3 struggles with instructions

GPT-3 learned by predicting the next word across a huge amount of internet text. That makes it good at continuing documents, but it was never trained to carry out a request safely or helpfully. In OpenAI's words, such models "aren't aligned with their users." Alignment here means that the model's behavior matches what the person using it actually wants.

This is why prompt writers have relied on careful framing and examples to get GPT-3 to behave. With InstructGPT, a plain instruction often works on its own.

OpenAI's post shows the contrast with a few prompts:

| Prompt | GPT-3 tends to... | InstructGPT tends to... |
|---|---|---|
| "Explain the moon landing to a 6 year old in a few sentences." | List more "explain X to a 6 year old" requests | Give a short, simple explanation |
| "Write a short poem about a wise frog." | Produce more writing prompts | Write the poem |

## What the results show

OpenAI reports several improvements. Labelers, the people paid to rate outputs, preferred answers from a 1.3-billion-parameter InstructGPT model over answers from the 175-billion-parameter GPT-3, even though the smaller model has more than 100 times fewer parameters. InstructGPT also invents facts less often and produces slightly less toxic text on standard tests.

The training is light compared with the original. It uses less than 2% of the computing power and data that went into pretraining GPT-3. OpenAI describes the effect as drawing out abilities GPT-3 already had but that were hard to reach through prompting.

There are limits. InstructGPT can still produce biased or toxic text and make things up. Because it follows instructions more readily, it can also follow harmful ones. And it mirrors the preferences of its labelers and of OpenAI's researchers; since it learned from English instructions, it leans toward the cultural values of English speakers rather than of everyone who might use it.

> **Try it:** Write the same request two ways: as a bare instruction ("Summarize this paragraph in one sentence for a ten-year-old") and as a GPT-3-style pattern with two worked examples. If you have API access, run both and note which gives the more useful answer, and whether the examples still help.

## Writing prompts for an instruction model

You can now lead with the instruction itself. Clear, specific requests work well:

```prompt
Rewrite the following note as a polite email to a landlord. Keep it under 80 words and end with a request for a reply by Friday.

Note: heater broken again, third time this month, need someone to come fix it.
```

Examples are still useful when you need an exact format, but they are no longer the only way to get the model on task.

## What does training with human feedback mean?

Training with human feedback means people, not just text from the internet, teach the model what a good answer looks like. OpenAI calls its method reinforcement learning from human feedback, or RLHF. Reinforcement learning is a way of training where a system tries actions and is nudged toward those that earn a higher reward. Here, the reward comes from human judgment.

OpenAI's post describes three steps:

1. **Demonstrations.** Labelers write ideal answers to real prompts sent to the API, and GPT-3 is fine-tuned on them.
2. **Rankings.** For many more prompts, the model produces several answers and labelers rank them from best to worst.
3. **A reward model.** A separate model learns to predict which answer people would prefer. GPT-3 is then trained to produce answers that score well with it, using an algorithm called PPO.

The idea is not new to OpenAI. In 2020 its researchers used the same approach to train summarization models: people compared pairs of summaries, a reward model learned their preferences, and the summarizer was tuned to please it. Those summaries were judged better than human-written reference summaries on Reddit posts and beat much larger models trained the usual way.

Human feedback is useful because "a good answer" is hard to define with a formula. Helpfulness, truthfulness and tone are judgments people make easily but software measures poorly. The open question is whose judgment counts; OpenAI says it is studying how labelers disagree and how to reflect the values of wider groups.

## Sources

- Aligning Language Models to Follow Instructions: https://openai.com/index/instruction-following/
- Learning to summarize from human feedback: https://arxiv.org/abs/2009.01325