What is learning from human feedback (RLHF)?

A language model such as GPT-3 learns by predicting the next word across a vast amount of web text. That teaches it grammar, facts and styles, but it never teaches the model what a person sitting at the keyboard actually wants. Reinforcement learning from human feedback, usually shortened to RLHF, adds that missing ingredient. It turns human preferences into a training signal.

Last week OpenAI posted the full research paper behind its InstructGPT models, "Training language models to follow instructions with human feedback", by Long Ouyang and colleagues. It lays out how RLHF works on a general-purpose model, step by step and with the numbers.

Three words to know first

Fine-tuning means taking a model that is already trained and training it a little more on a smaller, focused set of examples.
Reinforcement learning is training by trial and reward: the system tries something, gets a score, and is nudged toward whatever earned higher scores.
A reward model is a separate neural network whose only job is to look at a prompt and an answer and output one number: how much a person would probably like that answer.

RLHF combines all three. Its key trick is that people cannot score millions of answers by hand, so they score a manageable sample and a reward model learns to imitate them.

The three steps in the InstructGPT paper

OpenAI hired a team of about 40 contractors, chosen with a screening test, to provide the human side. The prompts mostly came from requests that API customers had sent to earlier test models, with some written by the labelers themselves.

Step	What people do	What gets trained	Size of the data
1. Demonstrations	Write the answer they would want for a prompt	GPT-3 is fine-tuned on these answers (supervised fine-tuning, or SFT)	About 13,000 prompts
2. Rankings	Read several model answers to one prompt and order them best to worst	A reward model learns to predict the order	About 33,000 prompts
3. Reinforcement learning	Nothing new; the reward model stands in for them	The SFT model is tuned to earn high reward scores, using an algorithm called PPO	About 31,000 prompts

PPO, short for proximal policy optimization, is a standard reinforcement learning method that changes the model in small, careful steps. The paper adds a safeguard: a penalty whenever the tuned model drifts too far from the step-1 model. Without it, a model can learn to exploit quirks of the reward model. The authors also mix in some ordinary next-word training during step 3 so the model does not lose ground on standard benchmarks.

A worked ranking example

Step 2 is the heart of the method, and it is easy to try on paper. Labelers in the study saw between four and nine answers at once and put them in order. Here is a small version with four answers to one request.

Prompt

Explain what a credit score is to someone who just got their first job. Keep it to three sentences.

Answer	What it says, in brief	Rank
A	A clear three-sentence explanation with one tip about paying bills on time	1
B	A correct explanation, but seven sentences long	2
C	A list of other questions about money, with no answer	4
D	Three sentences, friendly, but it states a made-up rule about score ranges	3

One ranking of four answers contains six pairs: A beats B, A beats D, A beats C, B beats D, B beats C, D beats C. The reward model is trained on pairs like these. For each pair, it learns to give the preferred answer a higher number than the other one. After tens of thousands of prompts, it can score an answer it has never seen.

Notice what the labeler had to weigh. B is correct but ignores the length limit. D follows the format but is wrong. Ranking forces a choice between those failures.

Try it: Ask a question you know well and write down four different answers to it: one excellent, one too long, one off-topic, one fluent but with a factual slip. Rank them, list every pair your ranking implies, and note which pairs were hard to decide. Those hard pairs are exactly where human labelers disagree most.

Where the idea comes from

RLHF did not start with InstructGPT. The paper credits earlier work, including a 2020 OpenAI study on summaries. In that project, people compared pairs of summaries of Reddit posts, a reward model learned their choices, and a summarizer was tuned against it. The resulting summaries were preferred to the human-written reference summaries in the dataset. InstructGPT applies the same recipe to a broad range of instructions.

Why do human ratings make a model more helpful?

Human ratings help because they measure the thing we actually care about. Next-word prediction rewards text that looks like the internet. A reward model trained on rankings rewards text that people judged useful, truthful and appropriate for the request. Optimizing for the second target pulls the model toward answering the question rather than continuing a page.

The paper's results show how large the effect is. Labelers preferred outputs from a 1.3-billion-parameter InstructGPT model to those from the 175-billion-parameter GPT-3. At full size, InstructGPT's answers were preferred to GPT-3's 85% of the time, give or take 3 points, and to GPT-3 with a carefully written few-shot prompt 71% of the time. On tasks where the answer should rely only on the text provided, such as summarizing, InstructGPT made up information about half as often.

Rankings also scale better than demonstrations. Saying "this one is better" takes less effort than writing the ideal reply from scratch, and showing labelers several answers at once, as the paper does, turns one judgment session into many training pairs.

The limits come from the same source. A model tuned this way reflects the people who ranked its answers, mostly the labelers and the researchers who wrote their instructions, not any universal standard. The authors state plainly that InstructGPT is not less biased than GPT-3, still makes mistakes, and can follow harmful instructions. Human feedback makes a model more helpful to the people whose judgment it learned, which is why who those people are matters so much.