What is AI alignment, in plain words?

Imagine hiring a very fast, very literal assistant and telling them to "clear my inbox". You come back to find every email deleted. They did exactly what you said, and not at all what you meant. AI alignment is the effort to close that gap: to build systems whose behavior matches the intentions of the people using them, including the parts those people never thought to spell out.

The word sounds abstract, but the problem shows up in ordinary use of today's language models. It also sits at the center of some of the most practical research of the past year.

The gap between the goal and the training

Every machine learning system is trained to push one number up or down. GPT-3's number measures how well it predicts the next word of internet text. That is a useful skill, but it is not the same as "help the person in front of you". OpenAI's InstructGPT paper describes this mismatch directly: predicting the next token on a web page is a different objective from following a user's instructions helpfully and safely, so in the researchers' terms the training goal is misaligned.

Two everyday analogies help:

• The teacher who teaches to the test. If a school is judged only by test scores, it may drill test questions instead of teaching the subject. The measure goes up while the real goal suffers.
• The GPS with the wrong priority. A navigation app told only to minimize minutes may send you down a dirt road at night. Fast, yes; what you wanted, no.

OpenAI ran into the first pattern with summaries in 2020. Researchers found that training a model to score well on ROUGE, an automatic metric that counts overlapping words with a reference summary, produced worse summaries in human eyes than training it to match what people actually preferred.

A worked example of a misaligned answer

OpenAI's InstructGPT announcement in January shows a clean case. Given this prompt:

Explain the moon landing to a 6 year old in a few sentences.

the original GPT-3 did not explain anything. It produced more lines in the same style, such as a request to explain gravity to a six-year-old, then relativity. From GPT-3's point of view that is a reasonable continuation: a line like this could easily sit in a list of similar writing prompts. From the user's point of view it is useless. Nothing is broken; the model is doing the job it was trained for, which is not the job the user had in mind.

Alignment also covers quieter failures. A model can answer confidently with invented facts, repeat stereotypes from its training data, or go along with a harmful request. Each is a case where likely text and good behavior come apart.

What the InstructGPT results show

To narrow the gap, OpenAI fine-tuned GPT-3 using judgments from about 40 hired labelers, who wrote example answers and ranked the model's attempts. The paper defines its target with three words borrowed from earlier work: models should be helpful (solve the user's task), honest (not make things up or mislead) and harmless (not cause physical, psychological or social harm). The results, reported in March:

The authors are careful about the limits. The models still make simple mistakes, and the paper says they are aligned to the preferences of a specific group, mostly the labelers and researchers, rather than to any broad idea of human values. Whose intentions count is part of the alignment question, not a detail.

Why do labs care whether a model follows instructions?

The first reason is usefulness. A model that answers the question it was asked is worth far more than one that needs elaborate prompt tricks to behave. OpenAI made the InstructGPT models the default in its API in January, after more than a year in beta, so following instructions is now a product feature, not just a research goal.

The second reason is safety. Instruction-following is a small, testable version of a much larger problem: making sure that increasingly capable systems pursue the goals people give them and avoid the harms people care about. The InstructGPT authors describe their approach as improving the alignment of the systems that exist now, rather than reasoning in the abstract about future ones, because working on real models shows quickly which methods help and which fail.

The third reason is cost. OpenAI reports that the fine-tuning used less than 2% of the computing power of GPT-3's original training. When a modest investment makes a model clearly more helpful and somewhat more truthful, it is an easy case for labs to make. The open questions are the harder ones: how to align models on tasks people cannot easily judge, and how to decide whose preferences a model should follow.