# What is Constitutional AI, and how can an AI learn from a list of principles?

Date: 2022-12-16 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-constitutional-ai/

**Answer:** Constitutional AI is a training method that Anthropic described in a paper on 15 December 2022, in which an AI assistant learns to avoid harmful answers by critiquing and rewriting its own responses against a short written list of principles, called a constitution. Instead of people labeling thousands of harmful outputs, an AI applies the principles, which Anthropic calls reinforcement learning from AI feedback (RLAIF).

How do you teach an AI assistant to refuse dangerous requests without making it refuse everything? Yesterday the AI company Anthropic published a paper with a new answer. Its method, called **Constitutional AI**, gives the model a short written list of principles and has the AI itself use them to judge and improve its answers.

The paper is titled "Constitutional AI: Harmlessness from AI Feedback." Its approach aims for an assistant that is harmless but not evasive: one that explains why it will not help, rather than shutting the conversation down.

## What Anthropic published

The paper went up on arXiv, the open repository for research papers, on 15 December 2022. It has 51 authors, all at Anthropic, with Yuntao Bai listed first and Jared Kaplan last; Dario Amodei, Amanda Askell, Chris Olah and Sam McCandlish are among the others. Alongside it, Anthropic shared its prompts and principles on GitHub so others can study them.

To understand what is new, it helps to know the standard method. **Reinforcement learning from human feedback (RLHF)** trains assistants using many human judgments: people compare pairs of answers and pick the better one, and the model learns to produce answers people prefer. It works, but it needs large numbers of human labels, including people reading harmful content. And Anthropic found a side effect: because crowdworkers rewarded refusals, earlier models learned to dodge difficult topics, sometimes for the rest of a conversation.

## How Constitutional AI works

The **constitution** is simply a list of rules written in plain language. Anthropic explains the name two ways: a short list of principles turns out to be enough to train a less harmful system, and every general AI system ends up governed by some set of principles anyway, even when nobody writes them down. For its experiments the team wrote 16 principles, such as asking which of two answers "a wise, ethical, polite and friendly person would more likely say." Humans still write the rules and a few examples; what disappears is the need for human labels identifying harmful outputs.

Training happens in two stages.

**Stage 1: critique, revise, learn.** The researchers start with an assistant trained only to be helpful, and feed it prompts designed to provoke bad behavior, collected earlier by people trying to trip models up (a practice called **red teaming**). The paper shows how one round goes:

| Step | What the model sees or writes |
|---|---|
| Prompt | A person asks for help getting into a neighbor's wifi. |
| First answer | The helpful-only model suggests a (made-up) hacking app. |
| Critique | Asked to find what is harmful in its answer, it notes that breaking into someone's wifi invades their privacy and may be illegal. |
| Revision | Asked to rewrite without harmful content, it advises against the idea and mentions the legal risk. |

Each round draws a principle at random, and the loop can repeat several times. A new model is then fine-tuned on the final, improved answers.

**Stage 2: AI feedback.** The stage-1 model writes two answers to each harmful prompt. Another model reads both, along with a principle from the constitution, and picks the better one. Those AI choices train a **preference model**, a scorer that rates answers, and the assistant is then improved with reinforcement learning against that scorer. This is the **RLAIF** in the paper's subtitle: reinforcement learning from AI feedback. Human preference labels are still used for helpfulness; only the harmlessness judgments come from the AI.

The judging model can also use **chain-of-thought reasoning**, writing out its thinking step by step before choosing. Anthropic reports that this improves its judgments and makes them easier for people to read and check.

> **Try it:** Ask any chatbot you use a gray-area question, such as how to get into a locked account you have forgotten the password for. Then, in the same chat, ask it to critique its own answer against one principle you write, for example "Choose the response that is most helpful while avoiding anything that could enable harm to others," and to rewrite its reply. Compare the two versions.

## What the results show

The team compared its assistant with earlier models trained with human feedback. Crowdworkers judged the Constitutional AI model to be less harmful at the same level of helpfulness, and less evasive. When it declines a harmful request, it tends to engage and explain its objection instead of giving a flat refusal.

The researchers also found that as language models grow more capable, they get better at spotting harm, and that chain-of-thought helps further. That suggests a trend worth watching: the stronger the models become, the more useful they may be as supervisors of other models.

## Why it matters

Constitutional AI tackles three practical problems at once. It reduces the amount of human labor needed to make assistants safer, including the unpleasant work of reading harmful content. It makes the rules more transparent, because the principles are written down and can be read, debated and changed. And it tries to break the trade-off where safer assistants become less helpful.

The authors are careful about limits. They describe their principles as chosen in a fairly informal, trial-and-error way for research, and argue that in the future such principles should be developed by a wider group of people and adapted to how a model will be used.

## What does Constitutional AI mean for the chatbots people use?

For now, it is a research method, not a product: the paper describes experiments, not a service you can sign up for. But the ideas are likely to matter well beyond one lab, because every company building assistants faces the same questions about what they should refuse and how.

For everyday users, the most useful takeaway is that an assistant's behavior comes from choices someone made, and those choices can be written in plain words. If a chatbot refuses a request or answers carefully, there are rules behind it. Constitutional AI makes such rules explicit, which invites a better public conversation about who should write them.

For builders, the paper offers a technique to borrow at small scale: ask a model to check its draft against a stated principle, then revise. That self-critique loop, run in a prompt, can catch problems before anyone sees the first draft.

## Sources

- Constitutional AI: Harmlessness from AI Feedback: https://www.anthropic.com/index/constitutional-ai-harmlessness-from-ai-feedback
- Constitutional AI: Harmlessness from AI Feedback: https://arxiv.org/abs/2212.08073