# How is a language model trained, from raw text to answers?

Date: 2022-04-05 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/language-model-trained-raw-text-answers/

**Answer:** A language model is trained in two broad stages. First it reads a huge collection of text, cut into small pieces called tokens, and learns to guess the next token over and over; then it is often fine-tuned on a much smaller set of examples so that its guesses turn into useful answers.

Every large language model, from OpenAI's GPT-3 to the PaLM model Google described this week, starts as a network of numbers set more or less at random. It knows nothing. Training is the slow process of nudging those numbers, billions of them, until the network becomes very good at one simple game: given some text, guess what comes next. Answers, summaries and translations all grow out of that game.

The path from raw text to a helpful reply has a handful of clear steps. None of them is magic, and seeing them one at a time makes it much easier to understand both what these models can do and why they still get things wrong.

## Step 1: Gather a mountain of text

The first ingredient is text, and a lot of it. The GPT-3 paper describes a training mix built mostly from a filtered copy of Common Crawl, a public archive of web pages, plus collections of books, English Wikipedia and other web text. The model saw about 300 billion tokens during training. Google's PaLM announcement describes a similar recipe with a wider spread: web documents in English and other languages, books, Wikipedia, conversations and code from GitHub, with roughly a fifth of the data in languages other than English and about 5% code.

Filtering matters. Raw web pages are full of spam, menus and repeated boilerplate, so teams clean and weight the sources, often showing higher-quality collections to the model more often.

## Step 2: Cut the text into tokens

A model does not read letters or whole words. It reads **tokens**, which are common words or pieces of words. A frequent word such as "the" is usually one token, while a rare word may be split into two or three pieces. Each token gets an ID number, so a sentence becomes a list of numbers the network can work with.

## Step 3: Play the next-token game, billions of times

Now the real training begins. The model is shown a stretch of text and asked to predict the next token. Its guess is compared with the token that actually came next, and a procedure called gradient descent adjusts the internal numbers, called **parameters**, so that the right token becomes a little more likely next time. GPT-3 has 175 billion parameters; PaLM has 540 billion and was trained across 6,144 of Google's TPU chips.

Here is the game on one toy sentence. Each row is one training example, and the model only ever sees the text to the left:

| What the model sees | What it should predict |
|---|---|
| The | kettle |
| The kettle | is |
| The kettle is | boiling |
| The kettle is boiling | , |
| The kettle is boiling, | so |

One sentence gives several lessons. Repeat that across hundreds of billions of tokens and the model picks up spelling, grammar, facts that appear often, the shape of a recipe or a news story, and the habit of following patterns. Nobody labels any of this by hand; the text itself provides the right answers, which is why this stage is called **self-supervised**.

## Step 4: Turn a text predictor into an assistant

A model trained only on the next-token game is a powerful autocomplete, not an assistant. OpenAI puts the problem plainly in its InstructGPT announcement: GPT-3 was trained to predict internet text, not to do what the user asks, so it can ramble, invent facts or produce toxic text. Ask it a question and it may answer, or it may continue with a list of more questions, because both are plausible next text.

The fix is a second, much smaller round of training. People write good example answers and rank the model's attempts, and the model is adjusted toward the replies people prefer. OpenAI reports that this extra step used less than 2% of the computing power and data of the original training, yet labelers preferred the tuned models' answers even when the tuned model was far smaller than GPT-3.

> **Try it:** In the OpenAI Playground, give the same instruction, such as "Explain why the sky is blue to a ten-year-old", first to the original `davinci` model and then to `text-davinci-002`. Compare whether each one answers the request or simply continues the text.

## What is the difference between pre-training and fine-tuning?

**Pre-training** is the big first stage: the next-token game played on a huge, mostly unlabeled collection of text. It is expensive, needs thousands of specialized chips, and produces a general model that has absorbed a broad picture of language and the world as described in its data. Only a few organizations can afford to do it, which is why GPT-3 and PaLM come from OpenAI and Google.

**Fine-tuning** starts from that pre-trained model and continues training on a much smaller, carefully chosen set of examples aimed at one goal: following instructions, answering customer questions, writing in a house style or sorting support tickets. Because the model already knows language, a comparatively small set of good examples can change its behavior noticeably, at a tiny fraction of the original cost.

A simple way to picture it: pre-training is like years of wide reading, and fine-tuning is the short apprenticeship that teaches what a particular job expects. The InstructGPT work adds a twist, using people's rankings rather than fixed examples to guide the fine-tuning, but the principle is the same. The broad knowledge comes from pre-training; the manners, focus and format come from fine-tuning.

## Sources

- Language Models are Few-Shot Learners: https://arxiv.org/abs/2005.14165
- Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance: https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html
- Aligning Language Models to Follow Instructions: https://openai.com/index/instruction-following/