What are tokens, and why do language models read text in pieces?

When you type into GPT-3 through OpenAI's new API, the model never sees letters or words the way you do. Before anything else, your text is cut into pieces called tokens, and each token is replaced by a number from a fixed list. The model reads that list of numbers, predicts the next number, and the result is turned back into text for you.

The cutting is not random. Common words like "the" or "morning" stay whole. Rarer words are split into familiar chunks, so "tokenization" becomes "token" plus "ization". This matters for anyone using GPT-3, because token counts set the limits on how long your prompt can be, and the way text is split explains some of the model's odd strengths and weaknesses.

Why not just use words, or letters?

There are two obvious ways to feed text to a model, and both have problems.

• Whole words. A list of every word would be huge, and it would still miss names, typos, new slang and words from other languages. Anything not on the list becomes an "unknown" placeholder, and the meaning is lost.
• Single letters. Every text can be spelled with letters, so nothing is unknown. But sentences become very long sequences, and the model has to work much harder to see that "c", "a", "t" make a cat.

Subword tokens sit in the middle. In 2015, Rico Sennrich and colleagues at the University of Edinburgh showed that translation systems handle rare words better when they are broken into smaller units. One method they used is byte pair encoding, or BPE, borrowed from data compression. It starts with single characters and repeatedly merges the pair of symbols that appears most often in a large body of text, adding each merged pair to the list. After tens of thousands of merges, frequent words have become single tokens, while rare words remain as a few common pieces.

GPT-2 adapted this idea to work on raw bytes, the basic units computers use to store text. According to the GPT-2 paper, that gives a starting set of only 256 symbols, so any string in any language can be encoded with no unknown words. The final GPT-2 vocabulary has 50,257 tokens. The GPT-3 paper says GPT-3 reuses this same tokenizer.

A worked example: splitting real sentences

Here is how GPT-2's tokenizer, which GPT-3 shares, divides a few short texts. Each vertical bar marks a boundary between tokens. A space usually belongs to the start of the following token, so " cat" with its space is one token.

The pattern is clear. Ordinary English words are mostly one token each, while "croissant" needs three. The French sentence, roughly as long as the English one, needs 14 tokens instead of 10, and the single French word "épanouissement" (blossoming, or personal fulfillment) breaks into five pieces.

The reason is the training data. The merges were learned from mostly English web text, so English words got their own tokens and other languages were left in fragments. The GPT-3 paper itself reports that the model translates into English much better than out of it, and suggests that its weakest translation results may come partly from reusing a tokenizer designed for almost purely English data.

What tokens mean for you

Three practical consequences follow.

1. Length is counted in tokens. Limits on prompts and outputs are set in tokens, not words or characters. Text in French, Spanish or German uses more of them for the same meaning.
2. Spelling tasks are hard. The model sees "croissant" as three chunks, not nine letters. The GPT-3 paper notes that its word-unscrambling tests require character-level work, which its BPE tokens make difficult.
3. Numbers are split oddly. "2,048" becomes "2", ",", "048". That is one reason arithmetic on long numbers trips the model up.

How much text fits in GPT-3's 2,048-token window?

GPT-3, like GPT-2 before it, reads its input through a window of fixed size called the context window. For GPT-3 that window is 2,048 tokens, double GPT-2's 1,024. Everything the model can take into account at once must fit inside it: your instructions, any examples, and the text the model is producing.

A sample paragraph gives a sense of scale. A 77-word English paragraph about a neighborhood bakery splits into 92 tokens, about 1.2 tokens per word. At that rate, 2,048 tokens hold roughly 1,700 words of everyday English. If you count a single-spaced page as about 500 words, that is three to three and a half pages. In French, the same paragraph's opening sentences came out at about 1.8 tokens per word, so the window holds noticeably less text, closer to 1,100 words.

That budget shapes how people use GPT-3. The paper explains that few-shot prompts usually hold between 10 and 100 examples simply because that is what fits in 2,048 tokens. A long document cannot be pasted in whole; it has to be summarized in sections or trimmed to the part that matters. And if your prompt takes up most of the window, there is little room left for the answer. A good habit is to keep instructions and examples short, and to leave a few hundred tokens free for the reply.