# How does self-attention work, step by step?

Date: 2021-03-16 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/self-attention-work-step-step/

**Answer:** Self-attention lets each word in a sentence build its meaning from the other words around it. For every word, the model scores how well it matches each other word, turns those scores into weights that add up to 1, and mixes the other words' information according to those weights.

Self-attention is the calculation at the heart of the Transformer, the network design behind GPT-3 and BERT. Its job is easy to state. When the model reads a word, it should be able to look at every other word in the passage and decide which ones help explain it. In "The cat sat because it was tired," the word "it" means little on its own. Self-attention is how the model connects "it" to "cat."

The word "self" means the sentence is attending to itself: every word looks at the other words of the same passage, rather than at a separate text. The method was set out in the 2017 Google paper "Attention Is All You Need," which calls its version scaled dot-product attention. Here it is in six steps, with numbers small enough to check by hand.

## Step 1: Turn each word into three short lists of numbers

Before attention starts, each word has already become a vector: a list of numbers that stands for the word. (In the original Transformer, that list is 512 numbers long.) The model multiplies each word's vector by three grids of numbers it learned during training. The results are three new vectors for each word, called its **query**, its **key** and its **value**. We will look at what each one means at the end. For now, just know that every word has all three.

To keep the arithmetic friendly, our example uses vectors with only two numbers. Real models use far more, but the steps are identical.

## Steps 2 to 4: Score, scale and normalize

Say the model is working on "it," whose query is [1, 2].

**Step 2, score.** Compare the query of "it" with the key of every word, using a dot product: multiply the matching numbers and add them. For "cat," with key [2, 1.5], the score is 1×2 + 2×1.5 = 5. A higher score means a better match.

**Step 3, scale.** Divide every score by the square root of the vector length. Our vectors have length 2, so we divide by about 1.41. The paper explains why: with long vectors, dot products can grow very large, and that pushes the next step into a range where training becomes unstable.

**Step 4, normalize with softmax.** Softmax is a function that turns any list of numbers into positive weights that add up to 1, giving big numbers most of the share.

Here is the whole calculation for "it." The key vectors are made up for the illustration; a trained model learns its own.

| Word | Key | Score (query · key) | Scaled (÷ 1.41) | Weight after softmax |
|---|---|---|---|---|
| The | [0, 0] | 0 | 0.00 | 0.02 |
| cat | [2, 1.5] | 5 | 3.54 | 0.63 |
| sat | [0.5, 0] | 0.5 | 0.35 | 0.03 |
| because | [0, 0.5] | 1 | 0.71 | 0.04 |
| it | [1, 1] | 3 | 2.12 | 0.15 |
| was | [0.5, 0.5] | 1.5 | 1.06 | 0.05 |
| tired | [1, 0.5] | 2 | 1.41 | 0.08 |

The weights add up to 1.00. "Cat" gets 63% of the attention, "it" keeps 15% for itself, and the small words get almost nothing.

## Steps 5 and 6: Blend the values

**Step 5, weight.** Multiply each word's value vector by its weight. The value for "cat" is multiplied by 0.63; the value for "The" by 0.02.

**Step 6, add.** Sum all the weighted values. The result is the new vector for "it." It is mostly made of information from "cat," with a little of everything else. When the next layer of the network reads this vector, "it" now carries the meaning "the cat."

The model does this for every word at the same time, not one by one. In practice the queries, keys and values of all words are stacked into tables (matrices) and the six steps run as a few large multiplications, which graphics processors handle quickly. That is a big part of why Transformers train faster than older models that read one word at a time. In a model that writes text, like GPT-3, there is one extra rule: a word may only attend to the words before it, so the model cannot peek at what comes next.

> **Try it:** Change the key for "tired" in the table to [2, 2] and redo steps 2 to 4 with a calculator. (Softmax: raise e, about 2.718, to the power of each scaled score, then divide each result by their total.) Watch how much attention moves from "cat" to "tired."

## What are queries, keys and values?

The three names borrow the vocabulary of looking things up, the way a search engine or a database does. A useful picture is a library. The **query** is what you are looking for. Each book's **key** is the label on its spine that you compare against your query. The **value** is what is inside the book, the content you actually take away. In self-attention, a word's query asks "what kind of word would help me?", each key answers "this is what I offer," and the values are the information that gets blended in.

All three come from the same word vector, but through three different learned grids of numbers. That separation matters. A word can be looked up for one reason and contribute something else. In the original Transformer, the 512-number word vectors are reduced to queries, keys and values of 64 numbers each, and the model divides scores by 8, the square root of 64, which is the scaling step from our example.

The paper also runs several sets of queries, keys and values side by side, each with its own learned grids. This is called multi-head attention, and the original design uses eight heads. Each head can learn a different kind of link. In the paper's own examples, some heads seem to track sentence structure, and two appear to help work out what a pronoun such as "its" refers to. The model then joins the heads' outputs together.

Nobody writes these grids by hand. They start as random numbers and are adjusted during training, millions of times, whenever the model's guesses are wrong. Over time they settle into values that make useful words find each other, which is why a trained model can link "it" to "cat" without ever being told what a pronoun is.

## Sources

- Attention Is All You Need: https://arxiv.org/abs/1706.03762
- The Illustrated Transformer: https://jalammar.github.io/illustrated-transformer/