Short answer
Your prompt is cut into tokens, each token becomes a list of numbers called an embedding, and a stack of Transformer layers uses attention to work out how every token relates to the others. The model then produces a probability for every possible next token, picks one, adds it to the text and repeats, one token at a time, until the answer is complete.
Follow-up question Where do tokens, embeddings and attention fit in?
When you press Enter in ChatGPT, the reply seems to arrive as a single act of understanding. Underneath, it is a loop that runs many times per second: read everything so far, predict the next small piece of text, add it, and go again. The words appearing one after another on your screen are a fairly honest view of that loop.
The steps below describe the general recipe that GPT-style models follow. The exact sizes and settings of the models behind ChatGPT differ from the published research versions, but the stages are the same.
Step 1: The prompt becomes tokens
A language model doesn't read letters or whole words. It reads tokens: chunks of text that may be a whole short word, part of a longer word, a space plus a word, or a punctuation mark. OpenAI's rule of thumb for English is that one token is about four characters, or roughly three quarters of a word. Other languages often need more tokens for the same amount of text.
Each token has an ID number in the model's vocabulary, so after this step your sentence is a list of integers. ChatGPT also includes the earlier turns of the conversation and any hidden instructions, which is why long chats eventually run out of room: everything must fit in the model's context window, measured in tokens.
Step 2: Tokens become embeddings
A token ID on its own is just a label. The model looks each ID up in a large table and swaps it for an embedding, a long list of numbers learned during training. Tokens used in similar ways end up with similar lists, so "cat" and "kitten" sit closer together than "cat" and "invoice."
The model also needs to know word order. The original Transformer design adds positional information to each embedding, so "dog bites man" and "man bites dog" look different to the layers that follow.
Step 3: Attention layers build context
Next, the embeddings pass through a stack of identical layers. In each one, attention lets every token gather information from the tokens before it. Technically, each token produces a query (what it is looking for), and every token offers a key (what it contains) and a value (the information it passes on). How well a query matches each key decides how much of each value flows in. Several of these attention "heads" run side by side, each free to track a different kind of relationship.
After attention, a small feedforward network processes each token's numbers on its own. Layer after layer, a token's numbers shift from describing the token itself to describing its meaning in this particular sentence.
Step 4: Predict, pick, repeat
At the top of the stack, the model turns the last token's numbers into a score for every token in its vocabulary, then converts the scores into probabilities. One token is chosen, added to the text, and the whole process runs again with the slightly longer sequence.
How the choice is made matters. A setting called temperature controls how often the model picks less likely tokens. At 0 it nearly always takes the top choice, which suits factual tasks; higher values give more varied, creative wording. OpenAI's guidance adds a useful warning: randomness and truthfulness are different things.
Finally, the chosen token IDs are turned back into text, and you see the reply.
A worked example
Take the short prompt Write a haiku about rain. The token split and probabilities below are illustrative, simplified to show the idea; a real tokenizer and model will give different values.
| Stage | What happens (illustrative) |
|---|---|
| Tokens | Write · a · ha · iku · about · rain · . (7 tokens) |
| IDs | Each token becomes a number, such as 8144, 264, 6520… |
| Embeddings | Each ID becomes a list of thousands of numbers, plus position |
| Attention | rain attends strongly to ha and iku, so the model "knows" a poem is wanted |
| Next-token scores | Soft 18%, Gray 12%, Drops 9%, Silver 6%… |
| Choice | With some randomness, Soft is picked and appended |
| Repeat | The loop runs again on 8 tokens, then 9, until the haiku ends |
Try it: Ask ChatGPT the same creative prompt three times in new chats and compare the first words of each reply. Then ask a factual question three times. The creative answers usually vary more, which is the sampling step at work.
Where do tokens, embeddings and attention fit in?
Tokens are the entry and exit points. They decide how your text is cut up on the way in, how the answer is assembled on the way out, and how much fits in a conversation. They also explain some odd behavior: a model may stumble when counting letters in a word because it never sees individual letters, only chunks.
Embeddings are the model's working language. Every token is represented as numbers, and all the "thinking" in the layers is arithmetic on those numbers. When people say a model has learned meaning, they are describing patterns in these learned lists.
Attention is the step that brings context into each token. It is what lets the model connect a pronoun to the right noun many sentences back, or keep a requested format in view while it writes. The 2017 paper that introduced the Transformer showed that attention alone, without older step-by-step designs, could handle language well and train efficiently in parallel. Every reply you get from ChatGPT runs through that same idea, once for every token it writes.
Sources
- What are tokens and how to count them? — OpenAI Help Center, March 21, 2022
- Attention Is All You Need — Vaswani et al., Google Brain / Google Research (arXiv), June 12, 2017
- Best practices for prompt engineering with OpenAI API — OpenAI Help Center