What does GPT stand for, and what do generative pre-trained transformers do?

The three letters appear everywhere this spring: GPT-4, ChatGPT, "GPT-powered" apps. They are not a brand slogan. Each one names a real idea about how the model was built, and knowing them explains a lot about what ChatGPT does well, where it slips, and why your prompt matters so much.

The name goes back to a 2018 OpenAI paper titled "Improving Language Understanding by Generative Pre-Training." Every model since, up to GPT-4 released last month, follows the same basic recipe. Let us take the three words one at a time.

G is for generative: it writes, it does not look up

A generative model produces new output rather than retrieving a stored answer. When you ask ChatGPT a question, it does not search a table of ready-made replies. It writes the answer one small piece of text, called a token, at a time, each time choosing a likely next piece given everything before it.

An everyday analogy: a librarian fetches the book that already contains your answer; a writer composes a fresh paragraph from what they have learned. A GPT model is the writer. That is why two people asking the same question can get differently worded replies, and why the model can sound confident about something that is simply wrong. It is producing plausible text, not quoting a checked record. OpenAI's own page on GPT-4 says the model can still make up facts and make reasoning errors.

P is for pre-trained: the reading happened before you arrived

Pre-training is the long first stage of learning, done once, before the model is offered to anyone. The model is shown a huge amount of text and practices one game over and over: guess the next word, check, adjust. OpenAI says the GPT-4 base model was trained this way on publicly available data, such as internet text, plus data it licensed.

Think of a new employee who, before their first day, has read an enormous library: manuals, novels, recipes, forum arguments, code. Nobody taught them your company's job yet, but they arrive with grammar, general knowledge and a feel for many kinds of writing. A short second stage, called fine-tuning, then shapes that general reader for a particular role. For GPT-4, OpenAI uses a method called reinforcement learning from human feedback (RLHF), where people rate answers and the model is nudged toward the preferred ones.

OpenAI makes a telling point about the split: the model's abilities seem to come mainly from pre-training, while the later stage mostly steers how it behaves. Without that steering, the base model needs careful prompting even to realize it should answer your question.

T is for Transformer: reading the whole sentence at once

The Transformer is a network design introduced by Google researchers in a 2017 paper. Earlier language models read text roughly one word after another, like following a line with your finger. The Transformer relates all the words in a passage to each other in parallel, using a mechanism called attention: for each word, it works out which other words matter most to its meaning, and by how much.

Consider "The trophy didn't fit in the suitcase because it was too big." To guess what "it" refers to, a reader has to connect "it" with "trophy," "suitcase" and "big." Attention is the machinery that lets the model make those links across the whole sentence. Being able to process text in parallel also made it practical to train on far more data, which is what made large-scale pre-training possible.

A worked example: what pre-training gives you for free

You never trained ChatGPT on your tasks, yet it can attempt many of them on the first try. Try these three requests in one chat:

1. Fix the grammar: "Me and her has went to the market yesterday."
2. Is this review positive or negative? "The soup was cold and the waiter ignored us."
3. Explain in two sentences, for a 10-year-old, why the sky is blue.

None of these is a skill someone added specially for you. Each comes from patterns the model absorbed while reading:

The original 2018 paper noticed this effect early. Even before any task-specific training, the pre-trained model could already attempt some tasks it was never set up for. That is the heart of the "P": general skill, ready before your first prompt.

Why does 'pre-trained' matter for everyday users?

First, it explains what the model knows and when it stopped learning. Pre-training happens once, on text gathered up to a certain point. OpenAI says GPT-4 generally lacks knowledge of events after September 2021. If you ask about last week's news, a model without a browsing tool is guessing, not reporting. For anything recent, give it the facts in your prompt or check another source.

Second, it explains why context in your prompt is so powerful. The model already holds broad knowledge; your job is to point it at the right part. A prompt that names the audience, the purpose and the format ("a 150-word email to a landlord, polite but firm") draws on what it learned far better than "write an email."

Third, it sets fair expectations. Because the model learned from a vast mix of text, including incorrect and contradictory material, it has absorbed mistakes and biases along with knowledge. Treat its answers the way you would treat a well-read colleague's first draft: often very useful, sometimes wrong, and always worth a quick check before it goes out under your name.