# How do you train a tiny language model from scratch to understand how it works?

Date: 2026-10-28 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/train-tiny-language-model-from-scratch/

**Answer:** Start with Andrej Karpathy's nanoGPT example, which trains a character-level model on Shakespeare in minutes on an ordinary laptop, then move to his nanochat project, which runs every stage of a small chatbot and can reach GPT-2-level ability for about $48 of rented GPU time. Watching the loss fall and the text improve teaches you more about language models than any explanation.

Using ChatGPT is like driving a car; training a tiny model yourself is like opening the hood. You will not build anything that competes with a commercial assistant, and that is not the point. In an afternoon you can watch a program that starts by producing random characters slowly learn spelling, then words, then something that looks like Shakespeare. Every abstract idea about language models becomes something you have seen happen on your own screen.

As of October 2026, a popular learning path comes from the AI researcher Andrej Karpathy, who publishes small, readable training code. His older project, **nanoGPT**, is now marked as deprecated in favor of **nanochat**, but its Shakespeare example remains the gentlest first step because it runs without a graphics card.

## Step 1: a character-level model on your laptop

nanoGPT's beginner exercise trains a **character-level** model: instead of words or tokens, it predicts one letter at a time. The training data is a single 1 MB file of Shakespeare's plays. You need Python and PyTorch installed, then three commands from the project folder:

```bash
python data/shakespeare_char/prepare.py
python train.py config/train_shakespeare_char.py --device=cpu --compile=False --eval_iters=20 --log_interval=1 --block_size=64 --batch_size=12 --n_layer=4 --n_head=4 --n_embd=128 --max_iters=2000 --lr_decay_iters=2000 --dropout=0.0
python sample.py --out_dir=out-shakespeare-char --device=cpu
```

The first command turns the text into numbers. The second trains a small transformer with 4 layers that looks back 64 characters at a time; the README reports about three minutes on a laptop processor. The third asks the model to write. On an Apple Silicon Mac, replacing `cpu` with `mps` uses the built-in graphics and can be two to three times faster. With a proper GPU, the default settings train a larger 6-layer model in about three minutes and produce noticeably better pastiche.

> **Try it:** Run the three commands, save a sample of the output, then train again with `--max_iters=200` instead of 2000 and compare the two samples side by side. You are looking at the difference that more training makes.

## Step 2: simple data, surprisingly fluent stories

The Shakespeare model produces the look of English rather than sense. A 2023 paper by Ronen Eldan and Yuanzhi Li, *TinyStories*, showed how far small models can go when the data is simple. The authors generated short stories with GPT-3.5 and GPT-4 using only words a typical 3- or 4-year-old understands. Models with fewer than 10 million parameters, some with a single transformer block, trained on those stories wrote fluent, consistent stories several paragraphs long with nearly perfect grammar. The lesson is that what a model can do depends as much on its data as on its size. Training a very small model on simple stories like these makes a good second project.

## Step 3: a whole chatbot pipeline with nanochat

**nanochat** covers every stage that turns text into a chat assistant: training a tokenizer, pretraining on a large text collection, fine-tuning for conversation, evaluation and a simple chat interface. One script, `runs/speedrun.sh`, runs all of it. On a rented server with eight Nvidia H100 GPUs, the README says it reaches GPT-2-level ability in about two hours for roughly $48, or around $15 on cheaper spot instances. Training GPT-2 itself cost about $43,000 in 2019. The code also runs on one GPU, about eight times more slowly, and a separate `runs/runcpu.sh` script trains a much smaller version on a Mac or a regular processor in a few tens of minutes, with weak but instructive results.

A single setting, `--depth`, the number of layers, controls the size of the model and sets the other choices automatically, which makes it easy to see how size changes quality. Karpathy compares talking to the finished speedrun model with talking to a kindergartner, and suggests asking it who you are to watch it hallucinate.

## What does it teach you that using ChatGPT never will?

**That a model is a next-token predictor and nothing more mysterious.** You see the training loss, a single number measuring how badly the model predicts the next character, fall step by step. When it plateaus, quality stops improving. Benchmarks and "intelligence" become the visible result of that one number going down.

**That data shapes everything.** Train on Shakespeare and you get Shakespeare-like text; train on children's stories and you get children's stories. A commercial model's knowledge, tone and blind spots come from the same mechanism at a far larger scale.

**Why hallucinations happen.** Your tiny model will invent words and confident nonsense, because producing plausible next characters is all it was trained to do. Seeing that happen at small scale makes it much easier to understand why large models still make things up.

**Where the cost comes from.** Changing the number of layers, the context length or the training steps shows directly how memory, time and money grow. After that, headlines about data centres and GPU budgets read differently: you have paid for a small piece of that bill yourself.

## Sources

- nanoGPT (README): https://github.com/karpathy/nanoGPT
- nanochat (README): https://github.com/karpathy/nanochat
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?: https://arxiv.org/abs/2305.07759