How do you tell whether a language model you trained is any good?

When training finishes, you have a file of numbers and a falling line on a chart. Neither tells you whether the model is any good. Testing a home-trained model calls for more than the checks used on a commercial assistant, because you also need to know whether training itself worked.

Each of the four levels below answers a different question, and they catch different failures. Run them in order, because the first ones are cheap.

Level 1: did it learn to predict text it never saw?

Loss measures how surprised the model is by the next token. Training loss is measured on text it practiced on, and validation loss on the slice you set aside. A falling training loss with a flat or rising validation loss means overfitting: the model is memorizing rather than learning. That comparison is the first thing to look at.

Loss alone is hard to compare, as it depends on the tokenizer. Two conversions help:

• Perplexity is the exponential of the loss: roughly how many tokens the model is torn between at each step.
• Bits per byte (bpb) divides by the number of bytes per token, so that models with different tokenizers can be compared. Lower is better. nanochat reports this as its main number.

import math

loss_nats = 2.05        # validation loss per token, from your training log
bytes_per_token = 3.1   # average, measured on your validation text
print("perplexity", round(math.exp(loss_nats), 1))
print("bits per byte", round(loss_nats / math.log(2) / bytes_per_token, 3))

Those sample inputs print a perplexity of 7.8 and 0.954 bits per byte. For scale, the leading GPT-2-grade runs on nanochat's leaderboard sit around 0.72. The loss is in "nats", which is what PyTorch's standard cross-entropy function gives, and dividing by the natural log of 2 converts to bits. Compare only runs scored on the same validation text.

Level 2: what does that look like?

Keep a fixed sheet of about 20 prompts and run it at every checkpoint: a story opening, a fact, a sum, a question with a false premise, one in French if that matters to you. Use the same settings each time, and read the results side by side. You will notice repetition, nonsense and invented facts that no single number shows. Remember that a pretrained model without chat training only continues text, so give it the start of a sentence rather than a question.

Level 3: how does it compare with known models?

A benchmark is a standard exam with known scores from other models. Three things make it fair: the same questions, the same format and the same scoring. Two tools handle this.

nanochat's built-in tests are the simplest if you trained with it. Its pretraining evaluation includes CORE, an average over 22 benchmarks from the DCLM research project, rescaled so that random guessing scores near zero and a perfect score is 1. GPT-2's CORE is 0.2565, and nanochat's recent leaderboard runs score from 0.2571 to 0.2690. Its chat tests cover tasks such as multiple-choice science and school math:

python -m scripts.base_eval
python -m scripts.chat_eval -i sft

On several GPUs, launch them with torchrun as the project's speedrun script does.

EleutherAI's lm_eval is the general tool, working with any model in Hugging Face's format, whether on the Hub or in a local folder. A nanochat checkpoint uses its own format and would need converting first.

pip install "lm_eval[hf]"
lm_eval --model hf \
    --model_args pretrained=./my-model \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

Use lm-eval ls tasks to see the available names. For a reference point, the DCLM authors report 64% on MMLU, a broad multiple-choice exam, for a 7-billion-parameter model on 5 worked examples per question. Note how the question was asked: results with different numbers of examples are not comparable.

Benchmarks have limits. Questions leak into training data, so a high score can reflect memorization. A public benchmark also tells you little about a narrow task, and an English one tells you nothing about French.

Level 4: does it do the job you built it for?

Only your own tests can answer this. Our guide to building an eval set from your own work covers the method, and the LoRA guide shows a before-and-after script. Three additions apply to a model you trained yourself:

How big must a difference be before you believe it?

Larger than most people expect. A benchmark score from a sample of questions has a margin of error, which you can estimate quickly:

import math

def margin(p, n):
    return 1.96 * math.sqrt(p * (1 - p) / n)

print(round(margin(0.30, 500) * 100, 1))   # 4.0
print(round(margin(0.30, 100) * 100, 1))   # 9.0

At 30% accuracy on 500 questions, which is nanochat's default sample per CORE task during training, the 95% margin is about 4 percentage points each way. So 31% against 30% on one task means nothing. On 100 questions the margin is 9 points. Averaging many tasks narrows it, but its author still warns that CORE can be a little noisy, which is why the leaderboard also asks for validation bits per byte.

Three habits help. Run on the full benchmark for any comparison you plan to report. Train twice with different random seeds if your budget allows: if two runs of one recipe differ by as much as your "improvement", you have not shown an improvement. And change one thing at a time, so that you can say what caused the difference. When you publish, report the benchmark names, sample sizes and settings next to the scores.