How do you know your model actually improved?

A higher number on one run is weak evidence. Models are trained with random starting points, tests hold a limited number of examples, and people tend to stop looking once the score goes up. Each of those can hand you an "improvement" that vanishes the next time you try.

This project measures four traps directly, using scikit-learn's digits and a set of pure random numbers. The companion script, 04_honest_testing.py, ran in about five seconds here on an Apple M5 Max with two processor threads. For testing language models specifically, including bits per byte and benchmarks, see how to tell whether a language model you trained is any good. This article covers what applies to any model.

Trap 1: luck looks like progress

Same data, same recipe, 20 different random seeds. Training on 300 digit images and testing on another 300:

So a three-point "improvement" can appear with no change at all except the seed. Before you compare two ideas, find out how much the score moves when nothing else changes. A change smaller than that band is not yet evidence.

Trap 2: comparing totals instead of examples

Two models tested on the same 300 images give you more information than two percentages. Count where they disagree.

import numpy as np

def compare(a_right, b_right, rounds=5000, seed=0):
    """a_right and b_right: arrays of True/False, one per test example."""
    rng, n = np.random.default_rng(seed), len(a_right)
    diffs = []
    for _ in range(rounds):
        i = rng.integers(0, n, n)                # resample the test set
        diffs.append(b_right[i].mean() - a_right[i].mean())
    return {"only_a": int((a_right & ~b_right).sum()),
            "only_b": int((b_right & ~a_right).sum()),
            "ci95": np.percentile(diffs, [2.5, 97.5]).round(4)}

This is a paired bootstrap: it reuses the test set many times, drawn at random with repeats, to see how much the difference would wobble with a different sample. Measured here:

The second result looks clearly better: 6 examples to 1. Yet the interval touches zero. An honest report says "probably better, and a larger test would settle it". Only seven of 300 test images even separate the two models. Everything rests on those seven, which is a good reason to read them.

Trap 3: the test set wears out

Every time you peek at a test score and react, some of the test set leaks into your decisions. To measure this, the script tried 60 settings on a small training set, kept whichever scored best on a test set of 60 images, and then checked that winner on 900 fresh images. It repeated this 20 times.

The gap of 1.2 points is modest here, because these settings do not differ wildly and the digits are easy. It grows with more tries, smaller tests and harder problems. Keep three piles: training data, a development set you may peek at freely, and a final test you open rarely. If you change anything because of the final test, it has become a development set.

Trap 4: leakage turns noise into skill

Data leakage happens when information unavailable at prediction time gets into model building. This is the most dramatic result in the project. The data is 100 rows of 2,000 random features with random labels, so nothing can be learned and the true skill is 50%.

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

# WRONG: choose the features using every row, test rows included
chosen = SelectKBest(f_classif, k=20).fit(N, Y).transform(N)
leaky = cross_val_score(LogisticRegression(max_iter=500), chosen, Y, cv=5).mean()

# RIGHT: choosing features is part of training, redone inside each split
honest_model = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=500))
honest = cross_val_score(honest_model, N, Y, cv=5).mean()

Same data, one line in a different place, and a useless model reports 84%. Common real-world forms: scaling on all the data, near-duplicates on both sides of the split, and a feature that exists only after the outcome is known, such as "refund issued" used to predict refund requests.

Two cheap sanity checks catch many of these. Train with shuffled labels: an honest pipeline must fall to chance, and here it scored 8.67% against a chance level of 10%. And compare with a baseline: always answering the most common digit got 10.33% on this test.

What should a results note contain, so that someone else can trust it?

Six lines are enough, and they take a few minutes to write:

The note's real value is the discipline of writing the decision after the numbers. For tools you use instead of train, the same idea applies with a small set of realistic tasks. A low score is useful too, since it tells you where to look. What wastes months is a high score you cannot explain.