How do you diagnose a failed training run before buying a bigger machine?

Training fails quietly. There is often no error message, just a loss that will not fall or a score that stays poor. It is tempting to read that as "not enough power". Most of the time it is a bug, a setting or the data. A larger machine reproduces the same failure sooner, at a higher hourly price.

This project breaks a small network on purpose, in nine known ways, and tests if a short list of measurements can name each fault. The script is 13_doctor.py. It runs on a CPU in a few minutes at most.

The four checks

Many of these ideas are the opening moves of Andrej Karpathy's recipe for training networks; the code and rules here were written for this project.

import time, torch, torch.nn.functional as F

def can_memorize(make_model, xb, yb, steps=300):
    """Try three sensible rates. Ignore whatever rate the failing run used."""
    best = float("inf")
    for lr in (1e-4, 1e-3, 1e-2):
        model = make_model()
        opt = torch.optim.Adam(model.parameters(), lr)
        for _ in range(steps):
            loss = F.cross_entropy(model(xb), yb)
            opt.zero_grad(); loss.backward(); opt.step()
        best = min(best, loss.item())
    return best                          # above 0.5 for 16 examples: something is wrong

# inside your training loop
t0 = time.perf_counter()
xb, yb = next(batches)                   # time spent here is data waiting
t1 = time.perf_counter()
# ... forward, backward, step ...
t2 = time.perf_counter()
wait += t1 - t0; work += t2 - t1

Nine failures and what they look like

Each row was run with five seeds on scikit-learn's digits, 600 steps. The numbers are from the last seed. "Wait" is the share of time spent getting data. Measured here on an Apple M5 Max's CPU with PyTorch 2.15.

Some things to notice. A forgotten zero_grad, one missing line, still reached 80%. It looks like a model that needs more work, and you could tune it for days. The inputs row shows Adam shrugging off huge inputs in this small case; only the first loss, far above 2.30, gave it away. The starved run gets the same accuracy as the healthy one, with 84% of its time spent waiting, so its score could never reveal the problem.

How much should you trust this score?

The first version of the rules scored 36 of 50, including 0 of 5 on both the too-high rate and dead units. Rewriting them, by testing memorization at several rates and counting weights with no gradient, brought the score on digits to 50 of 50. That is tuning on the test, the very trap from project four, so the 50 is not independent evidence.

To get a fairer figure, the same final rules ran on a second, synthetic problem they were not written against. They got 48 of 50. The two misses were on the model that was too small, where 3 of 5 runs were named correctly. These are also faults that were planted one at a time. Real runs can have two at once, and language models and large images were not tested. Use the checks as a way of looking and take the verdicts as suggestions.

When is a bigger machine actually the right answer?

Measure it. The same code ran on the CPU and on the machine's GPU:

Small jobs lose time moving data to the GPU, and that cost beats the arithmetic it saves. The GPU only earned its place when the arithmetic was large. These are one machine's figures, with Apple's built-in GPU; other hardware was not tested, and the balance will differ.

Upgrading makes sense when all of these are true: the model memorizes a small batch and validation improves; waiting for data is a small share, since PyTorch's tuning guide lists loader settings to try first; the GPU is actually busy; and you ran out of memory by the budget in project seven after trying the cheaper fixes of adapters, shorter sequences and checkpointing. Even then, renting for one run can be better than buying, and Google's tuning playbook is a good reference on spending experiments wisely. A faster machine shortens the loop between your experiments, which has real value, but it only helps once each experiment means something.