# How do you diagnose a failed training run before buying a bigger machine?

Date: 2026-09-23 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/diagnose-training-failures-before-bigger-machine/

**Answer:** Run four cheap checks: the first loss against what guessing should give, whether the model can memorize 16 examples, how large the gradients are and where the time goes. A small test harness that broke a model in nine ways got the cause right in 50 of 50 runs on the problem it was tuned on and 48 of 50 on another, and in none of those nine would more GPU have fixed it.

Training fails quietly. There is often no error message, just a loss that will not fall or a score that stays poor. It is tempting to read that as "not enough power". Most of the time it is a bug, a setting or the data. A larger machine reproduces the same failure sooner, at a higher [hourly price](/en/learn/what-training-llm-costs-2026/).

This project breaks a small network on purpose, in nine known ways, and tests if a short list of measurements can name each fault. The script is `13_doctor.py`. It runs on a CPU in a few minutes at most.

## The four checks

Many of these ideas are the opening moves of [Andrej Karpathy's recipe for training networks](https://karpathy.github.io/2019/04/25/recipe/); the code and rules here were written for this project.

| Check | How | What it tells you |
|---|---|---|
| First loss | Compare with ln(number of classes): 2.30 for 10 classes | Much higher means bad scaling or a bad start |
| Memorize a small batch | Try to reach near-zero loss on 16 examples, at a few rates | If it cannot, stop: the model or the loop is broken, or it is too small |
| Gradient size over time | Log the overall gradient size; check how many weights get none | Zero means no flow; steady growth means gradients are piling up |
| Where the time goes | Time spent waiting for data versus computing | Waiting means a faster GPU will sit idle |

```python
import time, torch, torch.nn.functional as F

def can_memorize(make_model, xb, yb, steps=300):
    """Try three sensible rates. Ignore whatever rate the failing run used."""
    best = float("inf")
    for lr in (1e-4, 1e-3, 1e-2):
        model = make_model()
        opt = torch.optim.Adam(model.parameters(), lr)
        for _ in range(steps):
            loss = F.cross_entropy(model(xb), yb)
            opt.zero_grad(); loss.backward(); opt.step()
        best = min(best, loss.item())
    return best                          # above 0.5 for 16 examples: something is wrong

# inside your training loop
t0 = time.perf_counter()
xb, yb = next(batches)                   # time spent here is data waiting
t1 = time.perf_counter()
# ... forward, backward, step ...
t2 = time.perf_counter()
wait += t1 - t0; work += t2 - t1
```

## Nine failures and what they look like

Each row was run with five seeds on scikit-learn's digits, 600 steps. The numbers are from the last seed. "Wait" is the share of time spent getting data. Measured here on an Apple M5 Max's CPU with PyTorch 2.15.

| What was broken | Validation accuracy | Training accuracy | Wait | Doctor's verdict | Will a bigger machine help? |
|---|---|---|---|---|---|
| Nothing (healthy) | 97.1% | 99.3% | 9% | Healthy | Only if you need speed |
| Learning rate far too high | 10.0% | 10.0% | 9% | Unstable | No: lower the rate |
| Learning rate far too low | 13.1% | 13.1% | 9% | Rate too low | No: raise the rate |
| Labels shuffled | 8.9% | 29.5% | 9% | Labels mean nothing | No: fix the labels |
| `zero_grad` forgotten | 80.4% | 79.2% | 9% | Gradients pile up | No: fix the loop |
| Inputs not scaled | 96.2% | 100% | 9% | Input scale | No: scale the inputs |
| All hidden units dead at the start | 10.0% | 10.1% | 9% | No gradient | No: fix the start or activation |
| Only 40 training examples | 78.2% | 100% | 2% | Overfitting | No: more or better data |
| Model far too small (2 units) | 23.3% | 23.5% | 13% | Cannot memorize | No: a larger model, which probably fits |
| Slow data loading | 97.1% | 99.3% | 84% | Input bottleneck | No: faster disk or loaders |

Some things to notice. A forgotten `zero_grad`, one missing line, still reached 80%. It looks like a model that needs more work, and you could tune it for days. The inputs row shows Adam shrugging off huge inputs in this small case; only the first loss, far above 2.30, gave it away. The starved run gets the same accuracy as the healthy one, with 84% of its time spent waiting, so its score could never reveal the problem.

## How much should you trust this score?

The first version of the rules scored 36 of 50, including 0 of 5 on both the too-high rate and dead units. Rewriting them, by testing memorization at several rates and counting weights with no gradient, brought the score on digits to 50 of 50. That is tuning on the test, the very [trap from project four](/en/learn/did-your-model-improve-honest-testing/), so the 50 is not independent evidence.

To get a fairer figure, the same final rules ran on a second, synthetic problem they were not written against. They got 48 of 50. The two misses were on the model that was too small, where 3 of 5 runs were named correctly. These are also faults that were planted one at a time. Real runs can have two at once, and language models and large images were not tested. Use the checks as a way of looking and take the verdicts as suggestions.

> **Try it:** Add these four checks to a training script you already have, then break it on purpose by removing `zero_grad`, and see which check catches it first.

## When is a bigger machine actually the right answer?

Measure it. The same code ran on the CPU and on the machine's GPU:

| Job | CPU | GPU | Result |
|---|---|---|---|
| Small model, 400 steps, healthy | 0.16 s | 1.13 s | GPU 7 times slower |
| Same job, starved for data | 1.33 s | 2.15 s | GPU still slower |
| Large model (2,048-wide layers, batch of 512), 30 steps | 1.38 s | 0.23 s | GPU 6 times faster |

Small jobs lose time moving data to the GPU, and that cost beats the arithmetic it saves. The GPU only earned its place when the arithmetic was large. These are one machine's figures, with Apple's built-in GPU; other hardware was not tested, and the balance will differ.

Upgrading makes sense when all of these are true: the model memorizes a small batch and validation improves; waiting for data is a small share, since [PyTorch's tuning guide](https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html) lists loader settings to try first; the GPU is actually busy; and you ran out of [memory by the budget in project seven](/en/learn/memory-budget-fit-vs-train/) after trying the cheaper fixes of adapters, shorter sequences and checkpointing. Even then, renting for one run can be better than buying, and [Google's tuning playbook](https://github.com/google-research/tuning_playbook) is a good reference on spending experiments wisely. A faster machine shortens the loop between your experiments, which has real value, but it only helps once each experiment means something.

## Sources

- A Recipe for Training Neural Networks: https://karpathy.github.io/2019/04/25/recipe/
- Performance Tuning Guide: https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html
- Deep Learning Tuning Playbook: https://github.com/google-research/tuning_playbook

---

From the Guide to Artificial Intelligence, by Robert Scales, published by Holistic Hybrid Inc. Share and adapt it under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/): credit the guide, link to https://guidetoartificialintelligence.com/en/learn/diagnose-training-failures-before-bigger-machine/, no commercial use without permission, and keep the same licence.