# What are parameters, loss and learning, and what do they look like when you watch them change?

Date: 2026-05-25 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/parameters-loss-learning-watch/

**Answer:** Parameters are the numbers a model is allowed to change, loss is one number saying how wrong it is, and learning is the repeated nudge of each parameter in the direction that lowers the loss. A two-parameter model makes this visible, because you can watch it rediscover the Celsius-to-Fahrenheit rule in about 50 steps.

A large language model has billions of parameters, too many to look at. The ideas are identical with two, and with two you can print every value at every step. This project needs only Python and NumPy, a library for arrays of numbers.

Here is the setup. Take 200 temperatures, in Celsius and Fahrenheit, with a little measurement noise added. Do not tell the program the conversion rule. Give it a model with two parameters, `w` and `b`, that predicts `w × input + b`, and let it find them.

## The learning loop in ten lines

```python
import numpy as np
rng = np.random.default_rng(0)
c = rng.uniform(-30, 40, 200)
f = 1.8 * c + 32 + rng.normal(0, 1.0, 200)   # the rule, plus noise
x = c / 40.0                                  # scaled input

w, b, lr = 0.0, 0.0, 0.3
for step in range(301):
    error = w * x + b - f
    loss = (error ** 2).mean()                # how wrong, as one number
    if step in (0, 1, 2, 5, 10, 20, 50, 100, 300):
        print(step, round(w, 3), round(b, 3), round(loss, 3))
    w -= lr * 2 * (error * x).mean()          # nudge against the slope of the loss
    b -= lr * 2 * error.mean()
print("slope per degree:", round(w / 40, 3), "offset:", round(b, 2))
```

The **loss** here is the mean squared error: the average of each prediction's error, squared. The two lines that update `w` and `b` use the **gradient**, the slope of the loss with respect to each parameter. It answers one question: if this number went up a little, would the loss go up or down, and how fast? Moving each number against its slope is gradient descent. The **learning rate**, `lr`, says how large each move is.

## What it looks like

Measured here (NumPy 2.5.4, Python 3.14, Apple M5 Max). `w` is per 40 degrees because the input was scaled; the true value is 72.

| Step | w | b | Loss |
|---|---|---|---|
| 0 | 0.000 | 0.000 | 3,544.176 |
| 1 | 17.343 | 27.542 | 1,052.205 |
| 2 | 28.188 | 36.537 | 546.490 |
| 5 | 46.789 | 38.213 | 177.752 |
| 10 | 61.328 | 34.718 | 32.403 |
| 20 | 69.986 | 32.433 | 2.031 |
| 50 | 71.845 | 31.941 | 1.042 |
| 300 | 71.855 | 31.939 | 1.042 |

The final rule is F = 1.796 × C + 31.94, against the true 1.8 and 32. Three things deserve a second look.

- `b` overshoots to 38 and comes back. Learning is not a straight path, even on the simplest possible problem.
- The loss stops at 1.042, not zero. The added noise has a variance of about 1, and no line can predict random noise. A loss that stops falling is not always a problem.
- Early gains are huge and late gains are tiny. The drop from step 0 to step 1 is larger than everything that follows. Reading a curve on a log scale helps.

To be sure a gradient is right, measure it. Nudge `w` by a tiny amount each way and see how much the loss moves. At `w=3, b=5` the formula gives −53.97233 and nudging gives −53.97233. This check catches real bugs when you write a training loop by hand.

## The setting that decides everything

The [learning rate](https://developers.google.com/machine-learning/crash-course/linear-regression/hyperparameters) is the first setting to learn. Too low and progress crawls; too high and the model never settles. Here is that advice as numbers, each run lasting 100 steps from the same start.

| Learning rate | Loss after 100 steps | What happened |
|---|---|---|
| 0.001 | 2,568.6 | Barely started |
| 0.01 | 381.6 | Still far off |
| 0.1 | 1.061 | Nearly there |
| 0.3 | 1.042 | Done |
| 0.6 | 1.042 | Done |
| 1.0 | 792,760,607,948 | Exploding |
| 1.2 | Diverged | Stopped once the loss passed 10^30 |
| 2.0 | Diverged | Stopped once the loss passed 10^30 |

Note how close good and catastrophic are: 0.6 works, and 1.0 sends the loss into the hundreds of billions. There is no warning before the edge.

Scaling the input matters just as much. With raw Celsius values instead of `c / 40`, the rates 0.01 and 0.003 diverged, and the stable rate of 0.001 reached only a loss of 631 after 100 steps. The same data, and the same model, needs far more steps when one input is large. This is why training recipes normalize inputs.

Real training rarely uses this plain update. [Adam](https://arxiv.org/abs/1412.6980) keeps running estimates of each parameter's gradients and scales each step accordingly, which makes it far less fussy about these settings. It also means extra stored numbers for each parameter, which comes back in the memory project.

> **Try it:** Set the true rule to `f = -40 + 0.5 * c`, keep everything else, and predict the final `w` before you run it.

## Why does a bigger model not simply learn more?

A straight line cannot represent everything, and more lines do not help unless something bends them. XOR is the classic test: output 1 when exactly one of two inputs is on. Trained for 5,000 steps with a sigmoid output, the straight-line model settles at 0.5 for all four cases, the best it can do. Adding a hidden layer with a non-linear function changes what the model can express at all.

But capacity does not guarantee success. These are results from 50 random starts each, measured here:

| Hidden units | Parameters | Solved XOR |
|---|---|---|
| 0 (a line) | 3 | No: outputs stay at 0.5 (one run) |
| 2 | 9 | 42 of 50 |
| 3 | 13 | 46 of 50 |
| 4 | 17 | 50 of 50 |
| 8 | 33 | 50 of 50 |

Two hidden units are enough in principle, yet 8 of 50 starts got stuck. Spare capacity made the search easier. That is part of why practical models are larger than the bare minimum, and why the seed can change your result. More parameters also give more room to memorize instead of generalize, which is the trap [Andrej Karpathy's training recipe](https://karpathy.github.io/2019/04/25/recipe/) tells you to use on purpose: get a model that can overfit first, then rein it in. The size of a model sets what it could learn. Whether it does depends on data, settings and the honest checks that come next in this series.

## Sources

- Linear regression: Hyperparameters: https://developers.google.com/machine-learning/crash-course/linear-regression/hyperparameters
- Adam: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980
- A Recipe for Training Neural Networks: https://karpathy.github.io/2019/04/25/recipe/

---

From the Guide to Artificial Intelligence, by Robert Scales, published by Holistic Hybrid Inc. Share and adapt it under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/): credit the guide, link to https://guidetoartificialintelligence.com/en/learn/parameters-loss-learning-watch/, no commercial use without permission, and keep the same licence.