# What is a neural network, and what are its weights?

Date: 2020-04-15 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/neural-network-weights/

**Answer:** A neural network is a large set of simple calculating units, loosely inspired by brain cells, arranged in layers that pass numbers to one another. Its weights are the adjustable numbers on the connections between units; training sets those weights so the whole network turns inputs into useful outputs.

The phrase "neural network" sounds like a brain in a box. The reality is more modest and easier to grasp. A neural network is a chain of simple arithmetic steps: multiply some numbers, add them up, squash the result into a convenient range, and pass it on. Each unit that does this is called an artificial neuron, and a network is many of them connected in layers.

What makes the network useful is not the arithmetic, which is trivial, but the particular numbers it multiplies by. These are its weights. A fresh network has random weights and produces nonsense. Training adjusts them, little by little, until the outputs are right most of the time. When people say a model "has 1.5 billion parameters", they mostly mean it has that many weights.

## One neuron, with real numbers

Let's build a single neuron that estimates whether you will enjoy a walk outside. It takes three inputs, each a number:

- Is it sunny? (1 for yes, 0 for no)
- How warm is it, on a scale from 0 (freezing) to 1 (perfect)?
- Are you tired? (1 for yes, 0 for no)

Each input has a weight that says how much it matters and in which direction. There is also one extra number, the bias, which acts like a starting point or baseline.

| Input | Value | Weight | Value × weight |
|---|---|---|---|
| Sunny | 1 | 2.0 | 2.0 |
| Warmth | 0.6 | 1.5 | 0.9 |
| Tired | 1 | −3.0 | −3.0 |
| Bias | | | −0.5 |
| **Total** | | | **−0.6** |

The neuron adds everything up and gets −0.6. Then it passes that total through an activation function, a fixed formula that turns any number into a more useful output. A common choice, the sigmoid, squeezes every number into the range 0 to 1. For −0.6 it gives about 0.35, which you can read as "probably not much fun today".

Now change one input: you are not tired (0). The total becomes 2.0 + 0.9 − 0.5 = 2.4, and the sigmoid gives about 0.92. Same neuron, same weights, different input, very different answer.

Look at what the weights encode. The large negative weight on "tired" means tiredness counts heavily against a walk. The positive weight on "sunny" counts for it. Nobody needs to type those preferences in as rules; in a trained network, they are learned from examples.

## From one neuron to a network

One neuron can only draw a simple line between "yes" and "no". The power comes from stacking them:

1. **The input layer** holds the raw numbers, such as pixel brightness in a photo or codes for words in a sentence.
2. **Hidden layers** each contain many neurons. Every neuron takes all the outputs of the layer before it, applies its own weights and passes its result forward.
3. **The output layer** gives the final answer, such as "cat" or "dog", or a score for every possible next word.

The weights pile up quickly. A layer of 1,000 neurons, each connected to 1,000 inputs, already has a million weights, plus 1,000 biases. OpenAI's GPT-2 language model has about 1.5 billion parameters spread across 48 layers.

Hidden layers are where the interesting part happens. In their 1986 Nature letter on backpropagation, David Rumelhart, Geoffrey Hinton and Ronald Williams showed that hidden units can come to represent useful features on their own. In one test, a network learning family trees developed hidden units that tracked ideas such as nationality and generation, even though no one had told it those concepts existed.

## How the weights get set

Training uses the same loop for almost every network. Show it an example, compare its output with the correct answer, and measure the error. Then work out, for every weight, whether nudging it up or down would shrink that error, and nudge it a tiny amount. The backpropagation method from that 1986 letter is the standard way to compute those nudges efficiently, working backward from the output layer. Repeat over millions of examples and the weights settle into values that work.

> **Try it:** With a calculator or a spreadsheet, rebuild the walk neuron above. Change the weight on "tired" from −3.0 to −1.0 and recompute both cases. You have just done by hand what training does automatically: changed a weight and watched the network's opinion shift.

## Why do bigger networks need so much data?

Every weight is a number the network has to learn, and learning a number takes evidence. A network with a handful of weights can be pinned down by a few dozen examples. A network with millions or billions of weights has room to store an enormous amount of detail, and if it sees too few examples, it uses that room to memorize them, quirks and all. This is called overfitting: the network does very well on its training examples and poorly on anything new.

A January paper from researchers at OpenAI and Johns Hopkins University, "Scaling Laws for Neural Language Models", measured this for language models. Its authors, led by Jared Kaplan, found that performance improves smoothly and predictably when model size and data grow together, but stalls if one grows while the other stays fixed. Their estimate: each time a model becomes eight times larger, it needs roughly five times more data to avoid overfitting.

There is good news in the same paper. Larger models are more sample-efficient, meaning they learn more from each example than small ones do. So data needs grow more slowly than size, but they still grow. That is why GPT-2 was trained on about 40 GB of text from some 8 million web pages, and why teams building ever larger networks spend so much effort collecting and cleaning data.

## Sources

- Learning representations by back-propagating errors: https://doi.org/10.1038/323533a0
- Scaling Laws for Neural Language Models: https://arxiv.org/abs/2001.08361
- Language Models are Unsupervised Multitask Learners: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf