Of Light Made Even, That No Face Be Lost in Shadow

Welcome back

Last time, we rode a bicycle and met optimisers. Momentum carries steady progress through the hollows, and Adam gives each weight its own gear. Our network now steps downhill with real skill.

But there’s a quieter problem we haven’t faced yet. It isn’t about how the network steps. It’s about the signals themselves, and whether they arrive in a sensible range.

The badly lit group photo

Imagine photographing a family at a summer lunch. Grandmother sits in bright sun by the window. Her grandson sits in deep shade under the awning. Click.

In the photo, Grandmother is a white glare and the boy is a dark smudge. Both faces were really there. The camera simply couldn’t handle two such different levels of light at once.

A good photographer fixes this before pressing the button. She adds a reflector, softens the sun, and brings everyone into a similar range of brightness. Now every face shows its detail.

Numbers in the glare and in the shade

A neural network faces the same trouble with its numbers.

Imagine a network predicting house prices. One input is the number of bedrooms: 2, 3, maybe 5. Another is the floor area in square feet: 1,200, 2,500, 4,000. The second input is in the blinding sun. Its numbers are hundreds of times bigger, so it swamps the weighted sum, and its gradients dominate the learning. The bedrooms are lost in shadow.

The remedy is called normalisation: bringing values onto a common, even scale before the network uses them.

The most common recipe has two small steps:

1. Centre it. Subtract the average, so the values sit around zero.
2. Even out the spread. Divide by how spread out the values typically are (the standard deviation), so most values land roughly between minus two and plus two.

Now bedrooms and floor area speak at the same volume, and each can earn its true weight through learning.

Keeping the light even inside the network

Normalising the inputs is a great start. But as signals travel through layer after layer, they can drift: brighter here, dimmer there. Worse, they keep shifting as the weights change during training. Each layer is trying to learn while the lighting from the layer before keeps changing. That makes learning slow and fragile.

So we normalise inside the network too, between layers.

• Batch normalisation looks at a whole batch of examples (remember the baking trays from Lesson 15). For each neuron, it centres and evens out that neuron’s values across the batch.
• Layer normalisation looks at one example at a time and evens out the values across all the neurons in a layer. It’s a favourite in the language models we’ll meet later.

There’s one thoughtful touch. After evening things out, each normalisation step has two small adjustable dials of its own, learned like any weight. If the network finds that a brighter or shifted signal works better, it can turn the dials and restore it. Normalisation sets a sensible default; it doesn’t force one.

What to carry with you