Welcome back
We’ve come a long way in the last few lessons. The loss tells us how wrong the network was. Gradient descent tells us to nudge every weight downhill. The learning rate tells us how big that nudge should be.
But we’ve quietly assumed something rather difficult. To step downhill, we need the slope for every single weight, including weights buried deep in the early layers, far from the answer. How can we possibly know how much each of those contributed to the mistake? Today we’ll borrow a detective’s method.
The case of the burnt dinner
Imagine a grand old country house. Dinner has been served, and it’s a disaster: the roast is burnt. The detective is called.
A clumsy detective would blame everyone in the house equally. A good detective starts at the scene and works backward.
She begins in the dining room. Who served this? The footman. But he only carried what he was given, so he’s barely to blame.
She steps back into the kitchen. Who cooked it? The cook. She left it in too long, but she was following the timing on a note. A fair share of blame there.
She steps back again. Who wrote the note? The housekeeper, who copied it from the butler, who misread the master’s instructions. More blame flows backward along that chain.
By working room by room, from the scene of the crime back to its source, the detective works out exactly how much each person contributed. Nobody needed to watch the whole house at once.
Backpropagation
This is what a network does, and it’s called backpropagation, which is short for “backward propagation of errors.”
Remember that in the forward pass, the signal travels from the input layer, through the hidden layers, to the output. Backpropagation goes the other way:
- Start at the scene. At the output, we know the loss: how wrong the answer was, and in which direction.
- Question the last layer. For each weight leading into the output, we work out how much it contributed to the error. That gives us its slope.
- Step back a room. Each neuron in the last hidden layer passed signals forward, so it gets a share of the blame, in proportion to how strongly it was connected. Now we can work out the slopes for its incoming weights.
- Keep stepping back, layer by layer, until we reach the beginning.
By the time the detective reaches the front door, every weight in the network has its slope. Then gradient descent does its job: every weight takes a small step downhill.
Why going backward is so clever
You might wonder why we don’t simply test each weight one at a time: nudge it, run the whole network again, and see what happens to the loss. That would work, but with millions of weights it would mean millions of forward passes for every single step. Far too slow.
Backpropagation finds every weight’s slope in one backward sweep, which costs roughly as much as one forward pass. It’s the detective who solves the whole case in a single walk through the house, rather than re-staging the dinner a million times. This one idea is a large part of why training big networks is possible at all.
What to carry with you
Backpropagation starts at the error and works backward through the layers, working out how much each weight contributed, so every weight knows which way to change.