Welcome back
So far we’ve built a whole network. Signals flow forward through layers of neurons, each one using its weights, bias and activation function, until an answer comes out at the end. That journey is called the forward pass.
But a network that only guesses never gets better. Before it can learn, it needs to know one thing: how wrong was I? Today we’ll find out how it measures that, with the help of an archer.
The archer and the target
Picture an archer at practice on a quiet autumn morning. She draws, releases, and the arrow thuds into the straw target, a hand’s width to the left of the bullseye.
She doesn’t just note “missed.” She sees how far she missed, and in which direction. A hair’s breadth off is very different from hitting the edge of the target, which is different again from the arrow sailing off into the hedge.
That distance between where the arrow landed and where it should have landed is precious information. Without it, she would have no idea how to adjust.
The loss function
A neural network has the same need, and it meets it with something called the loss function.
During training, we show the network an example and the correct answer. For instance: “Here’s a photo, and it’s a cat.” The network makes its guess in the forward pass. Then the loss function compares that guess with the correct answer and produces a single number: the loss.
- A small loss means the guess was close. An arrow near the bullseye.
- A large loss means the guess was far off. An arrow in the hedge.
- A loss of zero would mean a perfect bullseye.
That one number becomes the network’s whole sense of how it’s doing. Everything it learns from now on is in the service of making that number smaller.
A simple example
Suppose we’re teaching a network to guess the price of a house, and the true price is 300,000. The network guesses 280,000.
One common loss function says: take the difference, and square it. The difference is 20,000, and squaring it gives a large number. This is called mean squared error, because when we have many examples we take the average (the “mean”) of all those squared differences.
Why square it? Two gentle reasons. First, it makes every miss count as positive, whether the guess was too high or too low; a miss to the left is still a miss. Second, it makes big misses count much more than small ones. An arrow in the hedge is treated as far more serious than one just off centre, which is usually what we want.
For questions with categories, like “cat or dog?”, a different loss function is used, but the idea is the same: a number that’s small when the guess is right and confident, and large when it’s wrong.
Why one number is enough
It might seem strange that a network with millions of weights is guided by just one number. But think of the archer: she has dozens of muscles involved in every shot, yet one simple fact, “too far left,” is enough to set them all adjusting.
The loss is the network’s version of that fact. It doesn’t say which weights were at fault. It just says how far the arrow strayed. Working out what to change is the next part of the story.
What to carry with you
The loss function compares the network’s guess with the right answer and gives one number for how wrong it was. Learning means making that number smaller.