Turn Not the Tap Too Boldly, Lest Thou Be Scalded

Welcome back

Last time, we were lost in the fog. We learned about gradient descent: the network feels the slope of the loss under its feet and takes a small step downhill, over and over, until it reaches a valley.

But we skipped over one quiet little word: small. How small should each step be? It turns out this choice matters enormously. To feel why, let’s step into the shower.

The shower that’s never right

You know this moment. You step into the shower and the water’s freezing. You reach for the hot tap.

If you turn it a tiny fraction, you wait. And wait. The water creeps from freezing to merely cold, then to cool. You’ll get there eventually, but you might be late for work.

If you crank it all the way, a moment later you’re scalded. You lunge for the tap and twist it back hard, and now it’s freezing again. Back and forth you go, overshooting each time, never quite settling.

If you turn it a sensible amount, then wait and adjust, you’re standing in warm water within a few seconds.

You’ve been solving the learning-rate problem every morning of your life.

The learning rate

In gradient descent, the gradient tells the network which direction to move each weight. The learning rate decides how far to move it. It’s a single number, chosen by the person training the network, that sets the size of every step.

Each weight changes by roughly:

learning rate × the slope for that weight

So a bigger learning rate means bolder steps, and a smaller one means more cautious steps.

Too small, too large, just right

Let’s go back to the foggy hillside from last time, now with the shower in mind.

• Too small a learning rate: the hiker takes tiny shuffling steps. Every step is safe and downhill, but reaching the valley takes an age. Training becomes painfully slow, and it may stop long before the network is any good.
• Too large a learning rate: the hiker takes giant leaps. Instead of stepping down into the valley, they leap clean over it and land high up on the opposite slope. Then they leap back. The loss jumps around wildly instead of shrinking, and sometimes it even grows larger and larger. That’s the scalding shower.
• Just right: steps big enough to make real progress, small enough to settle into the valley rather than bounce across it.

There is no single perfect value that works for every network. People usually try a few and watch how the loss behaves. If it creeps down slowly, they try a bigger value. If it leaps around, they try a smaller one.

A clever compromise

Many people use a gentle trick: start with a larger learning rate and gradually make it smaller as training goes on. It’s like turning the tap generously at first, while the water is still freezing, and then making finer and finer adjustments as you get close to the perfect temperature.

This idea, changing the learning rate as training goes on, is often called a learning rate schedule.

What to carry with you