intermediate · ~25 min
Training Dynamics Lab
Compare optimizers, learning rates, and batch sizes side by side and see why the same model trains completely differently.
Every optimizer so far has been tf.train.sgd — the plain, unadorned version of gradient descent.
Real training almost always reaches for something smarter. Here are three, trained side by side
on the exact same starting weights and data, so the only thing that differs is the optimizer
itself.
🔍 Deep dive: Momentum, RMSProp, and Adam — the same rule, three refinements
Plain SGD takes a step proportional to the current gradient, full stop. Momentum adds a running average ("velocity") of past gradients, so it keeps moving through small bumps and noisy directions instead of reacting to every wiggle. RMSProp instead tracks a running average of each parameter's squared gradient, and divides the step by its square root — parameters with consistently large gradients get smaller effective steps, and vice versa, so every parameter gets an appropriately-sized step regardless of its typical scale. Adam combines both ideas: momentum's smoothing and RMSProp's per-parameter scaling. That's why it's usually the default choice — it needs far less learning-rate tuning to behave well.
Production note
A "step" here covers one batch, not the whole dataset — so a batch size of 8 and a batch size of 64 see very different amounts of data per step. Comparing raw loss-per-step across different batch sizes can be misleading for that reason; if you change batch size, watch accuracy over wall-clock training time, not just step count.
Beginner tip
If a curve is jagged and never settles, the learning rate is usually too high. If it barely moves at all, it's usually too low. A smooth, steadily-decreasing curve is what "well-tuned" looks like — that's the shape Adam tends to produce almost by default.
One epoch vs. step distinction worth keeping straight: shrinking the batch size gives you more steps per epoch, not fewer — that's the "examples seen" nuance the ProCallout above is pointing at.
Playground
- Adam (lr=0.1)
- Momentum (lr=0.1)
- SGD (lr=0.1)
Mini project
Three training runs below hit the same problem with three different setups. Guess which cause produced each curve, then reveal the answers.
Curve A
Curve B
Curve C