advanced · ~30 min
Improving Your Models
The practitioner's toolkit: learning rate, batch size, capacity, dropout, augmentation, early stopping, and checkpoints.
You have a model that trains. It's mediocre. Now what? This module is the practitioner's checklist — the knobs worth turning, in roughly the order they're worth trying, plus the two callbacks that make experimenting with them safe instead of tedious.
The knobs. Most of these should feel familiar from earlier modules — this is about having them all in one place as a toolkit, not new theory:
- Learning rate — too high and loss visibly bounces or diverges; too low and training crawls. Almost always the first thing to tune.
- Batch size — smaller batches mean noisier (but more frequent) gradient updates; larger batches are smoother but need more memory and, often, a proportionally higher learning rate.
- Number of layers / neurons per layer — more capacity can fit more complex patterns, but (see the overfitting module) also memorizes more easily with too little data.
- Dropout — randomly zeroes a fraction of units each training step, discouraging the network from over-relying on any one of them.
- Data augmentation — for image data especially, randomly flipping/rotating/cropping training
examples is a cheap way to synthetically grow a small dataset. Applied as model layers
(
RandomFlip,RandomRotation), it's automatically a no-op outside training.
The safety net. Turning all these knobs means running a lot of experiments, and a lot of those experiments will overfit or waste time training past their useful point. Two callbacks fix that:
tf.keras.callbacks.EarlyStopping(
patience=3,
restore_best_weights=True,
)
EarlyStopping watches
validation loss every epoch. If it hasn't improved for patience epochs in a row, training stops
— and with restore_best_weights=True, the model's weights are rolled back to whichever epoch
actually had the best validation loss, not just whatever the final (already-degrading) epoch
happened to produce. ModelCheckpoint(save_best_only=True) does the complementary job: writing the
best-so-far model to disk as training goes, so a crashed run or a patience set too low doesn't
cost you the good checkpoint.
🔍 Deep dive: Why restore_best_weights matters as much as patience itself
Without restore_best_weights=True, EarlyStopping only decides when to stop — the model you
end up with is still whatever the last epoch produced, which by definition is patience epochs
past the best one, i.e. already partway back down the overfitting curve. The playground below
makes this concrete: toggle early stopping off and watch validation loss keep climbing long after
its best point; toggle it on and the run stops itself and snaps back to that best point instead.
Production note
ReduceLROnPlateau is the sibling callback worth knowing: instead of stopping, it lowers the
learning rate when a metric plateaus, letting training keep making small refinements instead of
ending outright. Reach for EarlyStopping when you want a run to end itself; ReduceLROnPlateau
when you want it to adapt instead.
Beginner tip
If a change to one slider seems to do nothing, try a bigger change — a learning rate 20% higher rarely looks different, but 10x higher usually does. Hyperparameter effects are rarely linear.
Playground
- train loss
- validation loss
Quiz
Select the right code
1. Which snippet correctly stops training after 3 epochs without improvement and restores the best-performing weights?
2. Which snippet correctly saves only the single best checkpoint, by validation loss?
3. Which snippet applies data augmentation only during training, with no effect at inference time?