intermediate · ~25 min
Overfitting & Regularization
Shrink the training set, grow the model, and watch train/validation loss diverge — then fight back with L2 and dropout.
Every model so far has been graded on the same data it trained on. That's like grading a student on the exact questions they memorized from the answer key — it tells you nothing about whether they actually learned anything. A validation set is a held-out exam the model never studies from.
🔍 Deep dive: The bias-variance tradeoff
A model with too little capacity can't fit the training data well (high bias) — it underfits both sets equally. A model with too much capacity fits the training data perfectly, including its noise, and generalizes poorly (high variance) — it overfits. Between those extremes sits a sweet spot where validation error is lowest. Sweep capacity in the mini-project below and you'll trace this exact U-shaped curve.
Shrink the training set, grow the network, and watch the gap between the training and validation loss curves widen — that gap is overfitting, made visible.
Production note
L2 regularization adds a penalty for large weights, acting like a prior belief that simpler (smaller-weight) functions are more likely to generalize. Dropout randomly zeroes a fraction of neurons on every training step, which prevents any single neuron from being relied on too heavily — Hinton's original framing was that it trains an implicit ensemble of thinned subnetworks that all have to agree.
Beginner tip
If turning up dropout doesn't immediately shrink the gap the way L2 does, that's expected, not a bug — dropout adds noise to training, so it often needs more steps to show its benefit, and its effect on the raw weight sizes is genuinely different from L2's (L2 shrinks weights directly; dropout doesn't). Watch the validation curve over a longer run rather than judging from a handful of steps.
Playground
- train loss
- validation loss
Mini project
Same 20 training points, no regularization, only hidden units changes: 1, 2, 4, 8, 16, 32, 64. Run the sweep and watch validation loss trace the classic U-shaped bias-variance curve from the DeepDive above — too little capacity underfits, too much overfits, and there's a sweet spot in between.