intermediate · ~25 min
Sequence Models: RNNs
Order-sensitive data needs a network with memory — step through a recurrent cell by hand, then train one to detect palindromes.
Every model in this course so far has treated its input as a fixed-size bag of numbers — flatten
it, feed it in, done. Order didn't matter: shuffling the pixels of an image the same way every
time wouldn't change what a Dense layer computes. Some data is fundamentally sequential, though
— language, audio, time series — where order is the information. A recurrent network handles
this by carrying a hidden state forward
from one timestep to the next, updating it as each new input arrives.
The mini-task below — detecting whether a bit sequence is a palindrome — is a clean illustration of why order matters: the same set of 0s and 1s can be a palindrome or not depending purely on their arrangement, so any model that ignores order is stuck at 50% accuracy no matter how it's trained.
Beginner tip
Feed a sequence in one bit at a time in section 2 below and watch the hidden-state bar chart change color and intensity with every step — that vector is the network's entire "memory" of everything it has read so far, compressed into a fixed size regardless of how long the sequence gets.
🔍 Deep dive: Vanishing and exploding gradients through time
Training a recurrent network means backpropagating the loss through every timestep, one after another — the same gradient-times-weight-matrix multiplication happening once per layer in a deep feedforward network now happens once per timestep. For a long sequence that's effectively an extremely deep network, and the same instability this course has already covered (an SGD learning rate that's "too high" makes the loss diverge) shows up here in a different form: repeatedly multiplying by the same weight matrix can make gradients shrink toward zero (vanishing) or grow without bound (exploding) as they propagate back through many timesteps. This is exactly why LSTMs and GRUs exist — their gating mechanisms are specifically designed to let gradients flow backward through time without vanishing.
Production note
This uses a plain SimpleRNNCell for transparency — real sequence models almost always reach
for LSTM or GRU cells instead, which add gating mechanisms precisely to fix the vanishing-
gradient problem above and can learn dependencies across much longer sequences.
Playground
1. Train a palindrome detector
200 random 6-bit sequences, half palindromes. A model that ignores order entirely can't beat 50% here — only one that tracks position can.
- accuracy
- loss
2. Watch the hidden state evolve, one bit at a time
Hidden state (8 units)
Mini project
10-bit sequences are harder than the 6-bit ones above — there's more to remember across more timesteps. Tune hidden units until accuracy passes 95%.