intermediate · ~15 min
Explainability Basics
A trained model's gradient with respect to its input is a tiny saliency map — drag a probe point to see what it's sensitive to, and why.
Once a model is trained, a natural next question is: why did it predict that? For the linear
regression model back at the start of this course, the answer was trivial — the weight w
literally is the explanation, the same number everywhere. For a nonlinear network like the one
trained here, there's no single number that explains every prediction, because the local
sensitivity depends on where in input space you're standing.
That local sensitivity is just a gradient — the same gradient gradient descent has been computing this whole course, except now we're taking it with respect to the input, not the weights.
🔍 Deep dive: This is exactly how real saliency maps work
A saliency map for an image classifier is this same gradient trick, scaled up to thousands of pixels: compute the gradient of the predicted class score with respect to every input pixel, and the pixels with the largest gradient magnitude are the ones the model is currently "looking at." Real saliency maps are noisier than this toy example suggests — a well-known caveat (addressed by techniques like SmoothGrad, which averages the gradient over many small random perturbations of the input) is that raw gradients can be surprisingly sensitive to tiny, imperceptible input changes.
Beginner tip
The arrow you'll see below points in the direction that most increases the model's confidence in Class A from that exact spot — not toward the boundary, and not toward any particular point. It's a purely local reading, valid only very close to where the probe currently sits.
Production note
This model was already trained (elsewhere, once) — this module is about inspecting a trained
model, not training one. tf.grad here differentiates with respect to the input tensor
instead of the model's weights, which is the same underlying autodiff machinery every
optimizer.minimize call in this course has used, just pointed in a different direction.
Playground
Drag the probe point (or add more) to see the model's local gradient — the direction that most increases its confidence in Class A — at that exact spot.
Click empty space to add a point, drag a point to move it, double-click to remove it.
|∂/∂x| = 0.072, |∂/∂y| = 0.042 — whichever is larger is the axis the model is more sensitive to right here.
Mini project
Drag the probe anywhere you like, then guess: is the model more sensitive to a small move in x or in y right there? Reveal to check.
Click empty space to add a point, drag a point to move it, double-click to remove it.