intermediate · ~15 min
Explainability Basics
A trained model's gradient with respect to its input is a tiny saliency map — drag a probe point to see what it's sensitive to, and why.
Once a model is trained, a natural next question is: why did it predict that? This isn't just academic curiosity — a loan-approval model that can't explain a rejection is a legal liability in many jurisdictions, and a medical model a doctor can't interrogate is one they shouldn't trust. Explainability is the field of tools built to answer that question, and it exists precisely because modern models are too complex to just read off the answer.
For the linear regression model back at the start of this course, the answer was trivial — the
weight w literally is the explanation, the same number everywhere: bigger x always moves the
prediction by exactly w, full stop. For a nonlinear network like the one trained here, there's no
single number that explains every prediction, because the local sensitivity depends on where in
input space you're standing — the same input feature can matter a lot near one point and barely
at all near another.
That local sensitivity is just a gradient — the same gradient gradient descent has been computing this whole course, except now we're taking it with respect to the input, not the weights. This gradient-based approach is one of several explainability families in real use: model-agnostic techniques like LIME and SHAP instead probe a model by perturbing its inputs and watching how the output responds, which works even when you can't compute a gradient at all (a decision tree, say) at the cost of being slower and only approximate. Gradients are cheap and exact whenever the model is differentiable — which every network in this course is — so they're usually the first tool reached for.
🔍 Deep dive: This is exactly how real saliency maps work
A saliency map for an image classifier is this same gradient trick, scaled up to thousands of pixels: compute the gradient of the predicted class score with respect to every input pixel, and the pixels with the largest gradient magnitude are the ones the model is currently "looking at." Real saliency maps are noisier than this toy example suggests — a well-known caveat (addressed by techniques like SmoothGrad, which averages the gradient over many small random perturbations of the input) is that raw gradients can be surprisingly sensitive to tiny, imperceptible input changes.
Beginner tip
The arrow you'll see below points in the direction that most increases the model's confidence in Class A from that exact spot — not toward the boundary, and not toward any particular point. It's a purely local reading, valid only very close to where the probe currently sits.
Production note
This model was already trained (elsewhere, once) — this module is about inspecting a trained
model, not training one. tf.grad here differentiates with respect to the input tensor
instead of the model's weights, which is the same underlying autodiff machinery every
optimizer.minimize call in this course has used, just pointed in a different direction.
Playground
Drag the probe point (or add more) to see the model's local gradient — the direction that most increases its confidence in Class A — at that exact spot.
Click empty space to add a point, drag a point to move it, double-click to remove it.
|∂/∂x| = 0.072, |∂/∂y| = 0.042 — whichever is larger is the axis the model is more sensitive to right here.
Mini project
Drag the probe anywhere you like, then guess: is the model more sensitive to a small move in x or in y right there? Reveal to check.
Click empty space to add a point, drag a point to move it, double-click to remove it.