advanced · ~30 min
Production Best Practices
Converting, shipping, and watching a model in production — quantization, versioning, training-serving skew, and drift.
Everything up to this point has been about getting a model to train well. Shipping it is a different set of problems — mostly ones that never show up in a notebook, only once real traffic and real time are involved.
Picking a format. A trained Keras model is usually saved as a SavedModel, which a full TensorFlow Serving instance can load directly. That's the right choice for a server with a GPU and no size constraints. For a phone, a microcontroller, or a browser tab, you convert instead: TFLite for mobile/embedded, or TensorFlow.js's own format for the browser (which is, not coincidentally, what every playground on this site has been running the whole time).
Quantization. Converting to TFLite is also where you'd typically quantize — storing weights as 8-bit integers instead of 32-bit floats. That's a 4x size reduction and a real speedup on hardware without a fast float unit, for a surprisingly small accuracy cost. The playground below actually does this: it trains a real model, quantizes its real weights to int8, and shows you the real before/after — not a formula.
🔍 Deep dive: Why int8 quantization barely hurts accuracy
A trained network's weights are rarely using their full float32 precision meaningfully — most of what a weight "means" is its rough magnitude and sign, not its 7th decimal digit. Mapping each layer's weights onto 256 evenly-spaced values spanning that layer's own min/max throws away precision the network mostly wasn't relying on. It isn't free — you can see the small accuracy gap in the playground — but for a 4x size and latency win, it's usually a very good trade.
Training-serving skew. The single most common way a model quietly gets worse after deployment:
preprocessing done differently at training time than at inference time — a normalization constant
computed differently, an image resized with a different algorithm, a categorical feature encoded
inconsistently. Building your preprocessing into a tf.data pipeline (or into the model itself, as
Keras preprocessing layers) and reusing that exact pipeline at serving time is the fix — see the
data pipelines module if it's available in your curriculum.
Reproducibility. Pin your dependency versions, seed your RNGs, and log the exact config used to produce every checkpoint you might deploy. "It worked on my machine three weeks ago" is not a debugging strategy you want to need.
Monitoring and rollback. Once a model is live, its input distribution can drift out from under it — user behavior changes, an upstream data source changes format, a season changes. Track the distribution of your model's inputs and outputs over time, not just an offline accuracy number that was true on the day you shipped. Roll out changes to a small slice of traffic first (a canary), and always keep the previous model one config change away from being live again.
Production note
tf.lite.TFLiteConverter.from_keras_model(model) with
converter.optimizations = [tf.lite.Optimize.DEFAULT] is the one-line version of the
quantization story above — it applies int8 quantization automatically during conversion, no
manual weight surgery required.
Beginner tip
None of this replaces getting the model right first. Production concerns are worth learning, but they're rarely why a model fails — a model that's undertrained or overfit will fail in production exactly as it failed in your notebook, just with an audience.
Playground