What is model compression?

Model compression makes an AI model smaller and less demanding to run, so it can work directly on a phone, a smart speaker, or another device with tight limits on memory, battery, and processing power. Without it, the models behind chatbots, image recognition, and voice assistants would need to live on a distant server — with the delay and privacy trade-offs that brings. Compression strips away parts of the model that contribute little to its output while holding onto the behaviour you actually want. The three core techniques — pruning, quantization, and knowledge distillation — are what make on‑device AI practical at all.

Pruning: cutting what does not matter

A neural network is made of connections between artificial neurons, each carrying a numerical weight. Many of those weights are so close to zero that removing them barely changes the model’s output. Pruning finds them and takes them out. The result is a sparser network that needs less memory and fewer calculations. Retraining after pruning lets the remaining connections adjust to compensate, so accuracy usually stays high. Some strategies trim gradually, cycle by cycle; others delete whole groups of redundant neurons at once. The trade-off is simple: aggressive pruning can eventually hurt quality, so developers search for the right balance for each model and task.

Quantization: working with less precision

Weights and activations are typically stored as high‑precision floating‑point values. But everyday tasks rarely need that amount of detail in every single calculation. Quantization maps those numbers to a coarser representation — using fewer bits per value, even down to simple integers or binary — which shrinks the memory footprint and speeds up operations because simpler arithmetic runs faster on most hardware. There can be a small drop in accuracy, but modern techniques such as quantization‑aware training minimise it. Many devices now run quantized models natively, with dedicated chips built to handle low‑precision maths efficiently.

Knowledge distillation: learning from a teacher

Instead of shrinking a large model directly, distillation trains a separate, smaller “student” to mimic the behaviour of a bigger “teacher.” The teacher produces soft probabilities — a richer set of outputs than plain right‑or‑wrong labels — and the student learns to match them. In doing so, the student absorbs only the essential patterns, using far fewer parameters. Distillation is especially useful when the teacher is too large to deploy practically but its knowledge can be compressed into a tiny student that still performs well. It often works best combined with pruning or quantization.

Why compression matters for on‑device AI

Compression lets AI run entirely on a device instead of needing a round trip to a data centre. That means faster responses, offline capability, and stronger privacy because sensitive data never leaves the device. On‑device AI is growing fast — from real‑time photo editing and voice typing to health monitoring — and compression is the quiet enabler behind it all. As models grow more capable, the techniques that shrink them without breaking them become just as important as the models themselves.