model compression

The family of techniques that reduce a trained model's memory footprint, computation, latency or energy use while preserving as much of its accuracy as possible. The main approaches are quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. (fewer bits per number), pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well. and sparsitysparsity, n.: The property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation. (fewer nonzero parameters or structures), knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run. (training a smaller student to imitate a larger teacher), low-rank factorizationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation. of weight matrices, and weight sharingweight sharing, n.: Reusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data. or codebook schemes; efficient architecture design, as in mobile networks built from depthwise separable convolutionsdepthwise separable convolution, n.: A cheaper way to do the main operation of image networks: first scan each channel of the input separately, then mix the channels together with a tiny one-by-one step., is sometimes counted too. The methods compose: a model may be distilled, then pruned, then quantized. Compression is judged by the trade-off between size or speed and task quality, bearing in mind that headline metrics such as perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll. can hide losses in rarer abilities. It is essential for on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device. and edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers. and a major lever on inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text..

Model compression took the speech model from 1.2 GB to 90 MB so it could ship inside the app.