Skip to content
Last Dictionary
Plane
Inference & EfficiencyHardware & ComputeDeep LearningMathematics & Statistics

quantization

nounAlso: quantize, quantized model, model quantization, low-bit quantization, weight quantization, quantisation, W4A16, W8A8

In plain wordsStoring an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.

  1. 1

    Neural networks

    The mapping of a model's weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself., and often its activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. and KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., from high-precision floating-point numbers (32 or 16 bits) to low-precision formats such as 8- or 4-bit integers or FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits. and FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory. floats. In uniform affine quantization each value is divided by a scale, rounded and offset by a zero-point, with scales shared per tensor, per channel or per small group of values; dequantization approximately reverses the mapping. Fewer bits shrink memory roughly in proportion (a 70-billion-parameter model needs about 140 GB at 16 bits and 35 GB at 4 bits) and speed up memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries. decoding; if activations are quantized too, low-precision matrix units also raise arithmetic throughput. Schemes are written like W4A16 (4-bit weights, 16-bit activations). Quantization is applied after training (post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.) or simulated during it (quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.); rare outlier values, more than average error, are the main obstacle at low bit widths.

    The 4-bit quantized model fits on one 24 GB card and scores within a point of the original.

  2. 2

    Signal processing

    In signal processing and information theory, the replacement of each value of a continuous or finely graded quantity by one of a limited number of representative levels, as when an analog-to-digital converter records a voltage as the nearest of its fixed set of codes, or when vector quantization swaps each vector for the closest entry of a learned codebook (as in VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate.); the difference between the original and its replacement is the quantization error. This older sense is the source of the machine-learning one.

q=clamp⁡(round⁡(xs)+z,  qmin⁡,  qmax⁡),x^=s (q−z)q = \operatorname{clamp}\left(\operatorname{round}\left(\frac{x}{s}\right) + z,\; q_{\min},\; q_{\max}\right), \qquad \hat{x} = s\,(q - z)

Origin

From signal processing, where quantizing continuous signals was analyzed with the rise of pulse-code modulation, notably by W. R. Bennett (1948). Low-precision neural networks were studied from about 1990; 8-bit integer inference spread with mobile deployment in 2015–2018 (for example Benoit Jacob and colleagues at Google, 2017), and 8- and then 4-bit quantization of large language models followed with LLM.int8(), GPTQ and AWQ in 2022–2023.

Usage note

Quantization changes the model's numbers and can change its behavior, unlike speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.. 'No loss' claims usually rest on perplexity or a few benchmarks; long-context, multilingual and tool-use abilities often degrade first. Unrelated to quantum computing.

Related

See also
  • post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
  • quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.
  • GPTQGPTQ, prop. n.: A popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error.
  • AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them.
  • GGUFGGUF, abbr.: A file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama.
  • floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
  • model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy.
  • BitNetBitNet, prop. n.: A family of AI language models from Microsoft in which most of the learned numbers are limited to minus one, zero or one, so they need far less memory and energy to run.
Contrast with
  • pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well.
  • knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.

Sources

  1. Jacob, B. et al. (2017). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. arXiv:1712.05877 (CVPR 2018). arxiv.org/abs/1712.05877
  2. Bennett, W. R. (1948). Spectra of Quantized Signals. Bell System Technical Journal 27(3).

Mentioned in

Hugging FaceHugging Face, prop. n.: A company whose website is the main public library for sharing AI models and datasets — often called the GitHub of machine learning — and which makes widely used free AI software., inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text., knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device., open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private., optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., CPUCPU, abbr.: The main general-purpose chip in a computer, good at running many different programs step by step. In AI systems it usually coordinates the work while specialized chips do the heavy math., decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers., floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster., FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values., FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers., inference providerinference provider, n.: A company that runs AI models on its own computers and sells access to them over the internet, charging by use, so customers need not buy or operate the hardware themselves., INT8INT8, abbr.: Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions., llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models., memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed., mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster., model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., OllamaOllama, prop. n.: A free app that makes it easy to download and run AI language models on your own computer with a single command, keeping your data on your machine., parameter countparameter count, n.: How many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them., parameter-efficient fine-tuningparameter-efficient fine-tuning, n.: A family of tricks for customizing a large AI model by training only a tiny fraction of its numbers, or a small add-on, leaving the rest untouched to save memory and money.