post-training quantization
In plain wordsShrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
Quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. applied to a model after training is complete, without the full training pipeline or data. In its simplest, data-free form each weight tensor is rounded to the nearest representable value using scales computed from its range ('round-to-nearest'). Calibration-based methods pass a few hundred sample inputs through the model to set activation ranges and to choose quantized weights that minimize each layer's output error rather than the weights' own error: GPTQGPTQ, prop. n.: A popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error. uses approximate second-order information to compensate rounding errors, AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them. rescales the channels that matter most to activations, SmoothQuantSmoothQuant, prop. n.: A method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better. shifts outlier magnitude from activations into weights, and rotation-based methods spread outliers before rounding. PTQ is cheap (minutes to hours on one GPU even for very large models) and is how most open-weight models are compressed for local use, but accuracy falls off sharply below about 4 bits, where quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding. or native low-bit training does better.
We ran post-training quantization with 512 calibration samples and lost under one point on our eval.
Origin
Standard practice for 8-bit deployment of convolutional networks by the late 2010s, when tools such as TensorFlow Lite and NVIDIA TensorRT shipped calibration utilities; the label distinguished it from quantization-aware training, as in Raghuraman Krishnamoorthi's 2018 Google whitepaper on quantizing convolutional networks. For large language models, where retraining is prohibitively expensive, PTQ methods such as LLM.int8() (2022), GPTQ (2022), SmoothQuant (2022) and AWQ (2023) became the norm.
Related
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.
- GPTQGPTQ, prop. n.: A popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error.
- AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them.
- SmoothQuantSmoothQuant, prop. n.: A method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better.
- LLM.int8()LLM.int8(), prop. n.: An early, widely used method for running very large language models with 8-bit numbers, roughly halving their memory needs, by keeping a few unusually large values at higher precision.
- quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.
Sources
- Frantar, E., Ashkboos, S., Hoefler, T. & Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323. arxiv.org/abs/2210.17323
- Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J. & Han, S. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10438. arxiv.org/abs/2211.10438
- Krishnamoorthi, R. (2018). Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv:1806.08342.
Mentioned in
post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., INT8INT8, abbr.: Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them., BitNetBitNet, prop. n.: A family of AI language models from Microsoft in which most of the learned numbers are limited to minus one, zero or one, so they need far less memory and energy to run., FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory., GPTQGPTQ, prop. n.: A popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error., LLM.int8()LLM.int8(), prop. n.: An early, widely used method for running very large language models with 8-bit numbers, roughly halving their memory needs, by keeping a few unusually large values at higher precision., QLoRAQLoRA, n.: A memory-saving version of LoRA that squeezes the big model into a compact 4-bit form while training small add-ons, letting very large models be customized on a single graphics card., quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding., SmoothQuantSmoothQuant, prop. n.: A method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better., TensorRTTensorRT, prop. n.: NVIDIA's software for making a trained neural network run as fast as possible on NVIDIA graphics chips, by merging and simplifying its calculations and testing which way of doing each one is quickest.