Skip to content
Last Dictionary
Plane
Inference & EfficiencyLanguage Models & Transformers

SmoothQuant

proper noun

In plain wordsA method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better.

  1. A post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding. method, needing no retraining, that lets large language models run with both weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. and activations in 8-bit integers (W8A8), so their matrix multiplications can use integer hardware. The obstacle it removes is uneven difficulty: weights quantize easily, but a few activation channels carry values far larger than the rest, and one shared 8-bit scale cannot represent those and the small values accurately at once. SmoothQuant divides each activation channel by a smoothing factor and multiplies the corresponding row of the next layer's weights by the same factor, an exact reformulation that can be folded offline into the preceding layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. or linear layer; a migration strength α sets how much of the difficulty shifts from activations to weights. Its authors reported up to 1.56 times speedup and halved memory with negligible accuracy loss, and serving a 530-billion-parameter model on one node. Equivalent rescaling to tame outliers recurs in AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them. and in rotation-based methods.

    With SmoothQuant we moved from W8A16 to W8A8, and prefill got noticeably faster.

Y=(Xdiag⁡(s)−1)(diag⁡(s) W),sj=max⁡∣Xj∣αmax⁡∣Wj∣1−αY = \left(X \operatorname{diag}(s)^{-1}\right)\left(\operatorname{diag}(s)\, W\right), \qquad s_j = \frac{\max |X_j|^{\alpha}}{\max |W_j|^{1-\alpha}}

Origin

Introduced by Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth and Song Han of MIT and NVIDIA (arXiv, November 2022; ICML 2023), and adopted in NVIDIA's inference libraries.

Related

See also
  • post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
  • LLM.int8()LLM.int8(), prop. n.: An early, widely used method for running very large language models with 8-bit numbers, roughly halving their memory needs, by keeping a few unusually large values at higher precision.
  • AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them.
  • quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
  • layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.

Sources

  1. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J. & Han, S. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10438 (ICML 2023). arxiv.org/abs/2211.10438

Mentioned in

post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding., AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them., GPTQGPTQ, prop. n.: A popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error., LLM.int8()LLM.int8(), prop. n.: An early, widely used method for running very large language models with 8-bit numbers, roughly halving their memory needs, by keeping a few unusually large values at higher precision.