quantization-aware training
In plain wordsTraining or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.
Training or fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. a model with quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. simulated in the forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., so that the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. adapt to rounding error before deployment. 'Fake-quantization' operations round weights (and often activations) to the target grid and immediately convert them back to floating point, so the loss reflects quantized behavior while full-precision master weights receive the updates. Because rounding has zero gradient almost everywhere, backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. passes gradients through it as if it were the identity function, the straight-through estimator; scales and clipping ranges can be learned too. QAT typically recovers most of the accuracy lost by post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding. at 4 bits and below, at the price of a training run and data, and some open-weight model families have been released with QAT checkpoints for 4-bit use. Training natively in low precision, as in BitNetBitNet, prop. n.: A family of AI language models from Microsoft in which most of the learned numbers are limited to minus one, zero or one, so they need far less memory and energy to run., is the limiting case.
Post-training quantization to 4 bits cost us six points, so we ran a day of QAT and got five back.
Origin
Simulated quantization during training goes back to binarized and low-bit networks such as BinaryConnect (2015) and to the straight-through estimator discussed by Yoshua Bengio, Nicholas Léonard and Aaron Courville (2013). Benoit Jacob and colleagues at Google (December 2017) made it the standard recipe for 8-bit integer inference, and TensorFlow tooling popularized the name 'quantization-aware training'.
Related
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
- BitNetBitNet, prop. n.: A family of AI language models from Microsoft in which most of the learned numbers are limited to minus one, zero or one, so they need far less memory and energy to run.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster.
- post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
Sources
- Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H. & Kalenichenko, D. (2017). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. arXiv:1712.05877. arxiv.org/abs/1712.05877
- Bengio, Y., Léonard, N. & Courville, A. (2013). Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv:1308.3432.
Mentioned in
INT8INT8, abbr.: Storing numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions., mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster., post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., BitNetBitNet, prop. n.: A family of AI language models from Microsoft in which most of the learned numbers are limited to minus one, zero or one, so they need far less memory and energy to run., straight-through estimatorstraight-through estimator, n.: A shortcut that lets networks learn even when they contain steps with no useful slope, such as rounding: during learning, the computer simply pretends the step passed its input straight through.