AWQ

Activation-aware Weight Quantization, a post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding. method that compresses the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. of large language models to low-bit integers (typically 4-bit, weight-only) while giving extra protection to the weights that matter most. Which weights matter is judged from the data: its authors found that roughly 1% of input channels, those that meet the largest activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., account for a disproportionate share of the damage that rounding does. Instead of keeping those channels at higher precision, which would require slow mixed-precision kernels, AWQ scales each input channel of the weights up by a factor tuned on a small calibration set and scales the matching activations down by the same factor; the layer computes the same function before rounding, but the important weights now lose less to it. The procedure needs no backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. or layer-by-layer reconstruction, so it runs quickly and depends little on the particular calibration data. AWQ checkpoints are widely supported by serving engines.

We shipped the AWQ build because it held up better than round-to-nearest on our coding evals.