straight-through estimator

A heuristic for backpropagating through operations whose true derivative is zero almost everywhere or undefined — rounding, sign and threshold functions, sampling a discrete value, quantization — by substituting a surrogate derivative in the backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., most often that of the identity function, while the forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. uses the real discrete operation. The resulting gradient is biased but usually points in a useful direction, letting gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. train networks with binary or low-bit weights and activations (quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding., BitNetBitNet, prop. n.: A family of AI language models from Microsoft in which most of the learned numbers are limited to minus one, zero or one, so they need far less memory and energy to run.), discrete codes such as the codebook lookup in a VQ-VAEVQ-VAE, n.: A network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate., and hard routing decisions. Variants clip the surrogate outside a range or use the derivative of a smooth approximation; continuous relaxations such as the Gumbel-softmax trade bias and variance differently. In code it is often written x + (q(x) − x).detach(), so the forward value is q(x) while the gradient is that of x.