early exit
In plain wordsLetting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs.
A form of conditional or adaptive computation in which a neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. can produce its output from an intermediate layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage. instead of the last. Lightweight exit classifiers or a shared output head are attached at several depths; at inference time computation stops at the first exit whose confidence (for example a high maximum softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. probability or low entropy) passes a threshold, so easy inputs use fewer layers. In language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. early exit can operate token by token, but skipped layers leave no keys and values in the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. for later tokens, which methods must copy, recompute or approximate. Self-speculative schemes use the early layers as a draft model and the full network as verifier, so output quality is preserved. Exits must be trained jointly or added afterwards, and their thresholds calibratedcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time..
With early exit, simple sentiment queries leave at layer 4 of 12 and latency halves.
Origin
BranchyNet (Surat Teerapittayanon, Bradley McDanel and H. T. Kung, Harvard, 2016) popularized side-branch exits for image classifiers, and Alex Graves's adaptive computation time (2016) learned how many steps to take. Language-model work followed with the depth-adaptive Transformer (2020), Google's CALM (2022) and Meta's LayerSkip (2024).
Usage note
Not to be confused with early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples., which ends training when validation performance stops improving.
Related
- model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough.
- speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well.
- early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples.
Sources
- Teerapittayanon, S., McDanel, B. & Kung, H. T. (2016). BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks. ICPR 2016.
- Schuster, T. et al. (2022). Confident Adaptive Language Modeling. NeurIPS 2022.
Mentioned in
speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough., overthinkingoverthinking, n.: When an AI reasoning model spends far more thinking than a problem needs — pages of deliberation for a simple sum — wasting time and money and sometimes talking itself out of a right answer.