Skip to content
Last Dictionary
Plane
Inference & EfficiencyLanguage Models & Transformers

speculative decoding

nounAlso: speculative sampling, draft model, draft-and-verify decoding, assisted generation, SpecDec

In plain wordsA speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.

  1. A family of methods that accelerate autoregressiveautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. generation by drafting several future tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. cheaply and verifying them with the full (target) model in a single forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction.. A small draft model, extra prediction heads, an n-gram lookup in the prompt or the target's own early layers proposes k tokens; the target scores all k positions in parallel; the longest acceptable prefix is kept and the target supplies one corrected or extra token, so each iteration yields at least one token. With the acceptance rule shown (p the target's distribution, q the draft's), the output distribution is exactly the target model's, and outputs are identical under greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.. Because the decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far. is memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries., checking k tokens costs little more than generating one, giving typical speedups of two to three times; gains depend on the acceptance rate and shrink at large batch sizes, where spare compute is scarce. Variants include Medusa, EAGLE and lookahead decoding.

    With a 1B draft model accepting about 70 percent of its guesses, speculative decoding doubled our output speed.

P(accept x~)=min⁡(1,p(x~)q(x~)),else resample x∼max⁡(0, p(x)−q(x))∑x′max⁡(0, p(x′)−q(x′))P(\text{accept } \tilde{x}) = \min\left(1, \frac{p(\tilde{x})}{q(\tilde{x})}\right), \quad \text{else resample } x \sim \frac{\max(0,\, p(x) - q(x))}{\sum_{x'} \max(0,\, p(x') - q(x'))}

Origin

Blockwise parallel decoding (Mitchell Stern, Noam Shazeer and Jakob Uszkoreit, 2018) anticipated draft-then-verify. The name, borrowed from speculative execution in processors, was used by Heming Xia and colleagues for translation in March 2022. Yaniv Leviathan, Matan Kalman and Yossi Matias at Google (November 2022) gave the general, distribution-preserving algorithm, and Charlie Chen and colleagues at DeepMind independently published 'speculative sampling' (February 2023).

Usage note

Speeds up generation without changing what the model would output, unlike quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. or knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., which change the model. Not the same as multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster., a training objective, though models trained with it can supply their own drafts.

Related

See also
  • decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far.
  • memory-boundmemory-bound, adj.: Describes a computer task slowed by how fast data can be fetched from memory rather than by how fast the chip can calculate — like a fast cook kept waiting for deliveries.
  • tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant.
  • multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster.
  • early exitearly exit, n.: Letting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs.
  • model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough.
Contrast with
  • quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
  • knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.

Sources

  1. Leviathan, Y., Kalman, M. & Matias, Y. (2022). Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192 (ICML 2023). arxiv.org/abs/2211.17192
  2. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L. & Jumper, J. (2023). Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318. arxiv.org/abs/2302.01318
  3. Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F. & Sui, Z. (2022). Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation. arXiv:2203.16487 (Findings of EMNLP 2023). arxiv.org/abs/2203.16487

Mentioned in

inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text., autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back., decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones., inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers., llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models., memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., early exitearly exit, n.: Letting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs., Jakob UszkoreitJakob Uszkoreit, prop. n.: A German computer scientist, one of the eight authors of the transformer paper, credited with first proposing to build language models on attention alone., model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough., multi-token predictionmulti-token prediction, n.: Training a language model to guess several upcoming words at once rather than only the next one, which can help it learn better and later write faster., SGLangSGLang, prop. n.: A free, open-source engine for running AI language and image models on servers, built to be fast when many requests begin the same way, as in chat and agent workloads., TensorRT-LLMTensorRT-LLM, prop. n.: NVIDIA's free software library for running large language models as fast as possible on NVIDIA's own graphics chips, used by many companies that host AI models.