model cascade
In plain wordsA chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough.
An arrangement in which inputs pass through a sequence of models of increasing cost and capability, each stage either returning a result or deferring to the next according to a confidence score, an agreement check or a learned scorer. Because most inputs are easy, the expensive stages run on only a small fraction of them, cutting average cost and latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system. while approaching the accuracy of the strongest model. Stage thresholds trade accuracy against cost and must be calibratedcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.. In LLM applications a cascade might send a query to a small model, check the answer with a scoring model or by self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts., and escalate to a frontier model when confidence is low. Early exitearly exit, n.: Letting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs. applies the same idea inside one network, and speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality. is a related draft-then-check pattern at the level of tokens.
Our cascade answers 70 percent of queries with the small model and pays frontier prices only for the rest.
Origin
An early example is Ethem Alpaydın and Cenk Kaynak's 'Cascading classifiers' (1998), in which a cheap rule-learner passes the inputs it rejects to a costlier nearest-neighbor stage. Paul Viola and Michael Jones's boosted cascade of simple features for real-time face detection (CVPR 2001) made cascades a standard design in computer vision. For language models, FrugalGPT (Lingjiao Chen, Matei Zaharia and James Zou, Stanford, 2023) reported matching the best single model's accuracy at a small fraction of its cost with an LLM cascade.
Related
- model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality.
- early exitearly exit, n.: Letting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs.
- speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality.
- inference costinference cost, n.: What it costs to use an AI model after it has been built: the computing power, electricity and hardware needed every time it answers, usually billed per chunk of text.
- calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
- ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
- model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality.
- ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
Sources
- Viola, P. & Jones, M. (2001). Rapid Object Detection using a Boosted Cascade of Simple Features. CVPR 2001.
- Chen, L., Zaharia, M. & Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176.
Mentioned in
model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., early exitearly exit, n.: Letting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs.