A/B testing
In plain wordsComparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better.
A randomized controlled experiment in which users, sessions or requests are randomly assigned to a control (A) and one or more variants (B, C …) and predefined metrics are compared with statistical tests to estimate the causal effect of a change. In AI products it is the standard final check of a new model, prompt, retrieval setup or ranking algorithm after offline evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews., because offline metrics often fail to predict user behavior. Sound practice fixes the primary metric, minimum detectable effect and sample size in advance; watches guardrail metrics such as latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., cost and complaint rates; avoids repeatedly peeking at results unless sequential methods are used; and checks for sample-ratio mismatch and novelty effects. Related designs include interleaving, which mixes two rankers' results in one list, multi-armed banditsmulti-armed bandit, n.: A puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others. that shift traffic toward the better variant during the test, and champion–challenger setups in credit and risk modeling.
The new reranker won offline but lost the A/B test on click-through, so we kept the old one.
Origin
Randomized comparison descends from R. A. Fisher's agricultural experiments of the 1920s and from split-run tests in direct-mail advertising. Web companies made it routine around 2000 — Google ran its first such test that year — and Ron Kohavi and colleagues codified online controlled experiments in papers from 2007 onward and a 2020 textbook.
Usage note
An A/B test measures what users do, not whether outputs are correct or safe, so it complements offline evaluation rather than replacing it. Small effects need large samples, and many wins from underpowered tests fail to replicate.
Related
- shadow deploymentshadow deployment, n.: Quietly running a new AI model alongside the old one on real requests, recording its answers without showing them to anyone, to check it behaves well before it takes over.
- canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone.
- evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.
- multi-armed banditmulti-armed bandit, n.: A puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others.
- hypothesis testinghypothesis testing, n.: A formal procedure for deciding whether data show a real effect or could easily be explained by chance — for instance, whether one AI model is genuinely better than another.
- model deploymentmodel deployment, n.: Putting a trained AI model to work in a real product so that people or other software can use it — and doing so carefully enough that the change can be undone if something goes wrong.
Sources
- Kohavi, R., Tang, D. & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
Mentioned in
evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews., recommender systemrecommender system, n.: Software that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on., causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time., data sciencedata science, n.: The practice of collecting, cleaning, analyzing and interpreting data, using statistics, programming and machine learning, to answer questions and guide decisions., hypothesis testinghypothesis testing, n.: A formal procedure for deciding whether data show a real effect or could easily be explained by chance — for instance, whether one AI model is genuinely better than another., model deploymentmodel deployment, n.: Putting a trained AI model to work in a real product so that people or other software can use it — and doing so carefully enough that the change can be undone if something goes wrong., multi-armed banditmulti-armed bandit, n.: A puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others., p-valuep-value, n.: A number saying how surprising your data would be if nothing interesting were going on. A small p-value hints at a real effect, but it is not the probability that the effect is real., staged deploymentstaged deployment, n.: Releasing a new AI system step by step — first to testers, then to limited groups, then widely — so that problems can be spotted and fixed before everyone has it., canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone., shadow deploymentshadow deployment, n.: Quietly running a new AI model alongside the old one on real requests, recording its answers without showing them to anyone, to check it behaves well before it takes over., Simpson's paradoxSimpson's paradox, n.: A statistics trap in which combining groups reverses what each group shows: a treatment can do better than another for men and for women separately, yet worse when everyone is counted together., uplift modelinguplift modeling, n.: Predicting not who will buy, recover or respond, but whose behavior will actually change because of an action such as a discount or a treatment, so that it is offered only where it makes a difference.