p-value
In plain wordsA number saying how surprising your data would be if nothing interesting were going on. A small p-value hints at a real effect, but it is not the probability that the effect is real.
A number reported by a statistical hypothesis testhypothesis testing, n.: A formal procedure for deciding whether data show a real effect or could easily be explained by chance — for instance, whether one AI model is genuinely better than another. that gauges how surprising the data are if the null hypothesis holds: in a world where the null is true, it is the chance that the test statistic would be at least as extreme as the value actually obtained. Smaller values are stronger evidence against the null; by a widespread convention, p < 0.05 is called 'statistically significant'. As the American Statistical Association stressed in a 2016 statement, it is neither the probability that the null hypothesis is true nor the probability that chance alone produced the result, and it does not measure how large or important an effect is. In machine learning, p-values appear when comparing models or prompts on benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. — through paired, bootstrap or permutation tests — and in A/B testingA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better.. Common pitfalls include trying many model variants, seeds or metrics and reporting only the significant ones (the multiple-comparisons problem, or 'p-hacking'), and treating tiny but significant differences on huge test sets as meaningful.
The new prompt scored 1.2 points higher, but a paired bootstrap gave p = 0.31, so we can't claim an improvement.
Origin
Karl Pearson's 1900 chi-squared paper computed and tabulated 'the value of P', and Ronald Fisher's Statistical Methods for Research Workers (1925) made P values and the 0.05 benchmark central to scientific practice. The term 'p-value' itself is recorded from the 1940s, for example in W. Edwards Deming's Statistical Adjustment of Data (1943).
Usage note
In 2016 the American Statistical Association warned against bright-line thresholds and common misreadings; some statisticians favor confidence intervalsconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample., effect sizes or BayesianBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability. methods instead. Unrelated to the p of top-p samplingtop-p sampling, n.: A way of making AI-written text varied but sensible: the model picks at random among the smallest group of likely next words whose chances add up to a set share, such as 90%..
Related
- hypothesis testinghypothesis testing, n.: A formal procedure for deciding whether data show a real effect or could easily be explained by chance — for instance, whether one AI model is genuinely better than another.
- confidence intervalconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample.
- A/B testingA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
- central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like.
Sources
- Wasserstein, R. L. & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2).
Mentioned in
false positivefalse positive, n.: A false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there., confidence intervalconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample., hypothesis testinghypothesis testing, n.: A formal procedure for deciding whether data show a real effect or could easily be explained by chance — for instance, whether one AI model is genuinely better than another., hypothesis spacehypothesis space, n.: The full menu of possible models a learning method is allowed to choose from — for example, all straight lines. Learning means picking the item on the menu that best fits the data.