hypothesis testing

A statistical procedure for deciding between a null hypothesis (typically 'no effect' or 'no difference') and an alternative, based on how unlikely the observed data would be if the null were true. A test statistic computed from the data is compared with its distribution under the null, giving a p-valuep-value, n.: A number saying how surprising your data would be if nothing interesting were going on. A small p-value hints at a real effect, but it is not the probability that the effect is real.; the null is rejected when p falls below a significance level α fixed in advance, conventionally 0.05. Rejecting a true null is a type I error, with rate α; failing to reject a false one is a type II error, and the chance of avoiding it is the test's power, which grows with sample size and effect size. Common tests include t-tests, chi-squared tests, McNemar's test for paired classifier errors, and permutation and bootstrap tests. Comparing models on benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., A/B testingA/B testing, n.: Comparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better. and uplift studiesuplift study, n.: An experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet. all rely on hypothesis testing, and small evaluation sets often lack the power to detect real differences.

With only 200 test items, the hypothesis test had little power to detect a two-point gain.