baseline

Evaluation

A reference method, usually simple or well established, against which a new model's performance is compared to judge whether it adds value. Trivial baselines set the floor: predicting the majority class or the mean, random guessing, or, in forecasting, the last value or the same period last season. Standard baselines are established methods, properly tuned: logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve., gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it., BM25BM25, prop. n.: A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents. for retrieval, or the previous state of the artstate of the art, n.: The best performance or most advanced technique currently known in a field. In AI papers, a state-of-the-art result is one that beats every earlier published result on a test.. A fair comparison gives baselines the same data, features and tuning budget as the proposed method. Weak or under-tuned baselines are a recurring source of inflated claims: careful re-evaluations in recommender systems, metric learning and other areas have found that tuned older methods match many reported advances. For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., baselines also include human performance, earlier model versions and simple prompting strategies. Reporting improvements with confidence intervalsconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample. over several runs guards against noise.

Our transformer beat a tuned gradient-boosting baseline by only 0.3 points, so we shipped the trees.

Reinforcement learning

A quantity, typically an estimate of a state's value, subtracted from the return in policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. methods to reduce the variance of the gradient estimate without biasing it; the difference estimates the advantageadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse.. REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly. with a learned baseline and actor–critic methods depend on it, and group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average. uses the mean reward of a group of sampled answers as its baseline.

Interpretability

In feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down. methods such as integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way. and SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players., the reference input, such as a black image or an average input, against which a prediction is compared; attributions explain the difference from the baseline and can change substantially with its choice.