Skip to content
Last Dictionary
Plane
Machine LearningMathematics & Statistics

naive Bayes

nounAlso: naive Bayes classifier, simple Bayes, idiot's Bayes, independence Bayes, multinomial naive Bayes, Gaussian naive Bayes, Bayesian spam filter

In plain wordsA simple, fast way of sorting items into categories using probabilities, which pretends that each clue is independent of the others. It is the classic technique behind early email spam filters.

  1. A simple probabilistic classification method that scores each class with Bayes' theoremBayes' theorem, n.: A formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed. after assuming that, within each class, the features are independent of one another — the 'naive' assumption — so the class-conditional likelihood factorizes into a product of one-feature terms. Each term is estimated separately — from word counts in the multinomial and Bernoulli variants used for text, from per-class means and variances in Gaussian naive Bayes — with additive (Laplace) smoothing to avoid zero probabilities, and the predicted class is the one with the highest posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces., computed in log space. Training is a single counting pass, and the model copes with many features and little data. The independence assumption is almost always false and the resulting probabilities are poorly calibrated, usually overconfident, yet classifications are often good because only the ranking of classes matters. It is the textbook generative counterpart of logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve..

    A naive Bayes baseline on bag-of-words features took two seconds to train and set the bar at 86% accuracy.

y^=arg⁡max⁡y  P(y)∏j=1dP(xj∣y)\hat{y} = \arg\max_{y} \; P(y) \prod_{j=1}^{d} P(x_j \mid y)

Origin

Bayesian classification with independent features was studied in 1960s text retrieval and pattern recognition — Melvin Maron's 1961 work on automatic indexing is often cited — and in early computer-aided medical diagnosis. It became famous through spam filtering: Mehran Sahami and colleagues published a naive Bayes spam filter in 1998, and Paul Graham's essay 'A Plan for Spam' (2002) popularized Bayesian filtering.

Usage note

'Naive' refers to the independence assumption, not to Bayesian statistics. Andrew Ng and Michael Jordan (2001) showed that it can approach its best error with fewer examples than logistic regression, though its eventual error is usually higher.

Related

See also
  • Bayes' theoremBayes' theorem, n.: A formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed.
  • logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
  • generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them.
  • discriminative modeldiscriminative model, n.: A model that learns only to tell things apart — spam or not spam, cat or dog — without learning how to produce new examples of those things.
  • text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint.
  • classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled.
Contrast with
  • logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.

Sources

  1. Maron, M. E. (1961). Automatic Indexing: An Experimental Inquiry. Journal of the ACM, 8(3), 404–417.
  2. Ng, A. Y. & Jordan, M. I. (2001). On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes. Advances in Neural Information Processing Systems 14.

Mentioned in

classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., Bayes' theoremBayes' theorem, n.: A formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed., conditional probabilityconditional probability, n.: The chance of something once you already know that something else is true — for example, the chance of rain given that the sky is cloudy., discriminative modeldiscriminative model, n.: A model that learns only to tell things apart — spam or not spam, cat or dog — without learning how to produce new examples of those things., generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them., inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases., linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few., logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve., multiclass classificationmulticlass classification, n.: Sorting each item into exactly one of three or more categories — for example, deciding which of the ten digits a handwritten number shows., parametric modelparametric model, n.: A model whose shape is fixed in advance by a set number of adjustable settings, like a straight line's slope and intercept. More data refines those settings but never adds new ones., text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint.