Skip to content
Last Dictionary
Plane
Machine LearningMathematics & StatisticsHistory & Landmarks

support vector machine

nounAlso: SVM, support-vector network, support vector classifier, SVC, support vector regression, SVR, support vectors, soft margin, one-class SVM

In plain wordsA classic method that separates two groups of examples with the widest possible gap — like laying the broadest possible road between them. A mathematical shortcut lets it draw curved boundaries too.

  1. A supervised model that, for binary classificationbinary classification, n.: Sorting things into exactly two groups, such as spam versus not spam or sick versus healthy. Most yes-or-no decisions made by AI systems are of this kind., finds the hyperplane separating the classes with the largest marginmargin, n.: How far, and on which side, an example lies from a classifier's dividing line. A wide gap between the line and the nearest examples suggests the classifier will handle new data well. — the greatest distance to the nearest training points, called support vectors, which alone determine the solution. The soft-margin version lets some points violate the margin, trading width against training errors through a penalty parameter C; this is equivalent to minimizing hinge losshinge loss, n.: A way of scoring a classifier's mistakes that charges nothing for answers that are right with room to spare, and charges more and more the further an answer falls on the wrong side. plus an L2 regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. term. Training is a convex quadratic program, solved in practice by decomposition methods such as sequential minimal optimization (SMO) in libraries like LIBSVM. Because the dual problem uses the data only through inner products, the kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space. yields nonlinear decision boundariesdecision boundary, n.: The dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves.. Variants handle regression (with an ε-insensitive loss), multiple classes (via one-vs-restone-vs-rest, n.: A way to handle many categories with a method that only gives yes-or-no answers: train one detector per category ('this one or not?') and pick the category whose detector is most confident. or one-vs-one) and novelty detection (one-class SVM). SVMs led many benchmarks from the late 1990s until deep learning and remain effective on small or high-dimensional data such as text.

    On 3,000 labeled documents, a linear SVM on word counts beat our fine-tuned small transformer.

min⁡w,b  12∥w∥2+C∑i=1nmax⁡(0, 1−yi(w⊤xi+b))\min_{\mathbf{w}, b} \; \frac{1}{2}\lVert \mathbf{w} \rVert^2 + C \sum_{i=1}^{n} \max\big(0,\, 1 - y_i(\mathbf{w}^{\top}\mathbf{x}_i + b)\big)

Origin

Vladimir Vapnik developed the maximum-margin 'generalized portrait' method with Alexander Lerner and Alexey Chervonenkis in the Soviet Union in 1963–1964. Bernhard Boser, Isabelle Guyon and Vapnik added kernels at COLT 1992, and Corinna Cortes and Vapnik's 'Support-Vector Networks' (Machine Learning, 1995) introduced the soft margin and the 'support vector' name; 'support vector machine' was standard by the late 1990s. John Platt's SMO algorithm (1998) made training fast.

Usage note

SVM outputs are not probabilities; Platt scaling or isotonic regression is used to calibratecalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time. them. Kernel SVM training scales roughly quadratically to cubically with the number of examples, so very large datasets call for linear SVMs or other methods.

Related

See also
  • kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space.
  • marginmargin, n.: How far, and on which side, an example lies from a classifier's dividing line. A wide gap between the line and the nearest examples suggests the classifier will handle new data well.
  • hinge losshinge loss, n.: A way of scoring a classifier's mistakes that charges nothing for answers that are right with room to spare, and charges more and more the further an answer falls on the wrong side.
  • decision boundarydecision boundary, n.: The dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves.
  • linear separabilitylinear separability, n.: When two groups of points can be perfectly divided by a single straight line or, in more dimensions, a flat surface. Many simple learning methods only fully succeed in this case.
  • one-vs-restone-vs-rest, n.: A way to handle many categories with a method that only gives yes-or-no answers: train one detector per category ('this one or not?') and pick the category whose detector is most confident.
  • Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them.
Contrast with
  • logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.

Sources

  1. Cortes, C. & Vapnik, V. (1995). Support-Vector Networks. Machine Learning, 20(3), 273–297.
  2. Boser, B. E., Guyon, I. M. & Vapnik, V. N. (1992). A Training Algorithm for Optimal Margin Classifiers. Proceedings of the Fifth Annual Workshop on Computational Learning Theory (COLT '92), 144–152.

Mentioned in

classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., classifierclassifier, n.: A program or model that sorts inputs into categories, for example flagging an email as spam or a chatbot message as harmful. Many AI safety filters are classifiers., anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network., binary classificationbinary classification, n.: Sorting things into exactly two groups, such as spam versus not spam or sick versus healthy. Most yes-or-no decisions made by AI systems are of this kind., convex optimizationconvex optimization, n.: Finding the lowest point of a bowl-shaped landscape. Because a bowl has only one bottom, such problems can be solved reliably — unlike training a neural network, whose landscape is bumpy., decision boundarydecision boundary, n.: The dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves., discriminative modeldiscriminative model, n.: A model that learns only to tell things apart — spam or not spam, cat or dog — without learning how to produce new examples of those things., feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units., hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., kernelkernel, n.: A word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU., kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space., linear separabilitylinear separability, n.: When two groups of points can be perfectly divided by a single straight line or, in more dimensions, a flat surface. Many simple learning methods only fully succeed in this case., marginmargin, n.: How far, and on which side, an example lies from a classifier's dividing line. A wide gap between the line and the nearest examples suggests the classifier will handle new data well., multiclass classificationmulticlass classification, n.: Sorting each item into exactly one of three or more categories — for example, deciding which of the ten digits a handwritten number shows., nonparametric modelnonparametric model, n.: A model that lets the data decide how complex it becomes: the more examples it gets, the more detail it can capture — for instance, one that predicts by comparing a new case with every stored example., one-vs-restone-vs-rest, n.: A way to handle many categories with a method that only gives yes-or-no answers: train one detector per category ('this one or not?') and pick the category whose detector is most confident., perceptronperceptron, n.: One of the first learning machines: a single artificial neuron that sorts inputs into two groups by drawing a straight dividing line, nudging its weights whenever it guesses wrong., R-CNNR-CNN, prop. n.: A family of object detectors that first pick out image regions likely to contain objects, then use a neural network to label each region and tighten its box., scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way., supervised learningsupervised learning, n.: Teaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs., text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint., Vladimir VapnikVladimir Vapnik, prop. n.: A Soviet-born American mathematician who built the statistical theory of how machines learn from examples and co-invented support vector machines, a leading method before deep learning., Bell LabsBell Labs, prop. n.: A famous American industrial research laboratory, long owned by the telephone company AT&T, where the transistor, information theory and key early machine-learning methods were invented., hinge losshinge loss, n.: A way of scoring a classifier's mistakes that charges nothing for answers that are right with room to spare, and charges more and more the further an answer falls on the wrong side.