sigmoid function
In plain wordsAn S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.
In machine learning, usually the logistic function σ(x) = 1/(1 + e^(−x)), which maps any real number into the open interval (0, 1), is symmetric about the point (0, ½) and has derivative σ(x)(1 − σ(x)), at most ¼. It is the inverse of the logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity., so it converts log-odds into probabilities: it is the link function of logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve., the output activation of binarybinary classification, n.: Sorting things into exactly two groups, such as spam versus not spam or sick versus healthy. Most yes-or-no decisions made by AI systems are of this kind. and multilabelmultilabel classification, n.: Tagging each item with any number of labels at once — a photo can show a beach, a dog and a sunset — instead of picking just one category. classifiers, and the gate nonlinearity in LSTMslong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences., GRUsgated recurrent unit, n.: A simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step. and GLUsgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through.. As a hidden-layer activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer., standard from the 1980s to the 2000s, it fell out of favor because it saturates, so gradients vanish for large |x|, and its outputs are not zero-centered. More broadly, 'sigmoid' names any S-shaped function, including tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over., the arctangent and the normal cumulative distribution function.
Apply a sigmoid to each output separately, since an image can carry several labels at once.
Origin
Pierre-François Verhulst introduced the logistic function in 1838 to model population growth and named it 'logistique' in the mid-1840s. Raymond Pearl and Lowell Reed revived it for population studies in 1920, Joseph Berkson's bioassay work (1944) made it central to statistics, and backpropagation in the 1980s made it the standard neural-network activation.
Usage note
Not to be confused with softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score., its multiclass generalization: a two-class softmax equals a sigmoid of the difference of the two logits.
Related
- logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.
- logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
- tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over.
- softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
- activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer.
- vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
- softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
- ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.
Mentioned in
activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer., linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly., logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve., logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity., long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences., multilabel classificationmultilabel classification, n.: Tagging each item with any number of labels at once — a photo can show a beach, a dog and a sunset — instead of picking just one category., ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective., SiLUSiLU, abbr.: A smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models., softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score., tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over., universal approximation theoremuniversal approximation theorem, n.: A mathematical result showing that a neural network with enough neurons can imitate essentially any smooth input-to-output relationship as closely as you like, though it does not say how to find that network., vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn., Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths., gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through., highway networkhighway network, n.: An early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable., radial basis function networkradial basis function network, n.: A three-layer network whose hidden neurons each respond to inputs near a particular reference point, like bells placed across the data; the output is a weighted mix of the bells., squeeze-and-excitation networksqueeze-and-excitation network, n.: An image-network design whose add-on blocks let the network decide, for each picture, which of its internal feature channels to turn up and which to turn down.