highway network

A feedforward neural networkfeedforward neural network, n.: A neural network laid out like an assembly line: each stage hands its result only to later stages, never back, so every answer depends on the current input alone and nothing is remembered between inputs. whose layers compute y = H(x)·T(x) + x·(1 − T(x)), where H is an ordinary nonlinear transformation and T is a learned sigmoidsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on. 'transform gate', with 1 − T acting as a 'carry gate'. When T is near 0 the layer passes its input through unchanged, giving signals and gradients a direct path through depth, the same idea the LSTMlong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences. applies through time. Their authors reported that highway networks hundreds of layers deep could be trained directly by gradient descent, at depths where plain networks stalled. ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI., published seven months later, used ungated identity shortcuts (residual connectionsresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.), which proved simpler and became standard; highway layers persisted in some character-level language and speech models.

The character-aware language model passes its convolutional features through two highway layers.