gated recurrent unit
In plain wordsA simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step.
A recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. unit that streamlines the LSTMlong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences. design to one state vector and two gates. The reset gate decides how much of the previous state feeds into a freshly computed candidate state; the update gate z_t then blends the previous state with that candidate, and this weighted average supplies the additive path that eases the vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.. Folding the LSTM's cell state into the hidden state and doing without its output gate leaves three weight blocks instead of four, about a quarter fewer parameters. Head-to-head tests, starting with Junyoung Chung and colleagues (2014), have generally found little difference in accuracy, with the better unit varying by task. Conventions differ on whether z_t weights the old state or the new one.
We used a GRU instead of an LSTM because it trained faster with no loss in accuracy.
Origin
Introduced as a new hidden unit by Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio in 'Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation' (arXiv, June 2014; EMNLP 2014). The name 'gated recurrent unit' spread through Chung and colleagues' empirical evaluation of December 2014.
Related
- long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.
- recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
- vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
- sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step.
- encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.
Sources
- Cho, K. et al. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. EMNLP 2014; arXiv:1406.1078. arxiv.org/abs/1406.1078
- Chung, J., Gulcehre, C., Cho, K. & Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv:1412.3555.
Mentioned in
long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences., recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., sequence-to-sequence modelsequence-to-sequence model, n.: A model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step., sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on., tanhtanh, n.: An S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over., vanishing gradient problemvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn., gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through., Kyunghyun ChoKyunghyun Cho, prop. n.: A South Korean computer scientist who co-invented the encoder–decoder approach to neural machine translation and the gated recurrent unit, and co-wrote the paper that introduced attention.