bidirectional recurrent neural network
In plain wordsA recurrent network that reads a sequence both forwards and backwards, so its understanding of each word or sound can draw on what comes after it as well as what came before.
A recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. made of two independent recurrent layers, one processing the sequence from start to end and the other from end to start, whose hidden states at each position are concatenated or summed. Each output therefore depends on the entire input, past and future, which helps labeling and recognition tasks such as named-entity recognitionnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date., part-of-speech tagging, phoneme recognition and handwriting recognition, but rules the model out for streaming or left-to-right generation. Bidirectional LSTMslong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences. (BiLSTMs) were the standard NLP encoder of about 2015–2018 and the basis of ELMo contextual embeddings; encoder-only transformersencoder-only transformer, n.: A transformer that reads a whole text at once, in both directions, to understand it rather than to write new text. Such models power search, sorting documents by topic and spotting names in text. such as BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context. obtain bidirectionality through unmasked self-attention instead.
A BiLSTM tagger can use the following word to decide whether 'Washington' is a person or a place.
Origin
Introduced by Mike Schuster and Kuldip Paliwal in 'Bidirectional Recurrent Neural Networks' (IEEE Transactions on Signal Processing, 1997). Alex Graves and Jürgen Schmidhuber combined the idea with LSTM in 2005 for phoneme classification.
Related
- recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
- long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.
- encoder-only transformerencoder-only transformer, n.: A transformer that reads a whole text at once, in both directions, to understand it rather than to write new text. Such models power search, sorting documents by topic and spotting names in text.
- BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.
- named-entity recognitionnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date.
- causal language modelcausal language model, n.: A language model that reads text only from left to right, predicting each word from the words before it. The models behind today's chatbots are of this kind.
Sources
- Schuster, M. & Paliwal, K. K. (1997). Bidirectional Recurrent Neural Networks. IEEE Transactions on Signal Processing, 45(11), 2673–2681.
Mentioned in
long short-term memorylong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.