neural Turing machine

A recurrentrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. controller coupled to a separate memory matrix that it learns to read and write. Every memory access is soft, a weighted blend over rows rather than a hard choice, so the whole system is differentiable and the controller can learn by gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. where to look and what to store. Its read and write 'heads' locate rows by content (a softmax-normalized similarity between a key emitted by the controller and each row) and by location (shifting focus from the previous position), and write through erase and add vectors. In its first tests the machine learned to duplicate an input sequence, to sort items by priority and to retrieve the item paired with a cue, and its solutions carried over to longer sequences than an LSTMlong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.'s did. Its successor, the differentiable neural computer (2016), added dynamic memory allocation and temporal links and solved graph-traversal tasks. Such memory-augmented networks developed alongside the content-based attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. later central to transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. and foreshadowed retrieval-augmented models, but are now rarely used.

The neural Turing machine learned to copy sequences twice as long as any it saw in training.