architecture

Machine learning

The structural design of a model: the types of layerslayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage. or components, their sizes, how they connect and how information flows through them, together with choices such as activation functionsactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer. and normalization. Architecture fixes the family of functions a model can represent and its inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases.; trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better. then selects one member of that family by setting the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself.. Landmark architectures include the multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model., convolutional networksconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects., LSTMslong short-term memory, n.: A type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences., ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI., the transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. and variants such as mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network., and alternatives such as state space modelsstate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer.. Designing architectures automatically is neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them.. Since the late 2010s much progress has come from scaling a few architectures with more data and compute rather than from new designs, a pattern summarized in the bitter lessonthe bitter lesson, phr.: The argument, made by AI pioneer Rich Sutton in 2019, that AI has advanced mostly through general methods that use ever more computing power, not by programming in human knowledge..

Both models share the same architecture; the difference lies entirely in data and training.

Systems and hardware

The organization of a larger computing system: in software, how models, retrieval, tools and interfaces fit together in an AI application; in hardware, the design of a processor or accelerator, as in successive GPU architectures or the systolic arraysystolic array, n.: A grid of tiny calculators that pass numbers to their neighbors in rhythm, like a heartbeat, so each value is reused many times. AI chips use it to multiply large tables of numbers efficiently. of a TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud.. In cognitive science and symbolic AI, a cognitive architecturecognitive architecture, n.: A blueprint for a complete thinking system — how memory, attention, learning and decision-making fit together — used both to build general AI agents and to model how the human mind works. is a fixed framework for modeling the mind, such as Soar or ACT-R.