linear algebra

The branch of mathematics dealing with vectorsvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points. and vector spaces, the linear mapslinear transformation, n.: A rule for moving every point in a space that keeps straight lines straight and the origin in place: rotating, stretching, reflecting or squashing everything evenly. Every grid of numbers describes one. that act on them and the matricesmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this. that represent those maps once coordinates are fixed: systems of linear equations, matrix productsmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing., determinants, rankrank, n.: The number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks., eigenvalues and eigenvectorseigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies., orthogonalityorthogonality, n.: The mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map. and decompositions such as the singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.. Numerical linear algebra adds algorithms for computing these quantities quickly and stably in floating point. It is the main mathematical substrate of deep learning: data, parameters and activations are tensorstensor, n.: A block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way.; layers are linear maps followed by nonlinearities; attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. is built from matrix products and softmaxes; and hardware, compilers and libraries (from BLAS to cuBLAS) are organized around dense linear algebra. Interpretability tools such as linear probeslinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly. and steering vectorssteering vector, n.: A direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it., and the linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction., are claims about the linear structure of activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions..

Attention is mostly matrix products, so review your linear algebra before the transformer chapter.