matrix
In plain wordsA rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this.
A rectangular array of numbers with m rows and n columns (an m × n matrix), which represents a linear transformationlinear transformation, n.: A rule for moving every point in a space that keeps straight lines straight and the origin in place: rotating, stretching, reflecting or squashing everything evenly. Every grid of numbers describes one. from n-dimensional to m-dimensional space once bases are chosen. Matrices can be added, scaled, transposed and multiplied; a square matrix has a determinant and eigenvalues and eigenvectorseigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies. and may have an inverse, and any matrix's rankrank, n.: The number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks. says how many independent directions it preserves. In neural networks the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. of each fully connected layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage. form a matrix, and computation is dominated by matrix multiplicationsmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing. between weight matrices and batches of activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., which is why GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. and TPUsTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud. are built around fast matrix arithmetic. Special structures — diagonal, symmetric, orthogonal, positive-definite, sparse or low-ranklow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation. — allow cheaper or more stable computation.
The query projection is a 4,096 × 4,096 matrix, about 16.8 million parameters on its own.
Origin
James Joseph Sylvester coined 'matrix' (Latin for 'womb') in 1850 for a rectangular array from which determinants could be formed. Arthur Cayley treated matrices as algebraic objects in their own right, defining their product and inverse, in 'A Memoir on the Theory of Matrices' (1858). Arrays of coefficients for solving linear equations are far older, appearing in China's Nine Chapters on the Mathematical Art.
Related
- matrix multiplicationmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing.
- linear transformationlinear transformation, n.: A rule for moving every point in a space that keeps straight lines straight and the origin in place: rotating, stretching, reflecting or squashing everything evenly. Every grid of numbers describes one.
- tensortensor, n.: A block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way.
- vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points.
- eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies.
- singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.
- rankrank, n.: The number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks.
- linear algebralinear algebra, n.: The branch of mathematics about lists and grids of numbers and the even, straight-line transformations between them. It is the everyday language in which neural networks are written and computed.
Sources
- Cayley, A. (1858). A Memoir on the Theory of Matrices. Philosophical Transactions of the Royal Society of London, 148.
Mentioned in
vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points., eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies., linear algebralinear algebra, n.: The branch of mathematics about lists and grids of numbers and the even, straight-line transformations between them. It is the everyday language in which neural networks are written and computed., linear transformationlinear transformation, n.: A rule for moving every point in a space that keeps straight lines straight and the origin in place: rotating, stretching, reflecting or squashing everything evenly. Every grid of numbers describes one., low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation., Markov chainMarkov chain, n.: A chance-driven process that moves between states without memory: the odds of its next move are set by where it is, never by the path that led there. Forecasting tomorrow's weather from today's alone is a simple example., matrix multiplicationmatrix multiplication, n.: The rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing., rankrank, n.: The number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks., scalarscalar, n.: A single number, such as 3.7 or −2, as opposed to a list or grid of numbers. A model's error score and its learning rate are both scalars., singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data., tensortensor, n.: A block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way., HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass., JacobianJacobian, n.: A table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space.