eigenvector

A nonzero vector v that a square matrixmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this. A maps onto its own line through the origin, Av = λv; the scalar λ is the corresponding eigenvalue, and the eigenvalues are the roots of det(A − λI) = 0, which can be complex even when A is real. A real symmetric matrix has real eigenvalues and an orthonormal basis of eigenvectors, so it can be factored as A = QΛQᵀ (its eigendecomposition). Eigen-analysis runs through machine learning: principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores. takes the leading eigenvectors of the data's covariance matrix as the directions of greatest variance; spectral clustering uses eigenvectors of a graph Laplacian; Google's original PageRank was the leading eigenvector of a link matrix; and the eigenvalues of the HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass. describe the curvature of a loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys., distinguishing minima from saddle pointssaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass. and bounding stable learning rateslearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.. In recurrent networks, recurrent-weight eigenvalues above or below one in magnitude drive explodingexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down. and vanishing gradientsvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn..

The top Hessian eigenvalue hovered near 2/η, the signature of the edge of stability.