Skip to content
Last Dictionary
Plane
Deep LearningMathematics & StatisticsMachine Learning

neural tangent kernel

nounAlso: NTK, lazy training, kernel regime, infinite-width limit

In plain wordsA mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.

  1. The kernel formed by the inner product of the gradients of a network's output with respect to its parameters at two inputs; it governs how a gradient step on one example changes the prediction on another, and under gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. with small steps the network's function evolves by kernel gradient descent with this kernel. In the infinite-width limit under the so-called NTK parametrization, the kernel becomes deterministic at initialization and stays constant during training, so the network behaves like its first-order Taylor expansion — a model linear in its parameters — and training with squared loss is equivalent to kernel regression (see kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space.), with provable convergence to zero training loss. This 'lazy' regime made deep-network training analyzable, but in it internal representations barely change, so it cannot explain feature learning; real finite-width networks depart from it, and alternative scalings such as maximal update parametrizationmaximal update parametrization, n.: A recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one. preserve feature learning in the infinite-width limit.

Θ(x,x′)=∇θfθ(x)⊤ ∇θfθ(x′)\Theta(x, x') = \nabla_{\theta} f_{\theta}(x)^{\top} \, \nabla_{\theta} f_{\theta}(x')

Origin

Introduced by Arthur Jacot, Franck Gabriel and Clément Hongler of EPFL in 'Neural Tangent Kernel: Convergence and Generalization in Neural Networks' (arXiv June 2018; NeurIPS 2018). Lénaïc Chizat, Edouard Oyallon and Francis Bach named the 'lazy training' regime (2018), and Greg Yang and Edward Hu's 2020 feature-learning limit offered an alternative.

Usage note

'Kernel' here is the machine-learning sense of a similarity function, not a GPU or operating-system kernel; see kernelkernel, n.: A word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU..

Related

See also
  • overparameterizationoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps.
  • kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space.
  • Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them.
  • maximal update parametrizationmaximal update parametrization, n.: A recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one.
  • implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions.
  • weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers.

Sources

  1. Jacot, A., Gabriel, F. & Hongler, C. (2018). Neural Tangent Kernel: Convergence and Generalization in Neural Networks. NeurIPS 2018. arxiv.org/abs/1806.07572
  2. Chizat, L., Oyallon, E. & Bach, F. (2019). On Lazy Training in Differentiable Programming. NeurIPS 2019.

Mentioned in

central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like., convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them., kernelkernel, n.: A word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU., kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space., weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers., maximal update parametrizationmaximal update parametrization, n.: A recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one., overparameterizationoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps., statistical learning theorystatistical learning theory, n.: The mathematics of when and why learning from examples works: how much data is needed, and how well a model trained on some examples will do on new ones., Taylor seriesTaylor series, n.: A way of approximating a complicated curve near a chosen point by adding up simple pieces: a constant, a straight-line slope, a bend, and so on.