Skip to content
Last Dictionary
Plane
Mathematics & StatisticsDeep LearningGenerative Media

Jacobian

nounAlso: Jacobian matrix, Jacobian determinant, vector–Jacobian product, Jacobian–vector product, VJP, JVP

In plain wordsA table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space.

  1. The m × n matrixmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this. that records how each output of a function f from ℝⁿ to ℝᵐ responds to each input: entry (i, j) is the partial derivativepartial derivative, n.: How much a quantity changes when you nudge just one of its many inputs while holding all the others fixed. ∂fᵢ/∂xⱼ. It is the best linear approximation of f near a point, and by the chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer. the Jacobian of a composition is the product of the stages' Jacobians. Deep networks have enormous Jacobians that are almost never formed explicitly: reverse-mode automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically. (backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.) computes vector–Jacobian products and forward mode computes Jacobian–vector products, each at roughly the cost of a forward pass. When m = n, the absolute value of the Jacobian's determinant measures how f scales volume; normalizing flowsnormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is. are built so that its logarithm, needed by the change-of-variables formula, is cheap. Jacobian norms also appear in analyses of robustness and training stability.

    Each coupling layer's Jacobian is triangular, so its log-determinant is just a sum.

Jij=∂fi∂xj,J∈Rm×nJ_{ij} = \frac{\partial f_i}{\partial x_j}, \qquad J \in \mathbb{R}^{m\times n}

Origin

Named after Carl Gustav Jacob Jacobi, who studied functional determinants in his 1841 memoir De determinantibus functionalibus. James Joseph Sylvester coined 'Jacobian' in 1852 in the Cambridge and Dublin Mathematical Journal.

Related

See also
  • gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill.
  • HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.
  • chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer.
  • automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically.
  • normalizing flownormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is.
  • partial derivativepartial derivative, n.: How much a quantity changes when you nudge just one of its many inputs while holding all the others fixed.

Mentioned in

automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically., chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., inverse kinematicsinverse kinematics, n.: Working backward from where you want a robot's hand to be to the joint angles that will put it there, like working out how to bend your arm to touch your nose., Kalman filterKalman filter, n.: A method that keeps a running best guess of something you cannot measure exactly, such as a vehicle's position, by blending predictions from a motion model with noisy sensor readings., partial derivativepartial derivative, n.: How much a quantity changes when you nudge just one of its many inputs while holding all the others fixed., forward kinematicsforward kinematics, n.: The math that works out where a robot's hand or tool ends up, given the angles of all its joints, by chaining together the movement each joint adds., HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass., Lipschitz continuityLipschitz continuity, n.: A guarantee that a function never changes too abruptly: its output can move at most a fixed multiple of how far its input moves., normalizing flownormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is.