Skip to content
Last Dictionary
Plane
Mathematics & StatisticsSecurity & MisuseGenerative Media

Lipschitz continuity

nounAlso: Lipschitz, Lipschitz constant, Lipschitz condition, K-Lipschitz, L-smoothness

In plain wordsA guarantee that a function never changes too abruptly: its output can move at most a fixed multiple of how far its input moves.

  1. The property of a function f for which some constant K bounds the distance between f(x) and f(y) by K times the distance between x and y, for all x and y; the smallest such K is the Lipschitz constant, a global limit on steepness. For differentiable functions it equals the largest norm of the derivative — for vector functions, the largest singular value of the JacobianJacobian, n.: A table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space.. It matters in machine learning in several ways: gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. provably converges with a learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. below 2/L when the gradient is L-Lipschitz (L-smoothness); the critic of a Wasserstein GAN must be 1-Lipschitz to estimate the Wasserstein distanceWasserstein distance, n.: A way of comparing two spreads of probability by imagining one as heaps of sand that must be shoveled into the shape of the other: the less shoveling, weighted by distance, the more alike they are., enforced by weight clipping, gradient penalties or spectral normalization; and small constants yield certified robustnesscertified robustness, n.: A mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer. against adversarial perturbationsadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.. A network's constant is bounded by the product of its layers' constants, a bound that is often very loose.

    Spectral normalization keeps each layer 1-Lipschitz, so the whole discriminator is too.

∥f(x)−f(y)∥≤K ∥x−y∥for all x,y\lVert f(x) - f(y)\rVert \le K\,\lVert x - y\rVert \quad \text{for all } x, y

Origin

Named after the German mathematician Rudolf Lipschitz, who used the condition in the 1860s and 1870s in work on Fourier series and on the existence and uniqueness of solutions to differential equations. It rose to prominence in deep learning with the Wasserstein GAN (Martin Arjovsky, Soumith Chintala and Léon Bottou, 2017) and spectral normalization (Takeru Miyato and colleagues, 2018).

Related

See also
  • Wasserstein distanceWasserstein distance, n.: A way of comparing two spreads of probability by imagining one as heaps of sand that must be shoveled into the shape of the other: the less shoveling, weighted by distance, the more alike they are.
  • certified robustnesscertified robustness, n.: A mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer.
  • adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
  • JacobianJacobian, n.: A table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space.
  • generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.
  • gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.

Sources

  1. Arjovsky, M., Chintala, S. & Bottou, L. (2017). Wasserstein GAN. arXiv:1701.07875. arxiv.org/abs/1701.07875

Mentioned in

Wasserstein distanceWasserstein distance, n.: A way of comparing two spreads of probability by imagining one as heaps of sand that must be shoveled into the shape of the other: the less shoveling, weighted by distance, the more alike they are.