Skip to content
Last Dictionary
Plane
Generative MediaDeep LearningMathematics & Statistics

score-based generative model

nounAlso: score-based model, score-based diffusion, noise-conditional score network, NCSN, score matching, probability flow ODE

In plain wordsA generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step.

  1. A generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them. that learns the score — the gradient of the log probability density, ∇ₓ log p(x) — of the data distribution after it has been perturbed with noise at many scales, and generates samples by following the learned scores from noise toward data. The score network is trained by denoising score matching, which amounts to predicting the added noise. Early models sampled with annealed Langevin dynamics, moving from large to small noise levels. The stochastic-differential-equation (SDE) framework then showed that score models and DDPMsdenoising diffusion probabilistic model, n.: The 2020 recipe that made diffusion image generators work well: add noise to pictures over a thousand small steps, then train a network to guess the noise so it can be removed. are discretizations of two continuous-time processes of one family (variance-exploding and variance-preserving SDEs, respectively): in each, a forward SDE turns data into noise, a reverse-time SDE that depends only on the score turns noise back into data, and an equivalent deterministic 'probability flow' ordinary differential equation allows exact likelihoods and fast solver-based sampling. Score-based models and diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. are now treated as one family.

dx=[f(x,t)−g(t)2 ∇xlog⁡pt(x)] dt+g(t) dwˉ\mathrm{d}x = \big[f(x,t) - g(t)^2\, \nabla_x \log p_t(x)\big]\,\mathrm{d}t + g(t)\, \mathrm{d}\bar{w}

Origin

Yang Song and Stefano Ermon of Stanford introduced noise-conditional score networks in 'Generative Modeling by Estimating Gradients of the Data Distribution' (posted July 2019; NeurIPS 2019), building on Aapo Hyvärinen's score matching (2005) and Pascal Vincent's denoising score matching (2011). Song and co-authors unified the approach with diffusion in 'Score-Based Generative Modeling through Stochastic Differential Equations' (posted November 2020; ICLR 2021).

Related

See also
  • diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
  • denoising diffusion probabilistic modeldenoising diffusion probabilistic model, n.: The 2020 recipe that made diffusion image generators work well: add noise to pictures over a thousand small steps, then train a network to guess the noise so it can be removed.
  • diffusion samplerdiffusion sampler, n.: The step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality.
  • flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples.
  • energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys.
  • Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability.

Sources

  1. Song, Y. & Ermon, S. (2019). Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS 2019. arXiv:1907.05600. arxiv.org/abs/1907.05600
  2. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S. & Poole, B. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021. arXiv:2011.13456. arxiv.org/abs/2011.13456

Mentioned in

diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., diffusion samplerdiffusion sampler, n.: The step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality., classifier guidanceclassifier guidance, n.: A way to push a diffusion image generator toward a requested category by consulting a separate image-recognition model at every step, trading variety for images that more clearly show the target., consistency modelconsistency model, n.: A generator trained to jump from noise straight to a finished image in one step, instead of the dozens of small steps a diffusion model normally takes., denoising autoencoderdenoising autoencoder, n.: An autoencoder trained to recover clean data from a deliberately damaged copy, such as an image with random pixels erased, which teaches it the structure of the data rather than how to copy., denoising diffusion implicit modeldenoising diffusion implicit model, n.: A faster way to run a diffusion image generator that skips most of the steps and, for the same starting noise, always gives the same picture., denoising diffusion probabilistic modeldenoising diffusion probabilistic model, n.: The 2020 recipe that made diffusion image generators work well: add noise to pictures over a thousand small steps, then train a network to guess the noise so it can be removed., energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys., Jascha Sohl-DicksteinJascha Sohl-Dickstein, prop. n.: An American scientist who in 2015, with colleagues, introduced diffusion models, the technique behind most modern AI image and video generators, by borrowing ideas from physics., Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability., probability density functionprobability density function, n.: A curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.