Skip to content
Last Dictionary
Plane
Generative MediaDeep Learning

denoising diffusion probabilistic model

nounAlso: DDPM, DDPMs, improved DDPM

In plain wordsThe 2020 recipe that made diffusion image generators work well: add noise to pictures over a thousand small steps, then train a network to guess the noise so it can be removed.

  1. A diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. formulation with a fixed Markov forward process that adds Gaussian noise over T steps (T = 1,000 in the original) according to a variance schedule β₁ … β_T, and a learned reverse process whose Gaussian transitions are parameterized by a network ε_θ(x_t, t) that predicts the noise contained in the noisy sample x_t. Because any x_t can be drawn in closed form from the clean image x₀, training samples a random step and regresses the predicted noise onto the true noise — a simplified, reweighted form of the evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn. that the authors found gave better samples. Sampling runs the reverse chain from pure noise through all T steps, which made DDPMs slow; later work learned the reverse variances, introduced the cosine noise schedulenoise schedule, n.: The plan for how much static is mixed into training examples at each step of a diffusion model, from almost none to total noise; its shape affects quality and speed. and replaced this ancestral sampling with faster deterministic samplers such as DDIMdenoising diffusion implicit model, n.: A faster way to run a diffusion image generator that skips most of the steps and, for the same starting noise, always gives the same picture..

Lsimple=Ex0,ϵ,t[∥ϵ−ϵθ(αˉt x0+1−αˉt ϵ,  t)∥2]\mathcal{L}_{\text{simple}} = \mathbb{E}_{x_0, \epsilon, t}\Big[\big\lVert \epsilon - \epsilon_\theta\big(\sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon,\; t\big) \big\rVert^2\Big]

Origin

Introduced by Jonathan Ho, Ajay Jain and Pieter Abbeel of UC Berkeley in 'Denoising Diffusion Probabilistic Models' (posted June 2020; NeurIPS 2020), which reported an FID of 3.17 on CIFAR-10 and connected diffusion with denoising score matching. Alex Nichol and Prafulla Dhariwal's 'Improved DDPM' (February 2021) added learned variances and the cosine noise schedule.

Related

See also
  • diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
  • score-based generative modelscore-based generative model, n.: A generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step.
  • noise schedulenoise schedule, n.: The plan for how much static is mixed into training examples at each step of a diffusion model, from almost none to total noise; its shape affects quality and speed.
  • denoising diffusion implicit modeldenoising diffusion implicit model, n.: A faster way to run a diffusion image generator that skips most of the steps and, for the same starting noise, always gives the same picture.
  • evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn.
  • U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.

Sources

  1. Ho, J., Jain, A. & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239. arxiv.org/abs/2006.11239
  2. Nichol, A. & Dhariwal, P. (2021). Improved Denoising Diffusion Probabilistic Models. ICML 2021. arXiv:2102.09672. arxiv.org/abs/2102.09672

Mentioned in

diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., diffusion samplerdiffusion sampler, n.: The step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality., denoising diffusion implicit modeldenoising diffusion implicit model, n.: A faster way to run a diffusion image generator that skips most of the steps and, for the same starting noise, always gives the same picture., evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn., Jascha Sohl-DicksteinJascha Sohl-Dickstein, prop. n.: An American scientist who in 2015, with colleagues, introduced diffusion models, the technique behind most modern AI image and video generators, by borrowing ideas from physics., noise schedulenoise schedule, n.: The plan for how much static is mixed into training examples at each step of a diffusion model, from almost none to total noise; its shape affects quality and speed., Pieter AbbeelPieter Abbeel, prop. n.: A Belgian-American roboticist at Berkeley whose research taught robots to learn from demonstrations and from trial and error, and who co-founded the robotics company Covariant., score-based generative modelscore-based generative model, n.: A generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step.