diffusion model
In plain wordsAn AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
A generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them. that learns to reverse a gradual noising process. During training, examples are corrupted with Gaussian noise at randomly chosen levels along a noise schedulenoise schedule, n.: The plan for how much static is mixed into training examples at each step of a diffusion model, from almost none to total noise; its shape affects quality and speed. running from nearly clean to pure noise, and a neural network — historically a U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost., now often a transformerdiffusion transformer, n.: A diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows. — learns to predict the added noise (equivalently, the clean data, or the score: the gradient of the log-density) at each level. Generation starts from random noise and applies the network repeatedly, removing a little noise per step, with a diffusion samplerdiffusion sampler, n.: The step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality. choosing the step sizes; conditioning on text, images or other signals, usually amplified by classifier-free guidanceclassifier-free guidance, n.: A setting that makes an image generator follow the prompt more strictly by comparing what it would draw with and without the prompt, then exaggerating the difference., steers the result. The training objective is a simple regression and far more stable than GANgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people. training, and samples are diverse, but generation needs tens of network evaluations unless the model is distilled. Run in the compressed latent space of an autoencoder (latent diffusionlatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run.), diffusion underlies most image, video and audio generators released since 2022, including Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it., ImagenImagen, prop. n.: Google's family of AI image generators, first shown in 2022, whose research showed that better understanding of language yields pictures that match descriptions more faithfully. and SoraSora, prop. n.: OpenAI's video-generating AI, unveiled in 2024 with startlingly realistic clips; a 2025 version added sound and a social app, but OpenAI shut the service down in 2026.; the closely related flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. models are usually counted in the same family.
Swapping the U-Net for a transformer made our diffusion model scale far more predictably.
Origin
Jascha Sohl-Dickstein and colleagues introduced 'diffusion probabilistic models', inspired by nonequilibrium thermodynamics, in March 2015 (ICML 2015). The approach became competitive through Yang Song and Stefano Ermon's score-based models (2019) and Jonathan Ho, Ajay Jain and Pieter Abbeel's DDPM (2020), beat GANs on ImageNet in 2021 (Prafulla Dhariwal and Alex Nichol), and reached the public through DALL·E 2, Imagen and Stable Diffusion in 2022.
Usage note
The 'diffusion' is the forward noising process, named by analogy with particles spreading in physics; the model learns the reverse. Diffusion language modelsdiffusion language model, n.: A language model that writes by starting from a fully blanked-out answer and filling it in over several rounds, settling many words at once instead of writing strictly left to right. apply the idea to text, usually by masking tokens instead of adding Gaussian noise.
Related
- denoising diffusion probabilistic modeldenoising diffusion probabilistic model, n.: The 2020 recipe that made diffusion image generators work well: add noise to pictures over a thousand small steps, then train a network to guess the noise so it can be removed.
- score-based generative modelscore-based generative model, n.: A generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step.
- latent diffusion modellatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run.
- noise schedulenoise schedule, n.: The plan for how much static is mixed into training examples at each step of a diffusion model, from almost none to total noise; its shape affects quality and speed.
- diffusion samplerdiffusion sampler, n.: The step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality.
- classifier-free guidanceclassifier-free guidance, n.: A setting that makes an image generator follow the prompt more strictly by comparing what it would draw with and without the prompt, then exaggerating the difference.
- flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples.
- diffusion transformerdiffusion transformer, n.: A diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows.
- generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.
- autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.
Sources
- Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N. & Ganguli, S. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015. arXiv:1503.03585. arxiv.org/abs/1503.03585
- Ho, J., Jain, A. & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239. arxiv.org/abs/2006.11239
Mentioned in
deepfakedeepfake, n.: A fake but realistic video, photo or voice recording made with AI that puts words in a real person's mouth or shows them in a scene that never happened., generative AIgenerative AI, n.: AI that produces something new on request, such as an essay, a picture, a song or a piece of code, having absorbed from enormous numbers of existing examples what such things typically look like., text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds., AlphaFoldAlphaFold, prop. n.: A DeepMind AI system that predicts the 3D shape of proteins from their chemical sequence, often about as accurately as laboratory experiments. It cracked a problem biologists had worked on for 50 years and won a Nobel Prize., classifier-free guidanceclassifier-free guidance, n.: A setting that makes an image generator follow the prompt more strictly by comparing what it would draw with and without the prompt, then exaggerating the difference., ControlNetControlNet, prop. n.: An add-on for image generators that makes them follow a guide image — a sketch, a stick-figure pose or a depth map — so the layout comes out as intended., cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt., DALL·EDALL·E, prop. n.: OpenAI's pioneering series of image generators, named after the painter Salvador Dalí and the robot WALL·E, which showed in 2021–2023 that AI could draw almost anything described in words., density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare., diffusion samplerdiffusion sampler, n.: The step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality., diffusion transformerdiffusion transformer, n.: A diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows., flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples., generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people., generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them., Ian GoodfellowIan Goodfellow, prop. n.: An American computer scientist who invented generative adversarial networks, in which two neural networks compete to produce realistic fake images, and who pioneered research on fooling AI systems., ImagenImagen, prop. n.: Google's family of AI image generators, first shown in 2022, whose research showed that better understanding of language yields pictures that match descriptions more faithfully., image-to-image translationimage-to-image translation, n.: Turning one picture into another version of itself — a sketch into a photo, a day scene into night, a horse into a zebra — while keeping its layout., inpaintinginpainting, n.: Filling in a missing or erased part of a picture so it blends seamlessly with the rest — removing a stranger from a photo, or repairing a damaged one., instruction-based image editinginstruction-based image editing, n.: Changing a picture by telling an AI what to change — 'make it night', 'remove the car', 'give her a red coat' — instead of using editing tools by hand., latent diffusion modellatent diffusion model, n.: A diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run., mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better., mode collapsemode collapse, n.: When a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn., Monte Carlo methodMonte Carlo method, n.: Estimating an answer by running many random trials and averaging the results — like estimating the odds in a card game by dealing thousands of hands instead of calculating., music generationmusic generation, n.: Using AI to compose or produce music — from melodies written as notes to complete songs with vocals — often from a short text description like 'upbeat jazz with saxophone'.