DDPM: Learning to reverse noise

The event and the narrow claim

On 19 June 2020, three researchers at the University of California, Berkeley, Jonathan Ho, Ajay Jain and Pieter Abbeel, posted a paper titled Denoising Diffusion Probabilistic Models. Its headline result was concrete and task-specific: an unconditional image generator trained by learning to reverse a gradual noising process reached an Inception score of 9.46 and a Fréchet Inception Distance (FID) of 3.17 on CIFAR-10, which the authors reported as a state-of-the-art FID at the time, and produced 256x256 samples on the LSUN datasets of quality similar to ProgressiveGAN. Those are measurements on specific datasets, not statements about general capability.

The narrow contribution is easy to state and easy to overstate. The paper did not invent diffusion models, and it did not by itself create text-to-image generation. What it did was show that a simplified training recipe for an existing but underperforming class of models could match or beat the leading generative approaches on image sample quality, and it explained why the recipe worked by tying it to a separate line of research on score matching. The broad historical judgment, that iterative denoising became one of the dominant routes to generative media, rests on several years of work that followed, not on this single result. Keeping those two levels of claim apart is the entire discipline of reading the event well.

Two lineages that converged

The idea of generating data by reversing a corruption process was already five years old. In 2015, Jascha Sohl-Dickstein and colleagues published Deep Unsupervised Learning using Nonequilibrium Thermodynamics, which proposed slowly destroying the structure in a data distribution through an iterative forward diffusion, then training a model to reverse each small step. The framework was mathematically appealing because it gave a tractable, flexible probabilistic model, but in practice its sample quality lagged well behind the generative models that drew attention over the next few years. The idea existed; the results did not yet compel anyone to switch.

A second, initially separate lineage came from score-based modeling. In 2019, Yang Song and Stefano Ermon published Generative Modeling by Estimating Gradients of the Data Distribution, which learned the gradient of the log density (the score) at multiple noise levels and generated samples with annealed Langevin dynamics. DDPM's real conceptual move was to recognize that these two lineages describe nearly the same object. The paper showed that its denoising training objective is equivalent, up to weighting, to denoising score matching across noise scales, and that its sampling procedure resembles Langevin dynamics. That unification is why the work reads as a hinge: it took a physically motivated but weak model and a strong but differently framed estimation method and showed they were two views of one recipe.

The generative landscape it entered

To judge the 2020 result fairly, place it against its real competitors rather than a caricature. The dominant approach to high-fidelity image synthesis was the generative adversarial network, introduced by Ian Goodfellow and colleagues in Generative Adversarial Networks, in which a generator and a discriminator are trained against each other. By 2018, scaled adversarial training in Andrew Brock and colleagues' Large Scale GAN Training for High Fidelity Natural Image Synthesis reached an Inception score above 166 on 128x128 ImageNet, far beyond anything else on that benchmark. GANs owned sample quality, but they were notoriously unstable to train and prone to mode collapse, where the generator covers only part of the data distribution.

The likelihood-based alternatives traded that instability for other costs. Variational autoencoders, from Diederik Kingma and Max Welling's Auto-Encoding Variational Bayes, optimized a stable bound on the data likelihood but tended to produce blurry samples. Autoregressive models generated pixels or tokens in sequence with strong likelihoods but slow sampling, and normalizing flows required carefully constrained, invertible architectures. This was the tension diffusion addressed: it kept a stable, likelihood-flavored training objective, avoided adversarial dynamics entirely, and still reached sample quality competitive with GANs. It did not remove every tradeoff, but it moved one that had blocked stable models from the quality frontier.

How the forward and reverse processes work

The mechanism is a pair of processes over a fixed number of steps, set to T equal to 1000 in the paper. The forward process is not learned. It takes a real image and, over those 1000 steps, adds a small amount of Gaussian noise at each step according to a chosen variance schedule, so that by the final step the image is indistinguishable from pure noise. A convenient property of Gaussian noising is that the state at any step can be sampled directly from the original image in closed form, without simulating every intermediate step, which makes training efficient.

The reverse process is what the network learns. Starting from pure Gaussian noise, the model repeatedly estimates and removes a portion of the noise, taking a slightly less noisy sample at each step, until an image emerges. Because each reverse step only has to undo a tiny corruption, the individual prediction problem is easier than mapping noise to a finished image in one shot, which is roughly what a GAN generator attempts. The design decomposes a hard global synthesis problem into a long chain of easy local denoising problems. That decomposition is the structural reason the model trains stably, and it is also the source of its central cost, discussed below: quality is bought with many sequential steps.

The reparameterization that made it train

The 2015 formulation trained the reverse process to predict the mean of each denoising step. The practical advance in the DDPM paper was a change of what the network predicts and how the loss is weighted. Instead of predicting the posterior mean directly, the network predicts the noise component that was added to reach the current noisy sample, an approach usually written as epsilon prediction. Given that prediction, the mean of the reverse step follows analytically.

Paired with this, the authors replaced the full variational lower bound with a simplified objective: a plain mean-squared error between the true noise and the predicted noise, effectively dropping the per-step weighting that the variational bound would impose. They reported that this reweighting, which places relatively more emphasis on harder, higher-noise steps, was beneficial for sample quality even though it is no longer a bound on the likelihood. This is the crux of the paper as an engineering result. The model was expressible before; what changed was a parameterization and a loss that let it train to high quality and that exposed its equivalence to denoising score matching with Langevin dynamics. The lesson generalizes beyond diffusion: how you parameterize and weight a training objective can matter as much as the underlying probabilistic model.

The numbers, and what they did and did not show

On CIFAR-10, the reported unconditional Inception score of 9.46 and FID of 3.17 were the compelling evidence, because FID is a standard measure of how close generated images are to real ones in a feature space, and lower is better. On the 256x256 LSUN church and bedroom datasets, the authors judged sample quality comparable to ProgressiveGAN. These are the results that made researchers take diffusion seriously as a quality-competitive method rather than a theoretical curiosity.

The likelihood story was deliberately less flattering, and the paper was candid about it. Its model reported a negative log likelihood of at most 3.70 bits per dimension on the CIFAR-10 test set, worse than strong autoregressive likelihood models of the period; a sparse transformer reached 2.80 and a gated PixelCNN about 2.90 bits per dimension. The authors examined this gap and found that more than half of the model's lossless code length went to describing imperceptible distortions, so the density it assigns is dominated by fine detail that a human viewer does not register. This is an unusually honest self-assessment: it separates perceptual sample quality, where the model excelled, from likelihood, where it did not, and it warns against collapsing the two into a single notion of being a good generative model.

The network that does the denoising

The denoiser is a convolutional U-Net, an encoder-decoder with skip connections that keep fine spatial detail available at the output. The paper describes a backbone similar to an unmasked PixelCNN++, with group normalization throughout and self-attention applied at the 16x16 feature-map resolution. Because a single network handles every one of the 1000 denoising steps, it must know which noise level it is operating at; the model conditions on the timestep through a Transformer-style sinusoidal position embedding fed into its residual blocks.

Two points follow from these details. First, the architecture is not exotic: it recombines components established elsewhere in image modeling, which is part of why the approach was quickly reproducible and extensible. Second, the choice to share one network across all noise levels, conditioned on time, is what makes the long reverse chain practical to train and store, since it avoids learning a separate model per step. As with many architectural turning points, the novelty lies in the arrangement and the training objective more than in any single new building block.

The cost problem, and its first fixes

The obvious weakness was speed. Generating one image required running the network sequentially for on the order of 1000 steps, which made sampling far slower than a GAN's single forward pass. This limitation was recognized immediately and became the first target of follow-up work. Later in 2020, Jiaming Song, Chenlin Meng and Stefano Ermon introduced Denoising Diffusion Implicit Models, a non-Markovian, largely deterministic sampler that reuses the same trained network but produces high-quality samples with far fewer steps, reporting speedups of roughly 10 to 50 times in wall-clock terms.

In early 2021, Alex Nichol and Prafulla Dhariwal's Improved Denoising Diffusion Probabilistic Models added learned reverse-process variances and a cosine noise schedule, which improved log likelihoods and allowed sampling with an order of magnitude fewer forward passes. Around the same time, Yang Song and colleagues' Score-Based Generative Modeling through Stochastic Differential Equations recast both the DDPM and score-matching families as discretizations of continuous-time stochastic differential equations, which provided a unifying mathematical framework and additional sampling options. Within a year of the original paper, the step-count problem had gone from a fatal-looking flaw to an active and tractable research area.

From a research result to the diffusion era

The result that turned diffusion from promising to dominant came in 2021, when Prafulla Dhariwal and Alex Nichol published Diffusion Models Beat GANs on Image Synthesis. Using an improved architecture and classifier guidance, a technique that steers sampling toward a target class using gradients from a separately trained classifier, they reported an FID of 4.59 on 256x256 ImageNet, surpassing the strongest GANs on that benchmark while maintaining better distribution coverage. Jonathan Ho and Tim Salimans then removed the extra classifier in Classifier-Free Diffusion Guidance, obtaining the same quality-versus-diversity control by jointly training conditional and unconditional models, a technique that became standard in nearly every later text-to-image system.

The step that made diffusion economical at high resolution was Robin Rombach and colleagues' High-Resolution Image Synthesis with Latent Diffusion Models, which ran the diffusion process in the compressed latent space of a pretrained autoencoder rather than on raw pixels, and added cross-attention conditioning. That design is the direct basis of Stable Diffusion. In parallel, text-to-image systems built on diffusion, including Google's Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, demonstrated that pairing a diffusion image model with a large language encoder produced photorealistic results from text prompts. Each of these is a distinct contribution that reused the DDPM recipe and changed one other ingredient: the guidance method, the space the process runs in, or the conditioning signal.

The honest limits and caveats

Several qualifications keep the story accurate. DDPM was not the origin of diffusion modeling; that credit belongs to the 2015 nonequilibrium-thermodynamics work, with score matching as a parallel root. The 2020 paper's own datasets were modest by later standards, CIFAR-10 and category-specific LSUN scenes, not open-domain, text-conditioned generation. Its sampling was slow, and its likelihoods trailed the best likelihood-based models. Guidance, latent-space operation, large-scale text conditioning, and fast samplers were all later additions, not present in the original.

There are also caveats that apply to the whole family. Sample quality as measured by FID is not the same as factual correctness, safety, or coverage of rare cases, and a low FID does not certify that a model avoids memorizing training images or reproducing biases in its data. The generative capabilities that followed also raised real questions about consent for training data and the provenance of synthetic media that the 2020 research paper neither caused nor addressed. Reporting the event honestly means naming the benchmark it moved and refusing to inflate a strong CIFAR-10 FID into a claim about general intelligence or creativity.

How to place DDPM in AI history

The most defensible placement treats DDPM as the empirical turning point that made an older idea competitive, rather than as the invention of diffusion or of generative media. Its documented, narrow achievement is a state-of-the-art FID on CIFAR-10 and a training recipe, the epsilon-prediction network with a simplified reweighted objective, that trained stably to high quality and connected diffusion to score matching. Its broad significance is inferred from what depended on it: faster samplers, guidance, latent diffusion, and the text-to-image systems that reached the public a couple of years later.

A useful way to navigate outward from this node is to ask three questions of each neighbor. Which earlier idea supplied a component the recipe needed, as the 2015 diffusion framework and 2019 score matching did? Which later project reused the recipe while changing one ingredient, as guidance, latent diffusion, and large-scale text conditioning each did? And which limitation forced a new discovery, as the 1000-step sampling cost forced faster samplers? Answering those keeps the timeline from becoming a list of famous product names and preserves the distinction between a well-measured research result and the ecosystem that grew around it. DDPM matters because it redirected a research trajectory, not because it made everything that followed inevitable.

References

• Ho, Jain and Abbeel (2020). Denoising Diffusion Probabilistic Models.
• Sohl-Dickstein, Weiss, Maheswaranathan and Ganguli (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics.
• Song and Ermon (2019). Generative Modeling by Estimating Gradients of the Data Distribution.
• Goodfellow et al. (2014). Generative Adversarial Networks.
• Kingma and Welling (2013). Auto-Encoding Variational Bayes.
• Brock, Donahue and Simonyan (2018). Large Scale GAN Training for High Fidelity Natural Image Synthesis.
• Song, Meng and Ermon (2020). Denoising Diffusion Implicit Models.
• Nichol and Dhariwal (2021). Improved Denoising Diffusion Probabilistic Models.
• Dhariwal and Nichol (2021). Diffusion Models Beat GANs on Image Synthesis.
• Song et al. (2021). Score-Based Generative Modeling through Stochastic Differential Equations.
• Ho and Salimans (2022). Classifier-Free Diffusion Guidance.
• Rombach et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models.
• Saharia et al. (2022). Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.