Variational autoencoder: Learning a generative latent space

The event and the narrow claim

On 20 December 2013, Diederik Kingma and Max Welling, both then in the machine learning group at the University of Amsterdam, posted a paper titled "Auto-Encoding Variational Bayes." The submission date of the first arXiv version, not a later conference revision, marks the event. The paper proposed a way to fit continuous latent-variable models to large datasets using ordinary gradient descent, by combining a variational objective with a specific trick for differentiating through random sampling. It introduced the Stochastic Gradient Variational Bayes estimator and the Auto-Encoding Variational Bayes algorithm; the neural-network instantiation of that recipe is what the community came to call the variational autoencoder. The original paper states the estimator, the algorithm, and experiments on image data, and it was presented at the second International Conference on Learning Representations in April 2014.

The narrow documented claim is modest and precise: a lower-variance, differentiable estimator of a variational bound that lets a neural encoder and decoder be trained jointly by backpropagation. The broad historical judgment, that the VAE became one of the standard building blocks of deep generative modelling, rests on a decade of later work that reused it. Keeping those two claims apart is the whole discipline of reading this event correctly. A method that removes a training obstacle is not the same as a product that people use, and conflating the two produces the familiar mythology in which one paper is credited with everything that came after it.

Latent variables and the posterior that would not factor

A latent-variable model assumes each observation is generated from an unobserved code through a decoding distribution, with a prior over the codes. Fitting such a model requires reasoning about the posterior over codes given data, and for a nonlinear decoder that posterior has no closed form. Classical variational inference, surveyed by Jordan and colleagues in 1999 and later for a statistics audience by Blei, Kucukelbir and McAuliffe, replaces the intractable posterior with a simpler distribution and maximizes a lower bound on the data likelihood. The classical version optimizes separate variational parameters for every datapoint and usually relies on conditionally conjugate models where the updates have a convenient closed form.

Two constraints followed from that setup. First, re-optimizing per datapoint does not scale to hundreds of thousands of images. Second, the closed-form updates simply do not exist once the decoder is a general neural network. The autoencoder tradition, including Hinton and Salakhutdinov's 2006 demonstration that a deep network could compress data to a low-dimensional code and reconstruct it, had shown that neural encoders and decoders were powerful, but those models were deterministic compressors, not probabilistic generators you could sample from. Kingma and Welling's contribution sat exactly on the seam between the two traditions, borrowing the flexibility of neural networks and the principled objective of variational inference, and it removed the specific obstacle that had kept them apart.

The reparameterization trick

The technical heart of the paper is a way to take a gradient through a random sample. To train by gradient descent you must differentiate the expected reconstruction quality with respect to the parameters of the approximate posterior, but that posterior is what you draw the latent code from, so the sample itself depends on the very parameters you are differentiating. The naive Monte Carlo gradient of this quantity, the score-function estimator familiar from reinforcement learning, is unbiased but has high variance, which makes stable training difficult.

The paper replaces it with a reparameterization. Instead of sampling the latent code directly from a Gaussian with a given mean and standard deviation, it samples a parameter-free noise variable from a standard normal and forms the code as the mean plus the standard deviation times that noise. The randomness now lives in the noise variable, which does not depend on any parameter, so the gradient flows deterministically through the mean and the standard deviation produced by the encoder. The result is a low-variance estimator that works with ordinary backpropagation, and that practical property, more than any single equation, is what made the whole approach usable at the scale of real datasets. It converted an inference problem that had required bespoke derivations into one that a standard automatic-differentiation toolkit could handle. The reparameterization also carries a modelling assumption worth naming: it requires a continuous latent variable with a differentiable sampling path, which is why the original recipe applies cleanly to Gaussian codes and why extending it to discrete latents later demanded genuinely different machinery rather than a mechanical substitution.

An objective that reads like an autoencoder

The objective being maximized is the evidence lower bound, and its two terms are what give the model its name. Written per datapoint, it is the expected log-likelihood of the reconstruction under the decoder, minus the Kullback-Leibler divergence between the encoder's approximate posterior and the prior. The first term rewards reconstructing the input from its code, which is exactly the job of an autoencoder's decoder. The second term pulls every code distribution toward a shared prior, usually a standard normal, which regularizes the latent space so that nearby codes decode to similar outputs and codes drawn from the prior decode to plausible new data. That regularization is the difference between a compressor and a generator.

The encoder plays the role of an amortized inference network, sometimes called a recognition model: one shared network learns to map any input to the parameters of its approximate posterior, rather than solving a fresh optimization for each datapoint. That amortization is the answer to the scaling constraint described above. Kingma and Welling later wrote a book-length treatment, An Introduction to Variational Autoencoders, that develops this framing carefully and situates it within probabilistic modelling. The point worth holding onto is that the word "autoencoder" here is a consequence of the probabilistic objective, not the starting design; the architecture falls out of the mathematics rather than the other way around.

What the 2013 experiments actually showed

The paper's experiments were deliberately small by later standards and were aimed at validating the estimator, not at photorealism. Kingma and Welling trained the model on the MNIST handwritten-digit dataset and on the Frey Face dataset, and they compared the Stochastic Gradient Variational Bayes estimator against the wake-sleep algorithm, an earlier method for training layered generative models. Their results showed the variational lower bound improving faster and reaching better values with the new estimator than with wake-sleep, and that the advantage held as the number of latent dimensions grew, which is notable because more latent dimensions usually make inference harder.

They also demonstrated directly that the reparameterized estimator had far lower variance than the naive alternative, which is the concrete reason the method trained stably where the score-function estimator struggled. It is worth stating plainly what the reported numbers were: a bound on the log-likelihood achieved on two specific, modest image datasets. They are task-specific measurements of a fitting procedure, not a percentage of human ability or a claim about general quality. The historical weight of the paper comes from the generality of the method it validated, not from the visual quality of these first samples, which was limited.

A near-simultaneous derivation at DeepMind

The idea was ready to be found. Within weeks, in January 2014, Danilo Rezende, Shakir Mohamed and Daan Wierstra at Google DeepMind posted Stochastic Backpropagation and Approximate Inference in Deep Generative Models, which independently derived essentially the same class of gradient rules for propagating through Gaussian latent variables, framed as deep latent Gaussian models trained with a recognition network. That paper appeared at the International Conference on Machine Learning in 2014, and the two works are usually cited together as the origin of the modern amortized variational autoencoder.

Independent, near-simultaneous discovery is a useful signal for a historian. It suggests the result was the natural convergence of three maturing ingredients, variational inference, deep neural networks and stochastic gradient optimization, rather than a single isolated flash of insight that could easily not have happened. It also cautions against attributing the whole development to one team. The reparameterization idea itself had earlier antecedents in the statistics of transforming random variables, and the wake-sleep and Helmholtz-machine lineage had long pursued the goal of a recognition network paired with a generative one. What 2013 and early 2014 added was the specific, scalable, low-variance way to train that pairing end to end.

The competing generative approaches of the era

The VAE did not arrive into an empty field, and understanding its tradeoffs requires the real alternatives it stood beside. A few months later, Goodfellow and colleagues introduced generative adversarial networks, which trained a generator against a discriminator with no explicit likelihood at all. Adversarial models produced sharper images than VAEs for years, at the cost of unstable training and mode collapse, and the two families were the dominant rival paradigms for deep image generation through the late 2010s. A third family, autoregressive models, factorized the data distribution into a product of conditionals and modelled each element in turn, trading very slow sampling for exact, tractable likelihoods.

Normalizing flows offered a fourth route, building invertible transformations with tractable densities. Notably, that line of work fed back into the VAE rather than only competing with it: Rezende and Mohamed's 2015 flows paper and the later inverse autoregressive flow were used to make the approximate posterior far more expressive than a simple Gaussian, directly addressing one of the VAE's limitations. Placing the VAE beside these real alternatives, rather than against an obsolete caricature, is what makes its particular tradeoff legible. It offered a stable objective, an explicit likelihood bound and a reusable inference network, and it paid for those with less sharp samples than the adversarial approach delivered.

Disentanglement, discrete codes, and the latent-diffusion connection

The VAE's influence shows up less as a single dominant product and more as a reusable component that many later lines of work modified. beta-VAE, introduced by Higgins and colleagues at ICLR 2017, added a single hyperparameter that up-weights the divergence term and reported that the resulting models learned more interpretable, factorized representations on datasets such as faces, chairs and generated shapes. The conditional VAE of Sohn, Lee and Yan extended the framework to structured prediction by conditioning the encoder and decoder on side information. A separate branch replaced the continuous Gaussian code with a discrete one: van den Oord, Vinyals and Kavukcuoglu's VQ-VAE learned a codebook of latent vectors, and VQ-VAE-2 scaled that idea to generate high-fidelity images by fitting a powerful prior over the discrete codes.

The most consequential downstream tie is to image generation at scale. Latent diffusion models, the architecture behind Stable Diffusion, run the diffusion process not on raw pixels but inside the compressed latent space of an autoencoder, which is what makes high-resolution synthesis computationally affordable. This is the connection the timeline records, and it needs a precise caveat. The autoencoder used there is trained with additional perceptual and adversarial objectives, and the mere fact that a VAE-style component sits inside a diffusion pipeline does not make the original 2013 VAE a diffusion model, nor does it make diffusion a kind of VAE. The lineage of the compressed latent space is real; the identity of the two methods is not.

The honest limits and caveats

The VAE's characteristic weakness is blurry samples. A Gaussian decoder rewarded for average reconstruction quality tends to hedge, producing outputs that look smoothed compared with the crisp images from adversarial models. A second, subtler failure is posterior collapse: when the decoder is powerful enough to model the data largely on its own, the divergence term drives the approximate posterior back to the prior and the latent code ends up carrying little or no information. Bowman and colleagues documented this vividly when they applied VAEs to text with strong recurrent decoders, and their mitigations, including annealing the weight on the divergence term and weakening the decoder, became standard practice.

There is also a structural gap between the bound and the true likelihood. The evidence lower bound is tight only when the approximate posterior matches the true posterior, so a restricted posterior family and a single shared inference network both introduce slack, and maximizing the bound is not identical to maximizing likelihood. Much subsequent work, including the flow-based posteriors mentioned earlier, exists precisely to narrow that gap. A related caveat concerns evaluation: because the model reports a bound rather than the exact likelihood, comparisons across models are only as trustworthy as the tightness of each bound, and a better reported number can reflect a tighter estimator rather than a better generator. None of this diminishes the original contribution. It specifies what the contribution was and was not: a practical and general training method, not a guarantee of sample quality or of exact inference.

How to place the variational autoencoder in AI history

The most defensible reading treats the 2013 paper as an enabling method rather than a finished product. Its narrow achievement, a low-variance differentiable estimator that made amortized variational inference practical for deep nonlinear models, is precise and well documented in the paper itself. Its broad significance comes from what that estimator let others build over the following decade: interpretable representations, discrete codebooks, conditional generators, and above all the compressed latent spaces that modern image models operate in.

A reader tracing the timeline outward can use three questions to keep the levels of claim distinct. Which older idea did it complete? Variational inference and neural autoencoders, joined by the reparameterization trick and amortized inference. Which later work reused it while changing one ingredient? beta-VAE changed the objective's weighting, VQ-VAE changed the latent to a discrete codebook, latent diffusion changed what runs inside the latent space. Which limitation forced new research? Blurriness, posterior collapse and the bound gap each spawned their own literature. Answered this way, the variational autoencoder is neither the origin of generative AI nor a mere footnote to it, but one of the small number of general-purpose tools that a great deal of later work was built on top of, which is exactly the kind of event a sourced timeline should record with care.

References

• Kingma and Welling (2013). Auto-Encoding Variational Bayes.
• Rezende, Mohamed and Wierstra (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models.
• Jordan, Ghahramani, Jaakkola and Saul (1999). An Introduction to Variational Methods for Graphical Models.
• Blei, Kucukelbir and McAuliffe (2017). Variational Inference: A Review for Statisticians.
• Hinton and Salakhutdinov (2006). Reducing the Dimensionality of Data with Neural Networks.
• Goodfellow et al. (2014). Generative Adversarial Nets.
• Rezende and Mohamed (2015). Variational Inference with Normalizing Flows.
• Kingma et al. (2016). Improving Variational Inference with Inverse Autoregressive Flow.
• Sohn, Lee and Yan (2015). Learning Structured Output Representation using Deep Conditional Generative Models.
• Higgins et al. (2017). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.
• van den Oord, Vinyals and Kavukcuoglu (2017). Neural Discrete Representation Learning.
• Razavi, van den Oord and Vinyals (2019). Generating Diverse High-Fidelity Images with VQ-VAE-2.
• Bowman et al. (2015). Generating Sentences from a Continuous Space.
• Rombach et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models.
• Kingma and Welling (2019). An Introduction to Variational Autoencoders.