Adam optimizer: Adam changes the training toolkit

The event and the narrow claim

On 22 December 2014, two researchers, Diederik P. Kingma and Jimmy Lei Ba, posted a paper titled "Adam: A Method for Stochastic Optimization", later presented as a conference paper at the third International Conference on Learning Representations in San Diego in 2015. The narrow, documented claim is specific: Adam is a first-order, gradient-based algorithm that computes an individual, adaptive step size for every parameter from running estimates of the first and second moments of that parameter's gradients, and corrects those estimates for the bias introduced by initializing them at zero. The name is an acronym for "adaptive moment estimation," not a person.

That is the whole of the original contribution, and it is worth separating from the broad historical judgment that Adam became the default training algorithm of the deep learning era. The first is a compact mathematical proposal that fits on a single slide; the second is an empirical fact about adoption that accrued over years and depended on later corrections and variants. A careful reading keeps the two apart, because much of what practitioners now call "Adam" (decoupled weight decay, warmup, a tuned epsilon) was added afterward by other authors and should not be silently attributed to the 2014 paper.

The problem: noisy gradients and mismatched scales

Training a neural network by stochastic gradient descent means estimating the gradient of a loss from a small, random minibatch and taking a step against it. Two properties of that setting make a bare, fixed learning rate awkward. First, the minibatch gradient is a noisy estimate of the true gradient, so individual steps are erratic. Second, different parameters live on very different scales: a weight feeding a rarely active feature receives large, infrequent gradients, while another receives small, dense ones, and one global step size cannot suit both.

Classical remedies existed. Momentum accumulates an exponentially decaying average of past gradients so that consistent directions build speed and oscillations cancel, and Nesterov's accelerated variant refines where that look-ahead is evaluated. But momentum still applies one shared step size to every coordinate. The open question Adam addressed was how to combine the smoothing benefit of momentum with a per-parameter scaling that reacts to each coordinate's own gradient history, cheaply enough to run on models with millions of parameters. Every design choice in Adam is a response to those two constraints.

AdaGrad and the per-parameter idea

The per-parameter idea was not new in 2014. In 2011, John Duchi, Elad Hazan, and Yoram Singer published AdaGrad in the Journal of Machine Learning Research, which gave each parameter its own learning rate by dividing a global rate by the square root of the running sum of that parameter's squared gradients. The effect is exactly the adaptation the setting calls for: coordinates that have seen large gradients are damped, and rarely updated coordinates keep a large effective step, which is why AdaGrad worked well on sparse, high-dimensional problems such as text with many rare features, and came with regret guarantees for online convex optimization.

AdaGrad also exposed the constraint that Adam's ancestors were built to remove. Because its denominator accumulates every squared gradient ever seen, it grows without bound, so the effective learning rate decays monotonically toward zero. On a convex problem that is acceptable, and even desirable. On a non-convex deep network that must keep learning through many epochs, the step size can shrink to nothing long before training is done, and progress stalls. The next generation of optimizers all attacked that single failure.

RMSProp and AdaDelta: from a growing sum to a moving average

The fix was to replace AdaGrad's ever-growing cumulative sum with an exponentially decaying moving average of squared gradients, which forgets the distant past and therefore never drives the step size to zero. Geoffrey Hinton proposed exactly this in an unpublished 2012 Coursera lecture on RMSProp, dividing each gradient by a running root-mean-square of its recent magnitudes. Independently, Matthew Zeiler's ADADELTA reached a related design and further tried to make the update unit-consistent so that no global learning rate had to be chosen at all.

These methods are the immediate technical predecessors of Adam, and they define its contribution by subtraction. RMSProp supplied the decaying second-moment estimate that keeps per-parameter scaling alive indefinitely, but it discarded the smoothing that momentum provided over the gradient direction itself. Adam's move was to keep both: a decaying average of the squared gradient (as in RMSProp) and a decaying average of the gradient (as in momentum), and to treat them explicitly as estimates of the first and second moments that need correcting.

How Adam actually works

At each step Adam reads the current minibatch gradient (call it g) and updates two running averages. The first estimates the mean gradient, m = beta1 * m + (1 - beta1) * g; the second estimates the uncentered variance, v = beta2 * v + (1 - beta2) * g^2. The paper's default decay rates are beta1 = 0.9 and beta2 = 0.999. Because both averages are initialized at zero, they are biased toward zero in the first steps, so Adam divides them by 1 - beta1^t and 1 - beta2^t to form bias-corrected estimates m_hat and v_hat. The parameter update is theta = theta - alpha * m_hat / (sqrt(v_hat) + epsilon), with defaults alpha = 0.001 and epsilon = 1e-8.

The behavior this produces is the reason the method spread. The ratio m_hat / sqrt(v_hat) acts like a signal-to-noise quantity: when gradients are consistent it approaches one in magnitude, and when they are noisy it shrinks, so the effective step naturally contracts near a minimum. The magnitude of each step is also bounded roughly by alpha regardless of gradient scale, a built-in trust region, and the whole method is invariant to any fixed diagonal rescaling of the gradients. The paper also introduced a variant, AdaMax, that replaces the second-moment norm with an infinity norm. Crucially, Adam stores two extra values per parameter, which became a real memory cost once models grew to billions of weights.

The evidence in the original paper

The 2014 paper is an empirical argument, not a theorem-only one. Kingma and Ba reported experiments on logistic regression, on multilayer neural networks, and on convolutional networks, using standard datasets including MNIST and CIFAR-10, and compared Adam against SGD with Nesterov momentum, AdaGrad, RMSProp, and AdaDelta. Across those settings Adam reached low training cost at least as quickly as the best competitor and was robust to its hyperparameters, which is the practical property that mattered most: a method that works out of the box with its default beta and alpha values saves practitioners the search that tuning an SGD learning-rate schedule demands. These are task-specific optimization curves, not measures of intelligence.

On the theoretical side the paper offered a convergence analysis in the online convex optimization framework, claiming a regret bound of order square-root T, meaning the average regret vanishes as training proceeds. That bound was presented as reassurance that the adaptive machinery did not break the guarantees its predecessors enjoyed. As the next section describes, that particular proof did not survive scrutiny, a reminder that Adam's real justification was always primarily empirical.

A proof that did not hold, and AMSGrad

In 2018, Sashank Reddi, Satyen Kale, and Sanjiv Kumar identified a genuine error in Adam's convergence argument. Their paper, On the Convergence of Adam and Beyond, constructed a simple convex problem on which Adam provably fails to reach the optimum, and traced the cause to the exponential moving average of squared gradients: because that average can shrink from one step to the next, a rare but large and informative gradient can be forgotten before it has moved the parameters, so the effective learning rate can grow when it should decay. They proposed AMSGrad, which keeps the maximum of all past second-moment estimates (a long-term memory) so the denominator never decreases, restoring a convergence guarantee.

The episode is instructive on two levels. It corrected the formal record, and it also demonstrated the limits of formal correction: AMSGrad fixed the counterexample but was not clearly better in practice, and it never displaced Adam in day-to-day use. The community kept the algorithm whose proof was flawed because the empirical case for it remained strong, which is an honest and slightly uncomfortable fact about how the field actually chose its default optimizer.

Do adaptive methods generalize?

A second line of criticism questioned not whether Adam converges but where it converges to. In 2017, Ashia Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht published The Marginal Value of Adaptive Gradient Methods in Machine Learning, showing on a constructed classification problem that plain gradient descent could reach zero test error while AdaGrad, RMSProp, and Adam approached the error of random guessing, and reporting that on several image and language benchmarks well-tuned SGD generalized as well as or better than adaptive methods despite worse training loss. Their recommendation was to reconsider the reflexive use of adaptive optimizers.

That finding was real and also domain-bound. On the convolutional image models of the day, SGD with momentum and a hand-tuned schedule often did win. But the picture reversed for the architecture that came to dominate: the original Transformer was trained with Adam using a custom warmup schedule, and Transformers proved notoriously hard to train with plain SGD, so Adam and its descendants became effectively mandatory for the models that followed. The lesson is that the best optimizer is not a property of an algorithm alone but of an algorithm paired with an architecture, a dataset, and a tuning budget.

AdamW and the weight-decay correction

The most consequential revision to Adam concerned regularization. Practitioners routinely added L2 regularization to the loss, assuming it was equivalent to weight decay as it is under plain SGD. In 2017, Ilya Loshchilov and Frank Hutter showed in Decoupled Weight Decay Regularization that the equivalence breaks for adaptive optimizers: because Adam divides by sqrt(v_hat), the L2 penalty on a parameter gets scaled down exactly for the parameters with large gradients, so it fails to regularize them as intended. Their fix, AdamW, decouples weight decay from the adaptive update by shrinking the weights directly, separately from the gradient step. This decoupled the tuning of the learning rate from the tuning of the decay coefficient and measurably improved generalization.

AdamW is now the default optimizer for training large Transformer models, from BERT-scale language models to the largest contemporary systems, which is why "Adam" in modern practice usually means AdamW. It is also the clearest illustration of the caution the record demands: AdamW is a distinct 2017 contribution built on top of the 2014 method, and attributing its benefits to the original Adam paper would be a common but real error of historical bookkeeping.

The continuing search for a better optimizer

Adam did not end optimizer research; it set the baseline everything else is measured against. Liyuan Liu and colleagues' rectified Adam (RAdam) analyzed the large variance of the adaptive learning rate during the first few steps of training and showed that the widely used learning-rate warmup trick can be understood as a variance reduction, offering a principled rectification instead. More radically, a Google team used automated program search to discover a new optimizer, reported in Symbolic Discovery of Optimization Algorithms; the result, Lion, tracks only a single momentum term and updates using the sign of that momentum, using less memory than Adam while matching or beating AdamW on several large vision and language benchmarks.

These successors matter for placing Adam historically. Each targets a specific, identified weakness (early-training instability, memory footprint, the second-moment buffer) rather than the core idea of moment-based adaptation, which every one of them retains in some form. That pattern, incremental correction around a stable core, is itself evidence that the 2014 design captured something durable rather than accidental.

How to place Adam in AI history

The honest summary has two levels. The narrow claim is that Kingma and Ba combined a decaying average of the gradient (momentum) with a decaying average of the squared gradient (RMSProp) plus a bias correction, and packaged it with defaults robust enough to use untuned. That is a modest, elegant piece of engineering, and its formal convergence claim was later found wanting. The broad judgment is that this package, together with the AdamW correction, became the standard optimizer of the deep learning and large-language-model era, the algorithm that trained a large share of the models a general reader has heard of.

Neither statement should swallow the other. Adam did not invent adaptive learning rates (AdaGrad did), nor the moving-average denominator (RMSProp and ADADELTA did), nor decoupled weight decay (AdamW did). It also did not make optimizer choice a solved problem, as both the generalization debate and the ongoing search for successors show. What it did was find a combination robust enough that a working researcher could stop thinking about the optimizer and think about the model instead, and there is a serious case that lowering that cognitive cost, more than any single benchmark number, is why so much subsequent work became feasible. That Kingma also co-authored the variational autoencoder the year before is a reminder that the same hands shaped more than one of the tools the field now takes for granted.

References

• Kingma and Ba (2014). Adam: A Method for Stochastic Optimization.
• Duchi, Hazan and Singer (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.
• Tieleman and Hinton (2012). Neural Networks for Machine Learning, Lecture 6: rmsprop.
• Zeiler (2012). ADADELTA: An Adaptive Learning Rate Method.
• Reddi, Kale and Kumar (2018). On the Convergence of Adam and Beyond.
• Wilson et al. (2017). The Marginal Value of Adaptive Gradient Methods in Machine Learning.
• Loshchilov and Hutter (2017). Decoupled Weight Decay Regularization.
• Liu et al. (2019). On the Variance of the Adaptive Learning Rate and Beyond.
• Chen et al. (2023). Symbolic Discovery of Optimization Algorithms.
• Vaswani et al. (2017). Attention Is All You Need.
• Kingma and Welling (2013). Auto-Encoding Variational Bayes.