LSTM: Neural networks learn to remember

The narrow claim of a 1997 paper

In 1997, Sepp Hochreiter and Jürgen Schmidhuber published a paper titled Long Short-Term Memory in the journal Neural Computation. Its specific claim was modest and carefully bounded: it described a recurrent network unit that could learn to bridge time lags in excess of 1000 discrete steps, and it demonstrated this on controlled tasks where conventional recurrent training failed to learn at all. The central device was a memory cell whose internal state is protected by a self-connection of fixed weight, so that an error signal can flow backward across many steps without shrinking toward zero or blowing up. The published paper frames this as a solution to a training problem, not as a theory of memory, cognition, or general intelligence.

Separating the documented result from the broad historical judgment is the entire task of reading this event well. The narrow contribution is an architecture that made a specific class of gradient signal survive over long intervals. The wide judgment, that gated memory became one of the dominant tools for sequence modeling for roughly two decades, rests on a long chain of later work by many groups, most of it after 2007. Collapsing those into a single claim produces mythology; holding them apart produces history. The primary source PDF is the right place to see how narrow, and how carefully argued, the original proposal actually was.

The constraint: error signals that decayed exponentially

The problem LSTM addressed had been named before it was solved. Hochreiter's own 1991 diploma thesis had identified, in German and largely unread at the time, that the error signals a recurrent network sends backward through time tend to decay or grow exponentially with the number of steps, a diagnosis later summarized in Schmidhuber's history of deep learning. The consequence is that a network can, in principle, represent a dependency between an early input and a much later output, yet gradient descent supplies almost no usable signal to learn it.

The definitive contemporary statement of the difficulty came from Yoshua Bengio, Patrice Simard, and Paolo Frasconi, who proved that there is a structural trade-off: a recurrent system that stores information robustly against noise tends, for that very reason, to have gradients that vanish, so gradient descent cannot easily discover long-range structure. Their 1994 analysis is the theoretical backdrop against which LSTM should be read. It matters that the constraint was understood as fundamental, not incidental, because it explains why the fix had to be architectural. You could not simply train longer or tune a learning rate; the signal that training relies on was disappearing before it arrived.

The constant error carousel and how the gates work

The mechanism at the heart of LSTM is what the authors called the constant error carousel. Inside each memory cell sits a linear unit with a recurrent self-connection whose weight is fixed at 1.0. Because that connection neither amplifies nor attenuates, an error backpropagated through it stays constant across time steps, which is precisely the property Bengio and colleagues had shown ordinary recurrence lacks. This is the single idea that makes the rest possible, and everything else in the design exists to make that protected channel useful rather than rigid.

Around the carousel the paper places multiplicative gate units. An input gate learns when to let new information write into the cell, and an output gate learns when to expose the cell's contents to the rest of the network. Each gate is a small unit with a sigmoid output between 0 and 1 that scales the signal it controls, so the network learns, from data, when a memory should be protected from irrelevant inputs and when it should be read out. Christopher Olah's widely used walkthrough, Understanding LSTM Networks, lays out this flow gate by gate. The design is deliberately local in space and time: the paper notes its update cost per weight and time step is constant, which mattered for the modest hardware of the era.

What the original experiments actually showed

The 1997 evidence was drawn from synthetic tasks constructed to isolate the long-lag problem, not from speech or natural language. The paper studied problems such as the embedded Reber grammar and tasks that required a network to carry a piece of information across hundreds of intervening steps of distractor input, then reproduce or combine it. On these, LSTM learned solutions that standard recurrent networks, real-time recurrent learning, and back-propagation through time did not reach in the same conditions. That is the honest form of the original result: on constructed problems designed to require long memory, a method existed that learned where prior methods stalled.

It is worth stating what these experiments did not show. They did not, by themselves, demonstrate that LSTM would win at speech recognition, translation, or handwriting; those were later empirical questions answered by other work. The synthetic tasks were an existence proof about trainability, and the authors presented them as such. A reader who treats the 1997 benchmark numbers as evidence about language understanding is importing a conclusion the paper never claimed. The value of the original experiments is that they cleanly demonstrated the removal of a specific constraint under controlled conditions.

The forget gate: the LSTM people actually used

There is a subtlety that most casual histories omit. The architecture almost everyone means when they say "LSTM" is not exactly the 1997 architecture. The original cell had input and output gates but no way to actively clear its own memory, which caused the internal state to grow without bound when the network processed a long, continuous stream that was never segmented into tidy subsequences. Felix Gers, Schmidhuber, and Fred Cummins fixed this by adding a forget gate that lets a cell learn to reset itself at appropriate moments. Learning to Forget: Continual Prediction with LSTM introduced that gate in 2000, and it became standard.

This is not a pedantic point. The forget gate turned out to be one of the most important components of the whole design, a finding confirmed empirically fifteen years later. It also reframes the historical credit: the effective, deployable LSTM is a composite built over several years, with the 1997 constant error carousel as its indispensable core and the 2000 forget gate as the piece that made it robust on real, unsegmented data. Peephole connections, letting gates observe the cell state directly, were another later refinement from the same research line. The lesson is that a durable architecture is often an accretion of fixes, not a single frozen diagram.

The competing approaches of the same era

LSTM did not win by default; it competed with other responses to the same gradient problem. One school attacked the exploding-gradient half of the difficulty directly: Razvan Pascanu, Tomas Mikolov, and Bengio analyzed why gradients blow up and proposed gradient clipping alongside a soft constraint for the vanishing case, documented in On the difficulty of training recurrent neural networks. Clipping made plain recurrent networks trainable enough to be a real, cheaper alternative for some tasks, which is why LSTM's dominance was never a foregone conclusion.

A second alternative simplified the gating idea rather than clipping around it. Kyunghyun Cho and colleagues introduced the gated recurrent unit, which merges the input and forget gates into a single update gate and drops the separate output gate, in Learning Phrase Representations using RNN Encoder-Decoder. GRUs have fewer parameters and often match LSTM on many tasks, so through the mid-2010s practitioners genuinely chose between them case by case. Other lines, such as echo state networks that fixed the recurrent weights and trained only a readout, occupied yet another point in the design space. Placing LSTM against these live competitors, rather than against a strawman of the naive recurrent network alone, is what makes its eventual prominence a claim worth defending rather than assuming.

From toy sequences to speech, handwriting, and translation

The gap between the 1997 proof and LSTM's real influence was closed by a decade of applied results, mostly after graphics-processor training made deep recurrent models practical. In speech, Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton reported a test error of 17.7% on the TIMIT phoneme-recognition benchmark using deep bidirectional LSTM networks, a task-specific figure that was competitive with the best systems of its time, in Speech Recognition with Deep Recurrent Neural Networks. Shortly after, Haşim Sak, Andrew Senior, and Françoise Beaufays adapted LSTM for large-vocabulary acoustic modeling at Google, introducing a recurrent projection layer to control parameter count, in their 2014 architecture paper. These moved LSTM from a laboratory result to a production component.

The most consequential application was machine translation. Ilya Sutskever, Oriol Vinyals, and Quoc Le built an encoder-decoder entirely from LSTMs and reported a BLEU score of 34.8 on WMT-14 English-to-French, finding that reversing the source word order sharpened the optimization, in Sequence to Sequence Learning with Neural Networks. Two years later, Google's production translation system, described in the GNMT paper, stacked eight LSTM encoder and eight decoder layers with residual connections and attention. For a period, deep LSTM was the backbone of one of the most visible AI products in the world.

The empirical audit of the architecture

Because LSTM had accumulated variants over almost twenty years, it eventually invited a systematic audit of which pieces actually mattered. Klaus Greff, Rupesh Srivastava, Jan Koutník, Bas Steunebrink, and Schmidhuber ran what they called a search-space odyssey: eight LSTM variants across speech, handwriting, and polyphonic-music tasks, totaling roughly 5400 experimental runs and about fifteen years of aggregate CPU time. The headline result was reassuringly boring. No variant reliably beat the standard architecture, meaning the accreted design was close to a local optimum rather than a pile of arbitrary choices.

The study's diagnostic findings are the more useful ones for a historian. It identified the forget gate and the output activation function as the most critical components, which retroactively vindicated the Gers, Schmidhuber, and Cummins addition as central rather than cosmetic. It also found that some elaborations could be removed cheaply: coupling the input and forget gates, or dropping peephole connections, simplified the model without significant loss on the tested tasks. This is the kind of evidence that separates a real engineering artifact from a mythologized one. The parts that survived controlled ablation are the parts that earned their place.

The honest limits of what LSTM settled

LSTM removed one constraint; it did not remove all of them, and several of its own limits drove the next wave of research. Its computation is inherently sequential: each step depends on the previous cell state, so training does not parallelize across the time dimension the way later architectures would. Its memory, though far more durable than a plain recurrent network's, is still a fixed-size vector summarizing everything seen so far, which strains on very long inputs. And the "LSTM" label covers a family of subtly different designs, so a benchmark comparison is only meaningful when it says which variant, which regularization, and which training regime produced the number.

There is also a caution about interpretation that mirrors the field's broader habits. A strong TIMIT or BLEU result is a task-specific measurement under stated conditions, not a statement about understanding or reliability outside those tasks. The synthetic long-lag experiments of 1997 proved trainability, not comprehension. Keeping these distinctions sharp is not pedantry; it is what prevents a genuinely important architecture from being oversold into claims it never supported. LSTM was foundational precisely because it created new capabilities and, with them, new limits that subsequent work had to confront.

Placing LSTM in the arc of sequence modeling

The fair placement is this: LSTM was the architecture that made long-range dependencies practically learnable, and it held that role for roughly two decades before being displaced for most large-scale language tasks. Its constant error carousel solved the exact problem Bengio and colleagues had formalized, and the forget gate made that solution robust on real data. The applied breakthroughs in speech and translation, from Graves in 2013 to GNMT in 2016, are what turned a training trick into an era-defining tool. Those are separate contributions by separate teams, and the honest timeline keeps them distinct.

The displacement is part of the story too. The parallelization LSTM could not offer was exactly what the Transformer delivered, and Attention Is All You Need explicitly framed itself as removing the recurrent machinery that LSTM embodied. That does not diminish LSTM; it clarifies it. A useful way to navigate outward from this event is to ask three questions. Which earlier result named the constraint it removed? Which later application proved the architecture on real data? Which limitation forced the next design? Answering those keeps LSTM where it belongs: a decisive, well-evidenced turning point in how machines learned to remember, not a myth of a single paper that explained everything after it.

References

• Hochreiter and Schmidhuber (1997). Long Short-Term Memory.
• Hochreiter and Schmidhuber (1997). Long Short-Term Memory (Neural Computation).
• Bengio, Simard and Frasconi (1994). Learning long-term dependencies with gradient descent is difficult.
• Gers, Schmidhuber and Cummins (2000). Learning to Forget: Continual Prediction with LSTM.
• Greff, Srivastava, Koutnik, Steunebrink and Schmidhuber (2015). LSTM: A Search Space Odyssey.
• Pascanu, Mikolov and Bengio (2013). On the difficulty of training recurrent neural networks.
• Cho et al. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.
• Graves, Mohamed and Hinton (2013). Speech Recognition with Deep Recurrent Neural Networks.
• Sak, Senior and Beaufays (2014). Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition.
• Sutskever, Vinyals and Le (2014). Sequence to Sequence Learning with Neural Networks.
• Wu et al. (2016). Google's Neural Machine Translation System.
• Olah (2015). Understanding LSTM Networks.
• Schmidhuber (2015). Deep Learning in Neural Networks: An Overview.
• Vaswani et al. (2017). Attention Is All You Need.