Sequence-to-sequence learning: One sequence becomes another

The event and the narrow claim

On 10 September 2014, three Google researchers, Ilya Sutskever, Oriol Vinyals and Quoc V. Le, posted a paper arguing that a single neural system could learn to turn one sequence into another with almost no assumptions about the sequences involved. Their paper, later presented at NIPS 2014, used one deep long short-term memory network to read an input sentence into a fixed-length vector and a second deep LSTM to unroll that vector into an output sentence. The documented result was specific: on the WMT'14 English-to-French translation task, an ensemble of five reversed LSTMs reached a BLEU score of 34.81, and using the LSTM to rescore a phrase-based system's candidate list raised the score to 36.5. Those are task-specific translation numbers, not statements about intelligence in general.

The right question about this event is not whether it invented modern language models. It is which constraint it removed, which parts it inherited, and which later results depended on further work. Keeping the narrow documented claim separate from the broad historical judgment is what distinguishes a genuine turning point from a founding myth.

The constraint: fixed-width networks against variable-length language

The paper's opening frames the problem cleanly. Deep neural networks were already strong on speech and vision, but they could only be applied to problems whose inputs and outputs fit vectors of a fixed dimension. Translation, speech recognition and question answering do not: a sentence can be five words or fifty, the output length is not known in advance, and it rarely matches the input length. A standard recurrent network can map a sequence to a sequence only when the alignment between input and output positions is known ahead of time, which is exactly what translation lacks.

The task the authors chose, WMT'14 English-to-French, was demanding on purpose. Word order differs between the two languages, and the mapping between positions is non-monotonic. If one learned system could do well there without a hand-built alignment pipeline, the fixed-width limitation would be shown to be avoidable rather than fundamental. The claim under test was narrow and falsifiable: not that neural networks are intelligent, but that a general sequence-to-sequence mapping could be trained end to end and could compete with a mature engineered system on a hard, public benchmark.

Predecessors who mapped a sentence to a vector

Sutskever, Vinyals and Le were explicit that they were not first to the encoder-decoder idea. They credited Kalchbrenner and Blunsom, whose 2013 EMNLP paper on recurrent continuous translation models was, in their words, the first to map an entire input sentence to a vector. They also described their own design as very similar to the RNN Encoder-Decoder of Cho and colleagues, which appeared only months earlier in 2014, introduced the gated recurrent unit, and was used to rescore a phrase-based translation system rather than to translate on its own.

The distinctive contribution of the 2014 paper was therefore not the encoder-decoder concept but a demonstration that a clean, deep LSTM version of it could translate directly and beat a strong statistical baseline. The decoder network is essentially a recurrent language model of the kind Graves had described for sequence generation in 2013, with one change: it is conditioned on the vector the encoder produced. That lineage matters because it presents the system as a recombination of established parts, not a break from them. Reading the event as a first discovery of the idea would overstate it; reading it as the moment the idea was made to work standalone at scale is defensible.

How the two-LSTM encoder-decoder actually works

The mechanism is worth stating precisely. One LSTM reads the source sentence one token at a time, ending on a special end-of-sentence symbol, and its final hidden state becomes a single fixed-dimensional vector meant to represent the whole input. A second, separate LSTM takes that vector as its initial state and generates the target sentence token by token, each step producing a softmax distribution over the output vocabulary and feeding its own choice back in, until it emits its own end-of-sentence symbol. Using an explicit end symbol lets the model define a distribution over outputs of any length, which is how a fixed-width architecture escapes its fixed-width limitation.

Three design choices departed from the simplest version. The authors used two different LSTMs rather than one shared network, which added parameters cheaply and made it natural to train on several language pairs at once. They used four stacked layers rather than one, reporting that each additional layer reduced perplexity by nearly 10 percent. And they reversed the word order of the source sentence, discussed below. The whole design rests on the long short-term memory architecture Hochreiter and Schmidhuber introduced in 1997, chosen precisely for its ability to carry information across long gaps between related tokens.

The reversal trick and why it mattered

The single most cited practical finding of the paper is almost embarrassingly simple. Reversing the order of the words in every source sentence, while leaving the target sentences unchanged, improved results markedly. The authors report that this dropped test perplexity from 5.8 to 4.7 and raised the decoded translations' test BLEU from 25.9 to 30.6, and they called it one of the key technical contributions of the work. Mapping the source "c, b, a" onto the target "alpha, beta, gamma" put the opening words of each sentence near each other.

Their explanation is about optimization, not linguistics. When a source sentence is concatenated with its translation, corresponding words sit far apart, so the network faces a large minimal time lag between related tokens. Reversing the source leaves the average distance unchanged but places the first few source words very close to the first few target words, giving backpropagation short paths to establish communication early in the sequence. The authors expected this to help only the beginning of each translation; instead, reversed models did better on long sentences too, which they read as evidence of better memory use rather than a local fix. It is a revealing detail, because it shows the result depended on a training-dynamics hack, not on the architecture alone.

The evidence: BLEU on WMT'14 English-to-French

The numbers are concrete and were reported against real baselines. On the test set, a single forward LSTM with beam size 12 scored 26.17 BLEU; a single reversed LSTM scored 30.59; an ensemble of five reversed LSTMs reached 33.00 at beam size 1 and 34.81 at beam size 12. The phrase-based statistical baseline reported in the same paper scored 33.30, so the ensemble beat it by roughly 1.5 points. That was the headline: the first time a pure neural translation system outperformed a strong phrase-based baseline on a large task by a clear margin.

A second table records the hybrid setup. When the LSTM was used only to rescore the baseline's 1000-best candidate lists, a single forward model reached 35.61, a single reversed model 35.85, and an ensemble of five 36.5, within half a BLEU point of the 37.0 the authors measured for the contemporary state of the art. The oracle rescoring figure, around 45, showed how much headroom the candidate lists already contained. The system still could not handle words outside its 80,000-word output vocabulary, which penalized every translation containing a rarer word and capped the direct score. Reporting both the standalone and the rescoring results, rather than only the best one, is part of what makes the evidence credible.

Scale and engineering behind the result

The model was large for 2014, and the training was a systems effort as much as a modeling one. Each LSTM had four layers of 1000 cells with 1000-dimensional word embeddings, an input vocabulary of 160,000 and an output vocabulary of 80,000, for 380M parameters, of which 64M were pure recurrent connections. Training used a curated 12M-sentence subset of WMT'14, about 348M French and 304M English words. Optimization was plain stochastic gradient descent without momentum at a learning rate of 0.7, halved every half epoch after the fifth, for 7.5 epochs, with hard gradient-norm clipping and minibatches sorted so that sentences of similar length trained together.

Because a single GPU processed only about 1,700 words per second, the authors spread the model across an 8-GPU machine, one layer per GPU with four more GPUs parallelizing the softmax, reaching roughly 6,300 words per second, and even then training took about ten days. This belongs in any honest account of the event, because it shows the result depended on data selection, parallelization and decoding choices, not on the architecture in isolation. A beam of size 2 recovered most of the benefit of a beam of 12, so quality was not purely a function of expensive search.

The competing approach of the same season: attention

Seq2seq did not arrive alone. In early September 2014, Bahdanau, Cho and Bengio proposed jointly learning to align and translate, letting the decoder look back at all of the encoder's hidden states and weight them differently at each output step instead of relying on one summary vector. The seq2seq paper itself cited this line of work, noting that Graves had introduced a differentiable attention mechanism and that Bahdanau and colleagues had applied an elegant variant to translation.

This matters for judging novelty carefully. The fixed-vector encoder-decoder and content-based attention were near-contemporaries attacking the same limitation from different directions. Seq2seq showed that a plain deep LSTM could compress a sentence well enough to translate it; attention showed that the compression step could be sidestepped. Reversing the source words was, in a sense, a low-tech way of buying some of what attention would soon provide by construction. Treating the two as rivals of the same season, rather than one as the ancestor of the other, is closer to the record than a tidy succession story.

Downstream influence: captions, conversation, and the Transformer

The seq2seq framing proved unusually portable, and the same authors demonstrated it quickly. Show and Tell replaced the encoder LSTM with a convolutional network over an image and generated captions with the decoder. Grammar as a Foreign Language cast syntactic parsing as translating a sentence into a linearized parse tree. A Neural Conversational Model trained the same machinery on dialogue pairs to produce a chatbot. The common lesson was that many problems could be posed as mapping one sequence to another, after which the same architecture would attempt them.

The influence then compounded through production and architecture. Google's neural machine translation system in 2016 scaled deep LSTMs with attention, residual connections and wordpieces into a deployed service. And in 2017 the Transformer kept the encoder-decoder shape while replacing recurrence entirely with attention, inheriting the seq2seq problem statement even as it discarded the LSTM. The encoder-decoder interface that seq2seq popularized outlived the specific network that first made it work at scale, which is a strong form of downstream influence: the framing survived its own implementation.

The honest limits: the bottleneck it left behind

The paper's central weakness was structural, and its authors were partly aware of it. Forcing an entire source sentence through one fixed-dimensional vector asks a single object to carry everything the decoder will need, and information from the far end of a long input can be squeezed out. The reversal trick eased this for the tested lengths, and, surprisingly, the reversed model did not visibly degrade on long sentences, but the bottleneck was real and attention was designed specifically to remove it. The out-of-vocabulary problem was a second hard limit: a fixed 80,000-word target vocabulary penalized every rarer word, which later work on subword units, such as byte-pair encoding for rare words, addressed directly.

There are quieter caveats worth stating. The strongest headline number came from an ensemble of five models, not a single network, and the very best translation-quality figure came from rescoring an existing statistical system rather than translating from scratch. The learned sentence vectors clustered by meaning and were sensitive to word order in a two-dimensional PCA visualization, which is suggestive but not a rigorous account of what the representation encodes. None of this diminishes the result; it locates it precisely, which is the point of a sourced timeline.

How to place sequence-to-sequence in AI history

The defensible reading is that seq2seq was the paper that made the neural encoder-decoder a practical, general interface. It did not originate the encoder-decoder idea, which it credited to Kalchbrenner and Blunsom and to Cho and colleagues, and it did not introduce attention, which arrived alongside it. What it demonstrated, with numbers on a hard public benchmark, was that a clean deep LSTM could map variable-length inputs to variable-length outputs well enough to beat a mature phrase-based system, and that the same recipe transferred to captioning, parsing and dialogue.

A reader placing the event should follow three threads outward. Which earlier idea supplied a needed part: the LSTM, the recurrent language-model decoder, the earlier encoder-decoders. Which later project reused the framing while changing an ingredient: attention, GNMT, the Transformer, and the text-to-text lineage that followed. And which limitation forced new work: the fixed-vector bottleneck and the closed vocabulary. Read that way, seq2seq is neither the beginning nor the end of the story, but the hinge where "map one sequence to another" became a reusable engineering interface, and much of what came next is best understood as improving one of its parts.

References

• Sutskever, Vinyals and Le (2014). Sequence to Sequence Learning with Neural Networks.
• Sutskever, Vinyals and Le (2014). Sequence to Sequence Learning with Neural Networks, NeurIPS proceedings.
• Kalchbrenner and Blunsom (2013). Recurrent Continuous Translation Models.
• Cho et al. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.
• Bahdanau, Cho and Bengio (2014). Neural Machine Translation by Jointly Learning to Align and Translate.
• Graves (2013). Generating Sequences With Recurrent Neural Networks.
• Hochreiter and Schmidhuber (1997). Long Short-Term Memory.
• Vinyals et al. (2014). Show and Tell: A Neural Image Caption Generator.
• Vinyals et al. (2014). Grammar as a Foreign Language.
• Vinyals and Le (2015). A Neural Conversational Model.
• Wu et al. (2016). Google's Neural Machine Translation System.
• Vaswani et al. (2017). Attention Is All You Need.
• Sennrich, Haddow and Birch (2015). Neural Machine Translation of Rare Words with Subword Units.