Attention mechanism (Bahdanau, Cho and Bengio): Learning where to look

A decoder that learns where to look

On 1 September 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio posted a paper on neural machine translation that learns to align and translate jointly. Bahdanau was listed at Jacobs University Bremen and his co-authors at the Université de Montréal, in whose lab the work was done. The model, which the authors called RNNsearch, changed one thing about the neural translators of the day. Instead of compressing a whole English sentence into a single fixed-length vector before writing any French, its decoder built a fresh weighted summary of the source sentence for every word it produced, with the weights computed by a small network trained along with everything else. The paper described this as a mechanism of attention in the decoder, and the name stuck.

The narrow, documented claim is modest and specific. On the WMT'14 English-to-French task, the attention model clearly beat an otherwise comparable encoder-decoder without attention, held its quality on long sentences where that baseline collapsed, and came close to a conventional phrase-based system once sentences containing unknown words were set aside. The broad historical claim is different and rests on later events: that a soft, learned, content-based weighting over a set of vectors became one of the basic operations of deep learning, and eventually the core of the Transformer. The dates also need care. The arXiv preprint went through seven versions, the last in May 2016, and the paper was formally published as an oral presentation at ICLR 2015. This entry dates the event to the first public posting, because that is when the idea entered circulation and when rival groups began citing it.

The bottleneck the Montreal group had just measured

By mid-2014, neural networks had begun to translate whole sentences on their own rather than serving as one scoring feature inside a statistical pipeline. The common design was an encoder-decoder: one recurrent network read the source sentence and left behind a single vector, and a second recurrent network generated the translation from that vector. The weakness was plain in principle and, that summer, measured in practice. In a companion analysis of encoder-decoder translation models, posted on 3 September 2014, Cho, van Merriënboer, Bahdanau and Bengio reported that such systems handled short sentences without rare words reasonably well, but that quality fell quickly as sentences grew longer and as unknown words accumulated.

The same group tried more than one remedy. A second companion paper led by Jean Pouget-Abadie, posted the same day, attacked the length problem from outside the network: it automatically cut a long input into segments that the neural model could translate separately, then joined the translated pieces. That approach accepted the fixed vector and worked around it. The attention paper instead dropped the requirement that a sentence pass through one vector at all, conjecturing that the fixed length was itself the bottleneck.

The competition was close in time. Nine days after the attention paper appeared, Ilya Sutskever, Oriol Vinyals and Quoc Le at Google posted their sequence-to-sequence paper, which kept the single fixed vector and pushed it much further with deep LSTMs, an 8-GPU training machine and the trick of reversing the order of the source words. Even its first version cited the Montreal work, describing it as an elegant variant of a differentiable attention idea that Alex Graves had introduced, and listed its 28.45 BLEU score in the results table. The two papers are best read as rival answers to the same problem, posted within days of each other, one strengthening the single-vector design and the other abandoning it.

Soft search, step by step

The mechanism has three parts, and each is simple. First, the encoder is a bidirectional recurrent network: one pass reads the English sentence left to right, another right to left, and the two hidden states at each position are joined together. The result is not one vector but a sequence of them, which the paper calls annotations, one per source word, each summarizing that word along with its neighbours on both sides.

Second, before emitting each French word, the decoder scores every annotation against its own current state. The scoring function, which the paper calls an alignment model, is a feedforward network with a single hidden layer that combines the decoder state with one annotation and returns one number. A softmax turns the scores for all source positions into weights that are positive and sum to one. Third, the decoder takes the weighted average of the annotations under those weights and uses that context vector, together with the previous word and its own state, to predict the next word. Because every step is differentiable, the scoring network learns by ordinary backpropagation along with the rest of the model. Alignment is never supplied as a label, and it is not treated as a hidden variable to be searched over, as it was in classical statistical translation.

The paper's own examples show what this buys. Translating "European Economic Area", the model put its weight on "Area" when writing "zone", then stepped back through "Economic" and "European" to finish "zone économique européenne", reversing the order of the modifiers as French requires. When it rendered "the man" as "l' homme", the weights for the article spread across both English words. That matters because the French article depends on the noun that follows: a hard one-to-one alignment would map "the" to "l'" without seeing "man", while a soft alignment can consider both. The authors also plotted the weight matrices for whole sentences, producing the now familiar heat maps in which a mostly diagonal band bends wherever word order differs between the languages.

What the September 2014 numbers did and did not show

The experiment was deliberately controlled. The attention model and the baseline encoder-decoder were trained on the same English-French data, which the authors reduced from about 850 million words to 348 million with a data-selection method. Both used a shortlist of the 30,000 most frequent words in each language, mapping everything else to an unknown-word token, had 1,000 hidden units, and trained for roughly five days. The test set was the 3,003 sentences of the WMT'14 news test.

On all test sentences, the attention model trained on sentences of up to 50 words scored 26.75 BLEU, against 17.82 for the matching encoder-decoder. Even the attention model trained only on sentences of up to 30 words, at 21.50, beat the baseline trained on longer ones. A plot of BLEU against sentence length made the purpose visible: the baseline's score fell steeply as sentences lengthened, while the 50-word attention model showed no deterioration even past 50 words. Long test sentences printed in the paper make the same point concretely: the baseline drifted from the source meaning partway through, in one case after about 30 words, while the attention model preserved the full meaning.

The comparison with conventional translation needs more care than it usually gets. The phrase-based Moses system scored 33.30 BLEU on all sentences, well ahead of every neural model in the table. The near-parity claim applies only to the subset of sentences in which neither the source nor the reference contained an unknown word. There, the attention model scored 34.16 against 35.63 for Moses, and a version trained much longer, until development performance stopped improving, reached 36.15. Moses also drew on a separate 418-million-word monolingual corpus that the neural models did not use. The honest summary is that attention removed the long-sentence collapse and closed most of the gap, but that the vocabulary limit still left the neural system behind on real text, which the authors named as the main challenge left for future work.

Soft addressing was already in the air

The attention paper did not invent the idea of a network that weights parts of its input, and it said so. Its related-work section pointed to Alex Graves's 2013 work on generating sequences, in which a handwriting-synthesis network used a soft window, a mixture of Gaussian bumps sliding along the text, to decide which character to write next. The difference was structural. Graves's window could only move forward, which suits handwriting but not translation, where words often must be reordered across long distances. Bahdanau and colleagues replaced the sliding window with a score for every source position at every output step.

Other groups were working on neighbouring ideas at the same moment. A DeepMind model of visual attention, posted in June 2014, chose where to look in an image by taking discrete glimpses; that choice is not differentiable, so the model had to be trained with reinforcement learning. Seven weeks after the translation paper, Graves, Wayne and Danihelka's Neural Turing Machine used content-based addressing to read from and write to an external memory, and listed both the handwriting model and the Bahdanau paper among the differentiable models of attention that preceded it. The distinctive choice in Montreal was therefore not attention in the abstract. It was a soft, fully differentiable, content-based weighting over every element of an encoded sequence, trained end to end on a task where alignment is genuinely out of order.

Priority arguments reach further back. Jürgen Schmidhuber's group has long argued for an older lineage, and in 2021 Schlag, Irie and Schmidhuber showed that linearised self-attention is formally equivalent to fast weight controllers of the early 1990s, in which one network programs the fast weights of another through patterns now called keys and values. That result establishes a mathematical kinship with a later variant of attention, not a line of influence on the 2014 paper. Personal recollection is a weaker kind of evidence. In an email to Andrej Karpathy, reported by the newsletter Turing Post in December 2024, Bahdanau recalled that the idea came from school translation exercises in which the eyes move back and forth between source and target, that he named the model RNNSearch, and that Bengio added the word attention in a late pass. The newsletter observed that the word sits in the method section rather than the conclusion, which matches the September 2014 version. Such memories add color, but the documentary record is the paper itself.

From a translation fix to a general operation

Within a year the same operation was carrying other tasks. In February 2015, Kelvin Xu and colleagues, including Cho and Bengio, introduced an image-captioning model that attends over regions of a convolutional feature map while generating each word, citing the translation work as inspiration and reporting state-of-the-art results on Flickr8k, Flickr30k and MS COCO. In June, Jan Chorowski, Bahdanau and colleagues adapted it to speech recognition, where a direct port of the translation model reached a competitive 18.7 percent phoneme error rate on TIMIT; adding location awareness, and then keeping the weights from concentrating on single audio frames, brought that to 17.6 percent.

Translation research absorbed the idea fastest. At Stanford, Luong, Pham and Manning compared a global variant that attends to every source word with a local one that attends to a predicted window, tested simpler scoring functions such as a dot product, and found them better than the original concatenation-based score. Their ensemble set a new state of the art of 25.9 BLEU on WMT'15 English to German. By 2016 the method was winning shared tasks and had reached products. Edinburgh's entries to the WMT 2016 news task were attentional encoder-decoders built on the Bahdanau design, and in the human evaluation they were the best or tied-best constrained system in 7 of the 8 directions they entered. Google's production neural translation system joined eight-layer LSTM encoders and decoders with an attention module its authors described as similar to the Montreal one, and Google announced in September 2016 that it was serving all Chinese-to-English translations in its apps with it, about 18 million a day.

Explanations reached a wider audience as well. Chris Olah and Shan Carter's 2016 Distill essay presented attention as one of a handful of ways to augment recurrent networks and made the central design point plain: for a network to learn where to focus, it has to attend everywhere to some degree, so that the choice of focus stays differentiable.

The Transformer keeps the attention and drops the recurrence

In every system above, attention was an addition to a recurrent network that still processed words one after another. Two 2016 papers pointed the other way. Cheng, Dong and Lapata's LSTM-networks for machine reading applied attention within a single sequence, which they called intra-attention, and Parikh and colleagues' decomposable attention model for natural language inference used attention without relying on word order, making it trivially parallel and leaving it with almost an order of magnitude fewer parameters than earlier models. Jakob Uszkoreit, one of its authors, was also an author of the Transformer.

The 2017 Transformer paper made that move general. It cites the Bahdanau paper as its example of additive attention, contrasts it with the dot-product form it adopted (similar in theoretical cost, but much faster in practice because it reduces to optimized matrix multiplication), and describes its encoder-decoder attention layers as mimicking the attention of earlier sequence-to-sequence models. What changed was everything around the operation. Queries, keys and values replaced the decoder state and the annotations, positions attended to one another within the same sequence, and recurrence disappeared. The Transformer's own entry in this archive covers that reorganization. For this event the point is lineage: the weighted sum that the 2014 paper computed once per output word became the operation repeated in every layer of the dominant architecture.

Institutions recorded the debt later. In April 2025, ICLR named the paper runner-up for its Test of Time award, behind the Adam optimizer, crediting it with changing how sequence-to-sequence models process information and describing its attention as a basis for transformers and large language models.

The honest limits

The paper was candid about cost. Its scoring network has to run once for every pair of source and target positions, a burden the authors judged tolerable for translation, where most sentences run 15 to 40 words, but warned might limit the scheme elsewhere. The pairwise cost did not go away as attention spread. Luong and colleagues introduced local attention partly to avoid looking at every source word, the Distill essay listed sparse, cheaper attention as an open problem, and the Transformer paper itself suggested restricting attention to a neighbourhood for very long sequences.

Attention also left untouched the problems it was not designed for. The gap between the all-sentence and known-word results points to the 30,000-word vocabulary, rather than the bottleneck, as the main remaining handicap, and it took separate work on very large target vocabularies and on subword units, such as the wordpieces in Google's system, to address it. Purely content-based attention could also lose its place: Chorowski's speech experiments found that the translation-style model worked only on utterances about as long as those it was trained on, until location information was added. And the evidence in the original paper came from one language pair, one test set and an automatic metric, BLEU, that measures overlap with reference translations rather than meaning.

The most persistent misreading concerns interpretation. The alignment heat maps invited the view that attention weights show what a model is looking at, and therefore why it acts. Jain and Wallace argued in 2019 that across many language tasks the weights often fail to agree with gradient-based measures of importance, and that very different weight patterns can produce the same prediction. Wiegreffe and Pinter replied that the answer depends on what counts as an explanation, and proposed stricter tests. The debate stands as a caution: attention weights are part of a model's computation, not automatically an account of its reasons.

How to place the attention mechanism in AI history

The defensible reading is that the 2014 paper turned a general intuition, that a model should consult the relevant part of its input when it needs it, into a specific, trainable and reusable operation: score every element, normalize the scores, take the weighted sum. It did not originate the idea of soft addressing, which Graves had used a year earlier in a forward-only form, and it did not end the reign of recurrent networks, which it relied on. What it showed, with controlled numbers on a public benchmark, was that removing the single-vector bottleneck stopped neural translation from collapsing on long sentences, and that the learned weights lined up with sensible word alignments without any alignment supervision.

Three threads lead outward from the event. Backward, to the encoder-decoder models and the length analysis that defined the problem, and to Graves's soft window that supplied a precedent. Sideways, to the same-month sequence-to-sequence paper that pursued the fixed-vector design to its limit. Forward, to captioning, speech, Luong's simplifications, production translation, and finally the Transformer, which kept the weighted sum and discarded the recurrence around it. Much of what is called attention today differs in its details from the additive scoring of 2014. The operation it introduced, and the habit of letting a network decide for itself where to look, are what endured.

References

• Bahdanau, Cho and Bengio (2014). Neural Machine Translation by Jointly Learning to Align and Translate.
• Cho et al. (2014). On the Properties of Neural Machine Translation: Encoder-Decoder Approaches.
• Pouget-Abadie et al. (2014). Overcoming the Curse of Sentence Length for Neural Machine Translation using Automatic Segmentation.
• Sutskever, Vinyals and Le (2014). Sequence to Sequence Learning with Neural Networks.
• Graves (2013). Generating Sequences With Recurrent Neural Networks.
• Mnih et al. (2014). Recurrent Models of Visual Attention.
• Graves, Wayne and Danihelka (2014). Neural Turing Machines.
• Schlag, Irie and Schmidhuber (2021). Linear Transformers Are Secretly Fast Weight Programmers.
• Turing Post (2024). Who came up with the term attention?.
• Xu et al. (2015). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.
• Chorowski et al. (2015). Attention-Based Models for Speech Recognition.
• Luong, Pham and Manning (2015). Effective Approaches to Attention-based Neural Machine Translation.
• Sennrich, Haddow and Birch (2016). Edinburgh Neural Machine Translation Systems for WMT 16.
• Wu et al. (2016). Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.
• Le and Schuster, Google Research (2016). A Neural Network for Machine Translation, at Production Scale.
• Olah and Carter, Distill (2016). Attention and Augmented Recurrent Neural Networks.
• Cheng, Dong and Lapata (2016). Long Short-Term Memory-Networks for Machine Reading.
• Parikh et al. (2016). A Decomposable Attention Model for Natural Language Inference.
• Vaswani et al. (2017). Attention Is All You Need.
• ICLR (2025). Announcing the Test of Time Award Winners from ICLR 2015.
• Jean et al. (2014). On Using Very Large Target Vocabulary for Neural Machine Translation.
• Jain and Wallace (2019). Attention is not Explanation.
• Wiegreffe and Pinter (2019). Attention is not not Explanation.