Google Neural Machine Translation: Phrase tables give way to neural networks

The launch and the narrow claim

On 27 September 2016, Google announced that every machine translation from Chinese to English in the Google Translate mobile and web apps, about 18 million a day, was now produced by a single neural network. The announcement on the Google Research blog, written by Quoc Le and Mike Schuster of the Google Brain team, accompanied a 31-author technical report, Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, submitted to arXiv the previous day. The system was called GNMT. For one language pair, it replaced the phrase-based statistical engine that had run Google Translate since the service began.

The documented claim needs to be stated precisely, because it was quoted loosely almost at once. Using bilingual human raters who scored sampled sentences from Wikipedia and news sites, Google said GNMT reduced translation errors by 55 to 85 percent on several major language pairs compared with its own phrase-based production system. The technical report's abstract gives an average of 60 percent and specifies that the test material was a set of isolated simple sentences. Those numbers were Google's own measurements on Google's own test sets. Three dates also need keeping apart: the research announcement and the Chinese to English launch on 27 September, the switch of eight more languages in November, and a multilingual system that could translate between pairs it had never been trained on, described a week later. The larger historical judgment, that this was the moment neural sequence models became infrastructure used daily by hundreds of millions of people, rests on what followed rather than on the September figures alone.

Forty years of counting words

GNMT displaced an approach with a long and successful history of its own. In 1990, a group at IBM's Thomas J. Watson Research Center that included Peter Brown, Frederick Jelinek and Robert Mercer published A Statistical Approach to Machine Translation. They revived an idea Warren Weaver had floated in 1949 and argued that the real obstacles had been weak computers and too little machine-readable text. Their system split translation, by Bayes' theorem, into a language model that judged whether an output sentence was plausible English and a translation model that judged how well it matched the French. It learned both from about 100 million words of the Canadian parliamentary proceedings, the Hansards, which were published in French and English. The preliminary results were modest: 5 percent of decoded sentences were exact and 48 percent were judged reasonable.

The same group then set out five statistical models of increasing complexity for aligning the words of a sentence with the words of its translation, the family later known as the IBM models. A decade on, Philipp Koehn, Franz Och and Daniel Marcu showed in Statistical Phrase-Based Translation that translating short multiword phrases, learned heuristically from those word alignments, beat translating word by word across every language pair they examined. Och moved to Google, and in April 2006 he announced an online statistical system for Arabic and English, trained on billions of words and frankly described as better on news than on novels. By the service's tenth anniversary in April 2016, Google reported 103 languages, more than 500 million users and more than 100 billion words translated a day.

Phrase-based translation worked by decomposition. It broke a sentence into words and phrases, translated them largely independently, reordered them and scored the candidates with a language model, using a pipeline of separately built components and tuned feature weights. That design was robust and fast, but it had no single view of the whole sentence, which is why its output so often read as a string of locally correct fragments.

A challenger that was not yet ready for production

The neural alternative was itself partly a Google invention. In 2014, Ilya Sutskever, Oriol Vinyals and Quoc Le trained a deep LSTM network to read a sentence into a vector and generate its translation, scoring 34.8 BLEU on a standard English to French test against 33.3 for a phrase-based system (see sequence-to-sequence learning). In Montreal, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio added a mechanism that let the decoder look back over every source word rather than squeezing the sentence into one fixed vector (see the attention mechanism). The GNMT report itself lists the three weaknesses that still kept such systems out of products in 2016: slow training and inference, poor handling of rare words, and a habit of leaving parts of the input untranslated. By its account, the first neural system to beat a phrase-based one on the WMT 2014 English to French benchmark had done so by only 0.5 BLEU.

Academic evidence was nonetheless turning. In a careful study, Bentivogli and colleagues examined the IWSLT 2015 evaluation, where a neural system from Stanford had beaten established phrase-based systems on English to German by 5.3 BLEU. Using professional post-edits, they found that the neural output cut post-editing effort by 26 percent and word order errors by about half, though its quality fell faster as sentences grew longer. At the WMT 2016 shared task, Edinburgh's attentional encoder-decoder systems, which split rare words into subword units, were the best or tied-best constrained entries in 7 of the 8 translation directions they entered. By mid-2016 the open question was no longer whether neural translation could win a research contest. It was whether it could be trained on corpora that the GNMT team describes as two to three orders of magnitude larger than the public benchmarks, and then answer users quickly enough to sit inside a live product.

What GNMT engineered: depth, wordpieces and a cheaper decoder

The architecture described in the GNMT report is a sequence-to-sequence model with attention, built for scale. Its encoder stacks eight LSTM layers, the bottom one reading the sentence in both directions, and its decoder stacks eight more. Residual connections, which add each layer's input to its output, made the deep stacks trainable. One design choice reveals the constraint the team was working under: attention connects the bottom decoder layer to the top encoder layer, a compromise chosen so that each layer could run on its own GPU in parallel. Each model replica was split across eight GPUs, and the report says a basic English to French model took about six days on 96 NVIDIA K80 GPUs.

The second ingredient addressed rare words. Instead of whole words, GNMT reads and writes wordpieces, a data-driven set of subword units adapted from Mike Schuster and Kaisuke Nakajima's 2012 work on Japanese and Korean voice search. In the report's example, "Jet" becomes "_J" and "et" and "feud" becomes "_fe" and "ud", while common words stay whole. A shared vocabulary of 8,000 to 32,000 wordpieces let the model copy or transliterate names and handle words it had rarely seen, without the long sequences of a purely character-based model.

The third ingredient was the decoder. Plain beam search favors short outputs, so GNMT normalized scores by length and added a coverage penalty that rewarded candidates whose attention had touched every source word, a direct fix for dropped clauses. To meet latency targets, the team trained models to tolerate low-precision arithmetic and served them on Google's Tensor Processing Units. Decoding the WMT English to French development set took 384 seconds on one TPU against 1,322 seconds on CPUs, a 3.4 times speedup with no loss in BLEU. The CPUs were in turn faster than a GPU, because beam search shuttled data back and forth at every step. The team also tried reinforcement learning to optimize BLEU directly. It raised English to French BLEU by nearly a point, but human raters scored the outputs 4.46 and 4.44 out of 6 before and after, so it was left out of the production experiments.

On public benchmarks, single GNMT models averaged 38.95 BLEU on WMT 2014 English to French and 24.61 on English to German, and ensembles of eight reached 41.16 and 26.30. These were competitive with the best published results rather than far ahead of them. The decisive evidence Google offered came from its own data.

Reading the 60 percent

The production evaluation in the report asked raters fluent in both languages to score three translations of each source sentence side by side on a scale from 0 (nonsense) to 6 (perfect): the phrase-based production output, GNMT's output and a human translation. Each language pair used 500 sentences sampled from Wikipedia and news sites. For Chinese to English, the phrase-based system averaged 3.694, GNMT 4.263 and the human translations 4.636. For English to Spanish the figures were 4.885, 5.428 and 5.504.

The famous percentages come from those numbers in a specific way. The "relative improvement" is the share of the gap between the old system and the human translation that GNMT closed: 60 percent for Chinese to English, 87 percent for English to Spanish, 58 percent for English to Chinese. It is not a count of mistakes, and the ceiling is a human translation that itself scored only about 5. The authors were candid about the weaknesses of the yardstick. They note that raters did not always fully understand the sampled sentences, and in an example table they disagree with a rater who gave a perfect 6 to a French translation containing grammatical errors and a change of meaning. Testing on difficult cases and on inputs longer than single sentences was explicitly left for future work.

Google's announcement listed the system's failure modes itself: dropped words, mistranslated proper names and rare terms, and translation of each sentence in isolation without the context of the paragraph or page. Contemporary press tested the claims in kind. On 3 October 2016, NPR's All Things Considered ran a People's Daily commentary on the US election through the new system, and the English that came out was tangled where a professional translator's version was clear. The evaluation question did not go away. In 2018, Läubli, Sennrich and Volk cited GNMT's claim of approaching average bilingual human translators among a wave of parity claims, then showed, using a 2018 Chinese to English system that had prompted one such claim, that raters judging whole documents preferred human translation more strongly than raters judging isolated sentences. Sentence-level side-by-side scoring, the method behind GNMT's headline, is the setting most likely to flatter a machine.

From one pair to many, and to pairs never taught

The wider rollout came quickly. On 15 November 2016, Google Translate's product lead Barak Turovsky announced that neural translation now ran for eight languages to and from English: French, German, Spanish, Portuguese, Chinese, Japanese, Korean and Turkish, covering more than 35 percent of Translate queries, in Search, the app and the website. The same post said Translate was improving more in one leap than in the previous ten years combined, a product claim offered without a published metric, and it made the system available to businesses through the Google Cloud Translation API.

A week later Google described a further step. In Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation, Melvin Johnson and colleagues trained one GNMT model on many language pairs, changing nothing in the architecture except an artificial token at the start of each input naming the desired output language. One multilingual model of 255 million parameters handled 12 production language pairs that had previously needed 12 separate models totaling 3 billion, losing 5.6 percent of BLEU on average, a loss that shrank to 2.5 percent in a 650-million-parameter version. The surprise was zero-shot translation. A model trained only on Portuguese to English and English to Spanish could translate Portuguese to Spanish directly, scoring 21.62 BLEU, and 24.75 when trained in both directions on both pairs, against 30.91 for translating through English with two separate neural models. For unrelated languages the effect was weaker: Spanish to Japanese fell from 18.00 BLEU through English to 9.14 zero-shot.

The accompanying Google Research post presented visualizations in which sentences with the same meaning in different languages clustered together, and interpreted this, with appropriate hedging, as a sign of an interlingua inside the network. It also reported that multilingual models already served 10 of the 16 newly launched translation directions. Google was not alone. In October 2016, SYSTRAN described its own production neural engines with demonstrators for 12 languages and 32 language pairs, noting that several companies were announcing similar rollouts.

The honest limits

The strongest independent caution came from inside the statistical tradition. In Six Challenges for Neural Machine Translation, Philipp Koehn and Rebecca Knowles found in 2017 that neural systems lost more quality than phrase-based ones on text from unfamiliar domains, sometimes producing fluent sentences unrelated to the input, learned worse from small datasets, degraded on very long sentences, and could get worse rather than better as the beam was widened. They added that such systems were much harder to interpret. Fluency that hides inadequacy is a particular hazard for the reader of a free translation who cannot check the source.

Bias was a second limit. Prates, Avelar and Lamb built sentences such as "He/She is an engineer" in 12 gender-neutral languages, from Hungarian to Yoruba, and found that Google Translate chose male pronouns far more often than US labor statistics would predict, especially for science and engineering jobs. In December 2018 Google acknowledged that Translate had historically rendered the Turkish for "he/she is a doctor" as masculine and "he/she is a nurse" as feminine, and began offering both gendered translations for some queries.

Access was the third. GNMT itself, its weights and its production data were never released. The report described the method in detail, and in July 2017 Google published a TensorFlow tutorial that replicated key GNMT features, with an open baseline scoring 24.4 BLEU on English to German. Outside researchers could reproduce the recipe, but no one outside Google could audit the system that hundreds of millions of people actually used.

The road to the Transformer

GNMT's most consequential legacy may be the benchmark it set for its successor. In June 2017, eight researchers, most of them at Google Brain and Google Research and among them the GNMT co-author Łukasz Kaiser, published Attention Is All You Need, a paper about machine translation (see the Transformer). Its results table placed GNMT with reinforcement learning at 24.6 BLEU on English to German and its ensemble at 26.30; the Transformer reached 28.4 after 3.5 days on eight GPUs, for an estimated training cost equal to that of a single GNMT model and roughly an eighth of the GNMT ensemble's. GNMT's awkward bottom-to-top attention wiring had existed to win back parallelism from recurrence. The Transformer removed recurrence altogether. It still took GNMT's 32,000-unit word-piece vocabulary for English to French and its length penalty for beam search.

The wordpieces travelled further. When Google introduced BERT in 2018, its authors wrote that they used WordPiece embeddings, citing the GNMT report, with a 30,000-token vocabulary (see BERT). Inside Translate, GNMT's LSTMs lasted about four years. In June 2020 Google said it had replaced the original GNMT system with a hybrid of a Transformer encoder and an RNN decoder, as part of changes that raised BLEU by an average of 5 points over the previous GNMT models and by 7 points for the 50 lowest-resource languages. Google described that gain as comparable to the jump from phrase-based to neural translation four years earlier, and noted that its systems still lagged human performance in all but the most specific translation tasks.

How to place GNMT in AI history

GNMT did not invent neural machine translation. The encoder-decoder came from Sutskever, Vinyals and Le, attention from Bahdanau, Cho and Bengio, subword vocabularies from speech recognition and from Edinburgh, and the evidence that neural systems beat phrase-based ones was accumulating in public evaluations through 2015 and 2016. What GNMT demonstrated was that the method could be engineered into a production service: trained on corpora far larger than any benchmark, served within a consumer product's latency budget on custom hardware, and then extended to many languages with one shared model. In doing so it retired, in a few months, the statistical pipeline that a large part of the field, including Google, had refined since the IBM papers.

Its quality claims deserve the qualification its own authors gave them. The 55 to 85 percent figures are shares of a gap to human translation, measured by Google, on isolated sentences, with a rating scale the authors themselves found noisy. Its architecture was superseded within a year by a model built to remove the very constraint GNMT had worked around. What lasted was the demonstration that end-to-end neural networks could carry a mass-market language product, together with pieces of its engineering, wordpieces, length-normalized beam search and the target-language token for multilingual models, that passed into the Transformer era. Read that way, GNMT belongs between the research papers of 2014 and the Transformer of 2017 as the point at which neural language technology stopped being a laboratory result and became something that ordinary users relied on every day.

References

• Le and Schuster, Google Research (2016). A Neural Network for Machine Translation, at Production Scale.
• Wu et al. (2016). Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.
• Brown et al. (1990). A Statistical Approach to Machine Translation.
• Brown, Della Pietra, Della Pietra and Mercer (1993). The Mathematics of Statistical Machine Translation: Parameter Estimation.
• Koehn, Och and Marcu (2003). Statistical Phrase-Based Translation.
• Och, Google Research (2006). Statistical machine translation live.
• Turovsky, Google (2016). Ten years of Google Translate.
• Sutskever, Vinyals and Le (2014). Sequence to Sequence Learning with Neural Networks.
• Bahdanau, Cho and Bengio (2014). Neural Machine Translation by Jointly Learning to Align and Translate.
• Bentivogli, Bisazza, Cettolo and Federico (2016). Neural versus Phrase-Based Machine Translation Quality: a Case Study.
• Sennrich, Haddow and Birch (2016). Edinburgh Neural Machine Translation Systems for WMT 16.
• NPR (2016). Google Announces Improvements To Translation System.
• Läubli, Sennrich and Volk (2018). Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation.
• Turovsky, Google (2016). Found in translation: More accurate, fluent sentences in Google Translate.
• Johnson et al. (2016). Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation.
• Schuster, Johnson and Thorat, Google Research (2016). Zero-Shot Translation with Google's Multilingual Neural Machine Translation System.
• Crego et al. (2016). SYSTRAN's Pure Neural Machine Translation Systems.
• Koehn and Knowles (2017). Six Challenges for Neural Machine Translation.
• Prates, Avelar and Lamb (2018). Assessing Gender Bias in Machine Translation: A Case Study with Google Translate.
• Johnson, Google Research (2018). Providing Gender-Specific Translations in Google Translate.
• Luong and Brevdo, Google Research (2017). Building Your Own Neural Machine Translation System in TensorFlow.
• Vaswani et al. (2017). Attention Is All You Need.
• Devlin, Chang, Lee and Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
• Caswell and Liang, Google Research (2020). Recent Advances in Google Translate.