BERT reads both ways

The event and the narrow claim

On 11 October 2018, Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova of Google AI Language posted a paper describing BERT, short for Bidirectional Encoder Representations from Transformers. Its central claim was specific and testable: a language representation could be pre-trained to condition jointly on both left and right context at every layer, rather than reading a sentence in a single direction, and this deeply bidirectional pre-training would improve a wide range of language-understanding tasks after only light fine-tuning. The paper reported new state-of-the-art results on eleven natural-language-processing tasks, including pushing the GLUE benchmark score to 80.5%, a 7.7 point absolute gain over the prior best. Those are task-specific evaluation numbers under stated conditions, not a measurement of general intelligence.

The claim landed hard. Google released the code and pre-trained weights within weeks, the work went on to win Best Long Paper at NAACL 2019, and BERT became one of the most cited machine-learning papers of its era. But citations and awards are downstream of a modest, precise result. The useful historical question is not whether BERT invented pre-training or bidirectionality, because neither is true. It is which single constraint BERT removed, which tools it inherited, and which later systems depended on it. Separating that narrow documented result from the broad judgment of influence is what keeps a timeline honest.

The constraint: models that could read only one way

By 2018 the dominant recipe for transferring language knowledge was to train a model to predict the next token, then reuse it. That objective is inherently unidirectional: to predict a word without cheating, the model may look only at the words before it. OpenAI's Generative Pre-Training work used exactly this left-to-right Transformer decoder, and it worked well. But a left-to-right model builds each word's representation from earlier context alone, which is a poor fit for tasks where the meaning of a word depends on what comes after it: deciding what a pronoun refers to, tagging a word's role in a sentence, or judging whether one sentence entails another.

The leading attempt to capture both directions was ELMo, from the Allen Institute for AI and the University of Washington. ELMo trained a forward language model and a separate backward one, then concatenated their hidden states. That is a shallow combination of two independently trained directions rather than a single network that attends to both sides at once. BERT's argument was that the shallow combination left value on the table, and that genuinely joint conditioning, where every layer sees the whole sentence, would do measurably better. The obstacle was mechanical: in a multi-layer bidirectional model trained to predict the next word, each word can indirectly see itself through the stacked layers, which makes the prediction trivial. Removing that leak is the problem BERT had to solve.

From static vectors to contextual meaning

The deeper predecessor is the shift from static word vectors to contextual ones. word2vec, from Mikolov and colleagues at Google in 2013, and GloVe, from Pennington, Socher and Manning at Stanford in 2014, each assigned every word a single fixed vector learned from co-occurrence statistics. Those vectors captured real regularities, famously that vector arithmetic could move from "king" toward "queen," but they collapsed every sense of a word into one point: the "bank" of a river and the "bank" that holds money received identical representations regardless of the sentence.

Contextual representations broke that constraint by making a word's vector a function of the sentence it appears in. ELMo demonstrated the payoff; BERT extended it and deepened it. In Google's own explanation, published when the code was released, BERT "represents 'bank' using both its previous and next context," so the same string yields different vectors in different sentences and even in different clauses of the same sentence. The open-sourcing announcement frames this as the concrete difference from ELMo and GPT. The move from static to contextual embeddings is the through-line that connects word2vec to BERT, and it is why a single fixed lookup table gave way to a network that recomputes meaning on every read.

Masked language modeling and the cloze inheritance

To condition on both directions in one model without letting a word see itself, BERT borrowed an idea far older than deep learning: the cloze task, introduced by Wilson Taylor in a 1953 readability study in which readers filled in deleted words from the surrounding text. BERT's masked language model does the same to the network. During pre-training it randomly selects 15% of the WordPiece tokens in each sequence and asks the model to predict them from the full remaining context on both sides. Because the target words are hidden from the input, the leakage problem disappears and every layer is free to attend in both directions at once.

There is a subtlety the paper handles carefully. If masked positions were always replaced with a special [MASK] token, the model would specialize in reconstructing that token, yet [MASK] never appears when the model is later fine-tuned on real text. So of the 15% chosen tokens, 80% are replaced with [MASK], 10% with a random word, and 10% left unchanged. Because the model cannot tell in advance which tokens were altered, it is forced to build a rich representation of every position rather than only the obviously masked ones. BERT paired this with a second objective, next-sentence prediction, in which the model judges whether one segment genuinely follows another, using special [CLS] and [SEP] marker tokens and learned segment embeddings so it can reason over sentence pairs for tasks such as question answering and inference.

The encoder architecture and the pre-train, then fine-tune recipe

BERT is not a new neural architecture. It is a Transformer encoder, the bidirectional half of the design from Vaswani and colleagues' 2017 paper, stacked and scaled. The paper released two sizes: BERT-BASE with 12 layers, a hidden size of 768, 12 attention heads and 110 million parameters, deliberately matched to GPT for a fair comparison, and BERT-LARGE with 24 layers, hidden size 1024, 16 heads and 340 million parameters. Both used a 30,000-token WordPiece vocabulary, a subword scheme that keeps common words whole while breaking rare ones into pieces so the model never meets a truly unknown token, and both accepted sequences up to 512 tokens. Pre-training ran on the BooksCorpus (800 million words) plus English Wikipedia (2,500 million words), and BERT-LARGE took four days on 16 Cloud TPUs.

The workflow that made BERT practical was the two-stage recipe popularized for text by Howard and Ruder's ULMFiT earlier in 2018: pre-train once on unlabeled text, then fine-tune the whole network on a target task with only a small task-specific output layer added on top. The same pre-trained weights served classification, tagging, span extraction and ranking; the practitioner changed the output head and the training data, not the model. Because the expensive learning happens once, adapting BERT to a new labeled dataset took hours on a single GPU rather than days, and that low marginal cost is a large part of why the field adopted it so quickly.

The evidence the benchmarks actually showed

The paper's evidence was a broad sweep of standard benchmarks reported under stated conditions rather than a single headline. Beyond the 80.5% GLUE score, BERT-LARGE reached 86.7% accuracy on the MultiNLI natural-language-inference corpus, a 4.6 point absolute improvement. On the Stanford Question Answering Dataset, BERT reached a Test F1 of 93.2 on SQuAD v1.1 and 83.1 on the harder SQuAD v2.0, which adds unanswerable questions, gains of 1.5 and 5.1 points respectively over the previous best systems. The breadth mattered as much as any one figure: the same pre-trained model, lightly adapted, topped tasks that had previously each demanded a bespoke architecture.

Two cautions belong with these numbers. First, the SQuAD v1.1 result exceeded that benchmark's human-annotator baseline of 91.2 F1, which Google's blog highlighted, but this means BERT matched the answer spans humans produced on this particular dataset, not that it understood reading in general. Second, the largest single gains came on tasks with the least labeled data, exactly where a strong pre-trained prior helps most, which says as much about the scarcity of supervised examples as about the model. A careful reader treats the sweep as evidence for one proposition, that joint bidirectional pre-training transfers well across tasks, and resists reading it as evidence of comprehension.

The competing bets of 2018 and 2019

BERT did not arrive in an empty field, and its rivals embodied different bets about the same problem. GPT bet on a left-to-right decoder that both understands and generates, a natural fit for open-ended text production but weaker on tasks needing future context. ELMo bet on feature extraction: it produced contextual vectors fed into separate task-specific models rather than fine-tuning one network end to end. ULMFiT bet on fine-tuning but used a recurrent LSTM language model rather than a Transformer. BERT combined the two most productive of these choices, the Transformer backbone and full fine-tuning, and added the bidirectional objective the others lacked.

The most pointed response came the following year from XLNet, by Yang and colleagues at Carnegie Mellon and Google. XLNet accepted BERT's goal of bidirectional context but attacked two weaknesses in how it got there: the [MASK] token creates a mismatch between pre-training and fine-tuning, and masked-language modeling predicts each masked token independently, ignoring dependencies among the masked words themselves. Its answer was a permutation-based autoregressive objective that sees both sides without ever introducing a mask, and it briefly overtook BERT on many benchmarks. The exchange sharpened the field's understanding of exactly what BERT's masking cost. Even so, the encoder-plus-fine-tuning template BERT set is the one that stuck, and the deeper split it created still organizes the field: encoder-only models became the default for understanding tasks (classification, tagging, retrieval, ranking), while decoder-only models in the GPT lineage became the default for generation. Both descend from the same 2017 architecture and differ mainly in the direction of attention and the pre-training objective.

What probing revealed about what BERT learned

Because BERT was a fixed, downloadable network, researchers could open it up, and a whole subfield grew around asking what its representations actually encode. The survey A Primer in BERTology collects the findings, and two are especially concrete. Tenney and colleagues showed in "BERT Rediscovers the Classical NLP Pipeline" that the network appears to reconstruct the traditional stages of language analysis in order across its layers, with part-of-speech information emerging in lower layers, then parsing, named entities, semantic roles and finally coreference higher up, and that it can revise an earlier decision once later context disambiguates it. Clark and colleagues, in "What Does BERT Look At?", found individual attention heads that track interpretable syntactic relations with surprising accuracy, such as a head that links verbs to their direct objects and another that connects pronouns to their antecedents.

These results are genuinely interesting and also easy to overclaim. They show that useful linguistic structure is recoverable from BERT's internals, not that the model reasons the way a linguist does or that any single head "is" the grammar. The honest reading is narrower: masked pre-training on enough text pushes the network to encode features that align with categories humans find meaningful, which helps explain why so little task-specific machinery had to be bolted on top.

The downstream lineage and the real deployment

BERT's influence is documented rather than merely asserted, because a series of later papers explicitly built on it and named what they changed. RoBERTa, from Facebook AI in 2019, kept BERT's architecture but trained longer on more data, used larger batches, and dropped next-sentence prediction, showing that BERT had been undertrained: an unusual case of a follow-up proving that the original's recipe, not its design, was the limiting factor. ELECTRA replaced masked prediction with a more sample-efficient objective that learns from every token by detecting which ones a small generator had swapped in. DistilBERT compressed the model through knowledge distillation to roughly 40% fewer parameters and 60% faster inference while retaining about 97% of its performance, and Sentence-BERT restructured it so that semantic search over a large corpus, which with vanilla BERT could take tens of hours, dropped to seconds. Together these turned BERT from a single model into a family spanning research and production.

The most concrete real-world deployment came in October 2019, when Google announced it was using BERT in Search, calling it the biggest leap forward in five years and applying it to about one in ten English queries in the United States. A research artifact from one year earlier was, within a year, shaping how a large fraction of the world's searches were interpreted, and BERT-derived encoders remain workhorses in retrieval, ranking and classification systems well beyond that.

The honest limits

BERT's constraints are as instructive as its strengths. It is not an autoregressive generator: because it is trained to fill in masked positions rather than to continue text, it cannot fluently produce long passages, and it is the wrong tool for a chat assistant. The masked objective also carries the pre-training-versus-fine-tuning mismatch that XLNet named, which the 80/10/10 scheme mitigates but does not remove, and it is sample-inefficient, since only the 15% of masked positions produce a learning signal on each pass, a cost ELECTRA was designed to recover. Next-sentence prediction, presented as helpful in the original paper, was later shown by RoBERTa to add little or even hurt, a reminder that a plausible auxiliary objective needs empirical confirmation rather than a good story.

More broadly, strong benchmark transfer does not imply understanding. High GLUE and SQuAD scores measure performance on specific datasets that can contain exploitable statistical shortcuts, and a model can rank answer spans well while lacking robust reasoning; several of the benchmarks BERT topped were later shown to be partly solvable through surface cues. The narrow claim BERT actually established, that joint bidirectional pre-training improves fine-tuned performance across many tasks, is well supported by the evidence in the paper. Claims about comprehension in a human sense are not, and the original authors were careful not to make them.

Placing BERT in the arc of AI history

BERT is best understood as the event that made the pre-train-then-fine-tune paradigm the default for language understanding, and as the moment the encoder branch of the Transformer family found its purpose. It inherited the Transformer encoder from 2017, the contextual-embedding idea from ELMo and word2vec, the fine-tuning workflow from ULMFiT, and the cloze objective from 1950s readability research. Its own contribution was to fuse these into a single deeply bidirectional model and to prove the combination on a wide benchmark sweep, not to originate any one of the ingredients.

A reader can navigate outward from BERT along two lines. One follows the encoders it spawned: RoBERTa, ELECTRA, DistilBERT, Sentence-BERT and the multilingual variants that power search, retrieval and classification today. The other follows the decoder branch it ran alongside, the GPT lineage that scaled generation into conversational models. For a few years around 2019, BERT and its descendants were the most-cited and most-deployed models in natural language processing. The later rise of large generative models did not erase that: it reflects that understanding and generation are different problems, and BERT decisively answered the first while the field was still learning to scale the second. Read this way, BERT is neither the beginning of modern NLP nor a revolution that changed everything, but the hinge on which one well-defined problem turned from open to largely solved.

References

• Devlin, Chang, Lee and Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
• Vaswani et al. (2017). Attention Is All You Need.
• Peters et al. (2018). Deep Contextualized Word Representations.
• Radford et al. (2018). Improving Language Understanding by Generative Pre-Training.
• Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space.
• Pennington, Socher and Manning (2014). GloVe: Global Vectors for Word Representation.
• Taylor (1953). Cloze Procedure: A New Tool for Measuring Readability.
• Howard and Ruder (2018). Universal Language Model Fine-tuning for Text Classification.
• Rajpurkar et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text.
• Wang et al. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.
• Williams, Nangia and Bowman (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.
• Devlin and Chang (2018). Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing.
• Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach.
• Clark et al. (2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.
• Rogers, Kovaleva and Rumshisky (2020). A Primer in BERTology: What We Know About How BERT Works.
• Nayak (2019). Understanding searches better than ever before.
• Devlin, Chang, Lee and Toutanova (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
• Yang, Dai, Yang, Carbonell, Salakhutdinov and Le (2019). XLNet: Generalized Autoregressive Pretraining for Language Understanding.
• Tenney, Das and Pavlick (2019). BERT Rediscovers the Classical NLP Pipeline.
• Clark, Khandelwal, Levy and Manning (2019). What Does BERT Look At? An Analysis of BERT's Attention.
• Sanh, Debut, Chaumond and Wolf (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.
• Reimers and Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.