GPT-1: The first GPT

The event and the narrow claim

On 11 June 2018, OpenAI published a short blog post and a technical report describing a language system trained in two stages: an unsupervised pass over a large body of ordinary text, followed by a supervised adjustment for each target task. The post, Improving language understanding with unsupervised learning, and the paper, Improving Language Understanding by Generative Pre-Training by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, are the documents that fix this event in time. The label "GPT" arrived afterward: the paper speaks of "generative pre-training," and the model is called GPT-1 only in hindsight.

The claim itself was empirical and bounded. A single Transformer decoder, first trained to predict the next word and then fine-tuned with small task-specific changes, improved published results on nine of twelve language-understanding datasets. Those are scores on named benchmarks, not a percentage of human ability. The historical interest is not that one report invented modern AI, but that it demonstrated a reusable procedure: learn once, broadly, from unlabeled text, then adapt cheaply and repeatedly. Keeping that narrow demonstration separate from the sweeping product lineage that later carried the same three letters is the central task in placing the event correctly, because the two are routinely collapsed into a single story.

The constraint: labels were the scarce resource

The bottleneck the paper addressed was economic as much as technical. Supervised natural-language systems needed large sets of hand-labeled examples, and those sets were expensive, small and specific to one task. Unlabeled text, by contrast, was effectively unlimited. The obvious move was to learn from the abundant resource first, but earlier attempts to do so mostly produced word-level representations. Methods such as word2vec mapped each word to a vector learned from co-occurrence statistics, and global vectors such as GloVe shared the same ceiling: those vectors improved many systems, yet they assigned a single representation to a word regardless of the sentence around it, and they left the rest of each task model to be trained from scratch on scarce labels.

The open question was whether one could pre-train not just the input vectors but most of the model, and then transfer that whole network to a new task with minimal additional structure. Doing so raised two practical uncertainties the paper names directly: which training objective produces representations that transfer well, and how best to feed a pre-trained network into tasks whose inputs have different shapes (single sentences, pairs, triples). The first GPT is best read as a concrete, evaluated answer to those two questions, not as the first appearance of the idea of learning from unlabeled text.

The predecessors that made the recipe thinkable

The recipe drew on nearer precedents, and the paper is candid about them. Semi-supervised Sequence Learning by Andrew Dai and Quoc Le had already shown, in 2015, that pre-training a recurrent network as a language model or sequence autoencoder and then fine-tuning it improved downstream text classification: the two-stage shape of GPT is visible there. Learned in Translation produced contextual word vectors from a machine-translation encoder, showing that context-sensitive representations helped. Most directly, two 2018 systems pursued the same goal in parallel. Deep contextualized word representations, known as ELMo, derived contextual features from a bidirectional LSTM language model, and Universal Language Model Fine-tuning, or ULMFiT, fine-tuned an LSTM language model for classification.

What GPT changed was the combination. Instead of a recurrent network it used the Transformer decoder from Attention Is All You Need, whose self-attention gave the model a more direct path between distant tokens and allowed efficient parallel training. Instead of freezing features (ELMo's approach) it fine-tuned the entire pre-trained network. And instead of a task-specific architecture it reshaped each task's input into a single sequence the same network could read. None of the ingredients was new; the arrangement, and the demonstration that it worked across many tasks at once, was the contribution.

How the two-stage method actually works

Stage one is ordinary language modeling. The model, a twelve-layer Transformer decoder with masked multi-head self-attention (twelve attention heads, 768-dimensional hidden states, and 3072-dimensional feed-forward inner layers, roughly 117 million parameters in total), reads a window of text and is trained to maximize the likelihood of each token given the tokens before it. Because attention is masked to the left, the model only ever conditions on earlier positions, which makes it a coherent generative model of text but also means it never sees right-hand context during pre-training. Optimization used Adam with a warmup-then-cosine learning-rate schedule, a context of 512 tokens, learned position embeddings rather than the fixed sinusoids of the original Transformer, and GELU activations, run for 100 epochs over minibatches of 64 sequences.

Stage two is fine-tuning, and its clever part is that the same network handles very different tasks through what the authors call traversal-style input transformations. A sentence pair for entailment, a passage-question-answer triple for reading comprehension, or two sentences for similarity are each serialized into one ordered token sequence with special delimiter tokens, then passed through the pre-trained model to a small linear classification layer on top. Almost nothing task-specific is added. The authors also kept the language-modeling objective as an auxiliary loss during fine-tuning, reporting that it improved generalization and helped convergence. This is why one pre-trained model could be pointed at a dozen benchmarks without redesign.

The data and the tokens: BooksCorpus and byte-pair encoding

The choice of pre-training corpus was deliberate and consequential. GPT was trained on BooksCorpus, the roughly 7,000 unpublished books assembled for Aligning Books and Movies. The paper argues that books matter because they contain long stretches of contiguous prose, which lets the model learn to condition on long-range structure. That is contrasted, in the paper, with sentence-shuffled benchmarks like the one ELMo used, where long-range dependencies are destroyed by shuffling. The corpus was modest by later standards (on the order of hundreds of millions of words) but its continuity was the point, not its size.

On the input side, GPT represented text with byte-pair encoding, the subword scheme from Neural Machine Translation of Rare Words with Subword Units, using a vocabulary of about 40,000 merges. Subword tokenization matters because language has an open vocabulary: names, compounds and rare inflections cannot all be listed as whole words, and BPE lets the model compose them from smaller pieces while keeping frequent words intact. These choices, a continuous-text corpus and a subword vocabulary, are easy to overlook next to the architecture, but they are part of why the pre-training produced transferable representations rather than brittle memorized fragments.

The evidence: twelve datasets, nine records

The paper's argument rests on breadth. It evaluated the fine-tuned model on twelve datasets spanning four families of task: natural-language inference (including MultiNLI, QNLI, SNLI, SciTail and RTE), question answering and commonsense reasoning ( RACE and the Story Cloze Test), semantic similarity (MRPC, QQP and STS-B), and single-sentence classification (CoLA for linguistic acceptability and SST-2 for sentiment). On nine of the twelve it improved the previously published state of the art, and several gains were large for the time: the Story Cloze Test improved by 8.9 points, RACE by 5.7 points, and MultiNLI by 1.5 points. These are task-specific results under stated conditions, not one headline score. On CoLA, which tests whether a sentence is grammatically acceptable, the jump over prior systems was especially sharp, a sign that the pre-trained representations captured syntactic structure and not only topical word associations, and the community was beginning to aggregate exactly these tasks through the newly proposed GLUE benchmark.

Two secondary observations mattered as much as the leaderboard. First, the authors probed zero-shot behavior, using the pre-trained model on tasks without fine-tuning, and found that this performance rose steadily as pre-training progressed, suggesting the generative objective was quietly acquiring task-relevant structure. Second, ablations showed that transferring more layers helped, and that removing the auxiliary objective or replacing the Transformer with an LSTM hurt. The evidence pointed to a mechanism, not just a good score.

The rival design: ELMo's features and BERT's reply

GPT did not arrive into an empty field, and it did not stay ahead for long. Its closest contemporary, ELMo, took a feature-based route: it produced contextual embeddings from a bidirectional LSTM and fed them, frozen, into separately trained task models. GPT's bet was the opposite, that fine-tuning one pre-trained Transformer end to end would transfer better. The most pointed reply came four months later. BERT used a Transformer encoder trained with a masked-language-modeling objective, which let every token attend to both left and right context, and it explicitly framed GPT's left-to-right attention as a limitation for understanding tasks. BERT overtook GPT on much of GLUE.

This is a corrective to any triumphal reading of June 2018: for classification and inference benchmarks, the bidirectional encoder, not the decoder, briefly looked like the winning design. The decoder's advantage, open-ended generation, was not what those benchmarks measured. In the longer run both branches persisted: encoder models like BERT for understanding, decoder models like GPT for generation, and the field did not collapse to one. The rivalry is a reminder that the first GPT's benchmark lead was real but narrow and quickly contested, and that its lasting significance would come from a different property than the one the 2018 scores rewarded.

What the first GPT could not do

It is easy to project the later chat products backward onto this model, and worth resisting. The first GPT was not a conversational assistant. It did not follow instructions, it was not trained with human feedback, and it had no chat interface; it was a pre-trained network that still required a labeled dataset and a fine-tuning run to do anything useful on a given task. Its 117 million parameters and hundreds of millions of training words are tiny by the standards of the systems that carry the name today. Its context window was 512 tokens, and its left-to-right attention, an asset for generation, was a handicap on the understanding benchmarks BERT soon dominated.

There is also a naming caveat. Calling this "the first GPT" is a retrospective convenience: the paper does not brand itself GPT-1, and the three-letter acronym became load-bearing only once later releases turned it into a product family. Treating the 2018 report as the origin of an inevitable arc overstates what its authors claimed and what the evidence showed. What they demonstrated was a transfer-learning recipe that worked across a dozen benchmarks. Everything about scaling, emergent few-shot behavior, alignment and dialogue was later work, built on the recipe but not contained within it.

From a recipe to a lineage: GPT-2 and GPT-3

The recipe's real influence shows up in what reused it. In February 2019 OpenAI released GPT-2, the same decoder-only design scaled to 1.5 billion parameters and trained on a much larger web corpus, notable for producing fluent long-form text and for the decision to stage its release over concerns about misuse. In 2020, Language Models are Few-Shot Learners scaled the identical architecture to 175 billion parameters and reported that a large enough model could perform many tasks from a prompt containing a few examples, with no gradient updates at all. That few-shot behavior is the property that eventually made fine-tuning optional for many uses and set up the assistant products that followed.

The through-line is precisely the two-stage idea from 2018, held fixed while other ingredients changed: more parameters, more data, and a shift from per-task fine-tuning toward in-context prompting. The instruction-following and dialogue behavior that made the name famous with ChatGPT in late 2022 came from further steps, supervised instruction tuning and reinforcement learning from human feedback, layered on this foundation years later. This is the strongest case for the first GPT's importance and also its most honest framing: the architecture and the "pre-train, then adapt" discipline came from this paper and its Transformer parent, while the capabilities people now associate with the name came from repeatedly changing the scale and adaptation method around that stable core. Each of those steps was its own research result, not a corollary of the first.

Placing the first GPT in AI history

The defensible reading has two levels. The narrow, documented claim is specific: a Transformer decoder, generatively pre-trained on contiguous book text and lightly fine-tuned, set new results on nine of twelve NLU datasets in mid-2018, and did so with a nearly task-agnostic design. The broad historical judgment, that this launched the dominant paradigm of the following decade, is real but earned by later work, GPT-2, GPT-3 and their successors, that reused the recipe at larger scale. Both statements can be true at once, as long as they are not merged into a claim that one 2018 report contained everything that came after.

A reader navigating the timeline can hold three questions open. Which earlier idea did the recipe depend on? (The Transformer, subword tokenization, and the pre-train-then-fine-tune shape from Dai and Le.) Which later project reused the design while changing an ingredient? (GPT-2 and GPT-3 changed scale and adaptation; BERT changed the attention direction and briefly won.) And which limitation demanded new work? (Instruction-following, dialogue and alignment were absent and came later.) Answered honestly, those questions make the first GPT what it actually was: not the invention of intelligent machines, but a clean, well-evidenced demonstration that learning from unlabeled text first and adapting afterward is a powerful default, one the field has been building on ever since.

References

• Radford, Narasimhan, Salimans and Sutskever (2018). Improving Language Understanding by Generative Pre-Training.
• OpenAI (2018). Improving language understanding with unsupervised learning.
• Vaswani et al. (2017). Attention Is All You Need.
• Dai and Le (2015). Semi-supervised Sequence Learning.
• Peters et al. (2018). Deep contextualized word representations.
• Howard and Ruder (2018). Universal Language Model Fine-tuning for Text Classification.
• McCann et al. (2017). Learned in Translation: Contextualized Word Vectors.
• Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space.
• Zhu et al. (2015). Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books.
• Sennrich, Haddow and Birch (2015). Neural Machine Translation of Rare Words with Subword Units.
• Lai et al. (2017). RACE: Large-scale ReAding Comprehension Dataset From Examinations.
• Wang et al. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.
• Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
• Brown et al. (2020). Language Models are Few-Shot Learners.
• OpenAI (2019). Better Language Models and Their Implications.