Scaling laws for neural language models: Bigger models, predictably better

An ideal gas law for language models

On 23 January 2020, ten researchers at OpenAI posted a paper with a plain title and an unusually practical aim. Scaling Laws for Neural Language Models, led by Jared Kaplan, who listed both Johns Hopkins University and OpenAI as affiliations, and Sam McCandlish, introduced no new architecture and claimed no benchmark record. It asked how the test loss of a Transformer language model depends on three things a lab can actually buy: parameters, training tokens and training compute. The answer was that when none of the three is starved by the others, loss falls as a smooth power law in each one, with the compute trend holding across eight orders of magnitude, the parameter trend across six and the data trend across just over two. The paper itself compared these relations to the ideal gas law: macroscopic regularities that hold largely regardless of the microscopic details.

The documented claim is narrower than its reputation. Every number in the paper describes cross-entropy loss, measured in nats per token, on a held-out slice of WebText2, an enlarged version of the Reddit-linked web text that had trained GPT-2: 20.3 million documents and about 23 billion tokens, drawn from links posted through October 2018. The models ran from 768 to 1.5 billion parameters, not counting embeddings, and the datasets from 22 million to 23 billion tokens. The paper did not show that lower loss means a more capable system, and it said so. The broader historical judgment, that it turned model size into a quantity labs could plan and forecast, rests on what OpenAI and others did with it afterwards.

Learning curves before the laws

Researchers had seen performance climb steadily with data long before 2020. In 2001, Michele Banko and Eric Brill at Microsoft Research trained four learning methods on a language task that sounds trivial, choosing between easily confused words such as "then" and "than", using a corpus of one billion words, three orders of magnitude larger than anything previously applied to the problem. In their ACL paper on scaling to very large corpora the learning curves looked log-linear all the way out, and the authors suggested the field reconsider how much it spent on algorithms rather than on building corpora.

The modern version began at Baidu. Hestness and colleagues measured learning curves in 2017 for machine translation, language modeling, image classification and speech recognition and found power laws in every domain. For recurrent language models the exponents were small, between about 0.06 and 0.09, and better architectures shifted the curves downward without changing their slope. Pablo Villalobos's scaling-law literature review for Epoch AI dates the modern study of scaling laws to that paper. In 2019, Rosenfeld and colleagues fitted a single function of model size and data size across vision and language tasks, and Kaplan's paper acknowledges that this work used a similar form. The OpenAI group had its own groundwork too: McCandlish, Kaplan and Dario Amodei had shown in 2018 that a measurable gradient noise scale predicts the largest useful batch size, and that tool became the paper's way of accounting for compute fairly across runs.

What was new in January 2020 was the combination: Transformer language models at larger scale, all three resources varied together, a formula for overfitting, and, above all, a prescription for spending a fixed compute budget. One contrast shows why the advice was distinctive. Hestness found that the best model size grew more slowly than the data, roughly as data to the power 0.7, which implies data must grow faster than parameters. Kaplan found the reverse.

What the power laws say

The core result fits in three exponents. With ample data, loss scales with non-embedding parameters at an exponent of about 0.076; with an ample model, it scales with dataset size at about 0.095; and with an optimally sized model, it scales with compute at about 0.050. Concretely, doubling the parameter count multiplies the loss by about 0.95, a cut of roughly 5 percent. Because every doubling buys the same fractional gain, progress plotted on logarithmic axes is a straight line, which is what makes it forecastable, but each further step of improvement costs more in absolute resources than the last.

Two accounting choices made the lines clean. Counting only non-embedding parameters collapsed models of different depths onto one trend, and compute was estimated as about six operations per parameter per training token, covering the forward and backward passes. With size held fixed, shape barely mattered: the ratio of width to depth could vary by a factor of 40 with only a small change in loss, and a six-layer model reached a loss within 3 percent of a 48-layer model of similar size.

A joint formula governed overfitting. The penalty for too little data depended on parameters raised to the power 0.74 divided by tokens, so a model eight times larger needed only about five times more data to avoid it. Larger models were also more sample efficient, reaching a given loss after fewer steps and fewer tokens. Loss on other text, such as books, Wikipedia and Common Crawl, improved in parallel with a roughly constant offset. And in a comparison that echoed the Transformer's own case for itself, recurrent LSTMs matched Transformers on early tokens but stopped improving after fewer than 100 tokens of context, while Transformers kept using the whole window (see Attention is all you need).

The advice: big models, modest data, early stopping

The paper's most consequential section turned these curves into a spending rule. On the compute-efficient frontier, the optimal model size grew as compute to the power 0.73, the batch size as compute to the power 0.24, and the number of serial training steps barely at all, at 0.03. Data therefore grew only as compute to roughly the power 0.27. In plain terms, each tenfold increase in compute should go mostly into a model about five times larger, trained on about twice as much data, and training should stop well short of convergence. The authors condensed it into a line, big models may be more important than big data, that Pearce and Song later identified as the conclusion the field drew.

The paper was candid that its own extrapolations eventually contradict each other. Because compute-efficient training adds data so slowly, the compute law at some point predicts a lower loss than the data law allows. The authors placed that crossing near 10,000 petaflop/s-days of compute, around a trillion parameters and a trillion tokens, at about 1.7 nats per token, while warning that each figure could be off by an order of magnitude. They treated it as the point by which the laws must break down, and conjectured that it might roughly estimate the entropy of natural text. Their caveats were just as direct: no theory explained the laws, learning rates had not been tuned for short runs, and the small-data regime fitted poorly.

From fit to plan: GPT-3

The advice was put to work within four months. The GPT-3 paper, whose author list included all ten authors of the scaling-laws paper, trained eight models from 125 million to 175 billion parameters, all on 300 billion tokens. A figure caption states that, based on the scaling-laws analysis, the team trained much larger models on many fewer tokens than was typical: the 3-billion-parameter model was almost ten times the size of RoBERTa-Large yet used roughly the same 50 petaflop/s-days of compute. Model shapes were chosen for hardware efficiency because the laws said shape hardly mattered, and batch sizes were guided by the gradient noise scale. The smooth compute curve then held for two further orders of magnitude with only small deviations (see GPT-3 scales up).

The framework spread quickly. In October 2020, Henighan, Kaplan and colleagues found the same kind of law, a power law plus a constant, for image, video, image-text and mathematics models, with compute-optimal size exponents nearly the same across domains. The industry followed the allocation as well as the form. DeepMind's compute-optimal training paper observed in 2022 that, following Kaplan and the GPT-3 setup, many of the largest models had been trained on about 300 billion tokens, among them Jurassic at 178 billion parameters and Gopher at 280 billion. An independent group led by Tomer Porian later described the law as influential in determining the size of GPT-3 and several subsequent models.

The exponent that did not survive

The allocation rule was the part that broke. Hoffmann and colleagues read Kaplan as implying that ten times more compute should buy a model 5.5 times larger and only 1.8 times more tokens. From more than 400 trained models they concluded instead that parameters and tokens should grow in equal proportion, and their 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens with Gopher's compute, beat the four-times-larger Gopher ( Training Compute-Optimal Large Language Models; see Chinchilla). Their suggested explanation was a learning-rate schedule that had not been matched to each run's length.

Two independent 2024 studies dug into why the exponents differed, and neither settled on that explanation. Tim Pearce and Jinyeop Song argued in Transactions on Machine Learning Research that much of the gap comes from counting non-embedding rather than total parameters, combined with the small model sizes Kaplan studied. Simulating Chinchilla's own fitted behavior over Kaplan's range produced a local exponent of about 0.78 when parameters were counted Kaplan's way, against 0.51 when counted in total. Porian and colleagues reproduced Kaplan's result in more than 900 runs, obtaining an exponent near 0.71, then moved it to about 0.50 by counting the compute of the final output layer, shortening a warmup that was too long for small models, and tuning hyperparameters for each model size. They found that careful learning-rate decay was not essential, contrary to Hoffmann's conjecture.

The two accounts emphasize different mechanisms, but they agree on the lesson. The power-law form survived scrutiny; the allocation exponent was sensitive to accounting conventions and to how small models were tuned, which is roughly what Kaplan's own caveat about untuned learning rates had warned.

Forecasting a model before training it

What outlasted the spending rule was the method of predicting a large model from small ones. OpenAI's GPT-4 Technical Report states that the team predicted GPT-4's final loss on an internal code dataset from models trained with at most 10,000 times less compute, making the prediction shortly after the main run began, and predicted pass rates on a subset of the HumanEval coding problems from models with at most 1,000 times less (see GPT-4 raises the bar). The same report concedes that certain capabilities remained hard to predict.

Meta's account of its 2024 flagship is more explicit still. The Llama 3 Herd of Models describes scaling-law experiments between 6×10¹⁸ and 10²² floating-point operations on models from 40 million to 16 billion parameters, citing both Kaplan and Hoffmann. Extrapolated to the planned budget of 3.8×10²⁵ operations, the fit recommended a 402-billion-parameter model trained on 16.55 trillion tokens, and Meta built a 405-billion-parameter one (see Llama 3.1). Because existing laws predict next-token loss rather than benchmark accuracy, Meta added a second step linking loss to task scores.

The effect on incentives was noticed early, including by the authors. By 2022 Kaplan, McCandlish, Henighan, Brown and Dario Amodei were writing from Anthropic, and their paper on predictability and surprise argued that many institutions had begun building large models in response to the predictability scaling laws offered, while specific capabilities and harms remained hard to anticipate.

What the loss curve does not tell you

The original paper flagged its central gap: whether falling loss translates into better performance on tasks people care about, adding that smooth quantitative change can hide qualitative jumps. That gap became a research program. Wei and colleagues catalogued emergent abilities that appear in large models but cannot be predicted by extrapolating smaller ones. Schaeffer, Miranda and Koyejo countered that many such jumps reflect the researcher's choice of a nonlinear or discontinuous metric, such as exact string match, and become smooth under continuous ones. As Quanta Magazine reported, the dispute did not close: critics noted that no one can say in advance which metrics will show abrupt gains, and Wei argued that for arithmetic only the right answer counts.

Data is a second limit. The laws assume a steady supply of fresh tokens, and the paper's own crossing point shows the assumption straining. Studying that constraint directly, Muennighoff and colleagues found that repeating data for up to 4 epochs costs almost nothing, but that with more repetition the value of added compute decays toward zero. Theory is a third. Kaplan's group admitted it had none and hoped for a statistical mechanics beneath the observed thermodynamics; later work by Bahri, Dyer, Kaplan, Lee and Sharma, with Kaplan now writing from the physics department at Johns Hopkins and published in PNAS in 2024, proposed distinct scaling regimes in which models effectively resolve a smooth data manifold. Finally, a law describes one recipe. Hestness's finding that better architectures shift the curve rather than its slope means an algorithmic advance can move the whole line, and Villalobos's review found no simple universal law for transfer between very different tasks.

Compute becomes a governance variable

Once capability looked like a function of training compute, compute looked like something to regulate. Sastry and colleagues argued in 2024 that compute is an unusually tractable lever, detectable, excludable, quantifiable and made by a concentrated supply chain, and they described scaling laws as instrumental to predicting performance while warning that they do not reliably predict individual downstream tasks. The European Union's Artificial Intelligence Act adopted the premise in Article 51: a general-purpose model trained with more than 10²⁵ floating-point operations is presumed to have high-impact capabilities, and the recitals call cumulative training compute one of the relevant approximations for capability (see The EU AI Act enters into force). The Act does not cite Kaplan, and the Commission may amend the threshold as algorithms and hardware improve, so the connection is one of shared premise rather than direct lineage. Meta's 3.8×10²⁵ budget for Llama 3 was nearly four times that line.

The question Kaplan posed, how to divide a fixed compute budget, has since moved beyond pretraining. In a comparison matched for floating-point operations, Snell and colleagues showed in 2024 that spending inference-time compute well can let a smaller model beat one 14 times larger on problems the smaller model can sometimes solve, the trade-off that reasoning models made central (see Thinking before answering).

How to place the scaling laws in AI history

The 2020 paper did not discover that more data and bigger models help, and its central piece of advice was wrong in a way the field corrected within two years. Its lasting contribution was to turn a vague faith in scale into measured exponents and a procedure: fit small models, extrapolate, then decide how large a model to build and how long to train it. That procedure shaped GPT-3, was refined by Chinchilla, remained standard practice in the GPT-4 and Llama 3 reports, and its premise now sits inside European law.

Three questions help locate it. Which earlier results did it scale up rather than invent? Banko and Brill's curves, Hestness's power laws and the group's own batch-size work. Which part failed? The allocation exponent, for reasons that independent groups traced to accounting and tuning choices. And which parts remain open? Whether loss keeps translating into capability, how far fresh data can stretch, and how compute should be split between training and inference. The paper is best read as the moment language-model progress acquired an engineering forecast, with all the confidence and all the blind spots a forecast brings.

References

• Kaplan et al., OpenAI (2020). Scaling Laws for Neural Language Models.
• Banko and Brill, Microsoft Research (2001). Scaling to Very Very Large Corpora for Natural Language Disambiguation.
• Hestness et al., Baidu Research (2017). Deep Learning Scaling is Predictable, Empirically.
• Pablo Villalobos, Epoch AI (2023). Scaling laws literature review.
• Rosenfeld et al. (2019). A Constructive Prediction of the Generalization Error Across Scales.
• McCandlish, Kaplan and Amodei, OpenAI (2018). An Empirical Model of Large-Batch Training.
• Brown et al., OpenAI (2020). Language Models are Few-Shot Learners.
• Henighan et al., OpenAI (2020). Scaling Laws for Autoregressive Generative Modeling.
• Hoffmann et al., DeepMind (2022). Training Compute-Optimal Large Language Models.
• Pearce and Song (2024). Reconciling Kaplan and Chinchilla Scaling Laws.
• Porian et al. (2024). Resolving Discrepancies in Compute-Optimal Scaling of Language Models.
• OpenAI (2023). GPT-4 Technical Report.
• Llama Team, Meta (2024). The Llama 3 Herd of Models.
• Ganguli et al., Anthropic (2022). Predictability and Surprise in Large Generative Models.
• Wei et al. (2022). Emergent Abilities of Large Language Models.
• Schaeffer, Miranda and Koyejo (2023). Are Emergent Abilities of Large Language Models a Mirage?.
• Stephen Ornes, Quanta Magazine (2024). How Quickly Do Large Language Models Learn Unexpected Skills?.
• Muennighoff et al. (2023). Scaling Data-Constrained Language Models.
• Bahri et al. (2024). Explaining Neural Scaling Laws.
• Sastry et al. (2024). Computing Power and the Governance of Artificial Intelligence.
• European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence.
• Snell et al. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.