Chinchilla scaling laws: Scale the data, not just the model

The event and the compute-optimal claim

On 29 March 2022, a team led by Jordan Hoffmann at DeepMind posted a paper arguing that the largest language models of the moment were being trained the wrong way. The claim was narrow and empirical: for a fixed amount of training computation, the field had been building models that were too big and feeding them too little data. The corrective was a model called Chinchilla, a 70-billion-parameter network trained on roughly 1.4 trillion tokens, which used the same training compute as DeepMind's earlier 280-billion-parameter Gopher yet scored higher across a broad battery of downstream tasks. The compute-optimal training paper reports these results and the reasoning behind them.

It is worth stating the documented claim precisely before reaching for its significance. Chinchilla is not evidence that smaller models are always better, nor that data matters more than parameters in some absolute sense. The claim is conditional: given a compute budget measured in floating-point operations, there is a balance between parameter count and training-token count that minimizes the model's loss, and that balance had been badly misjudged. Chinchilla is the demonstration that operating at the corrected balance produces a better model at equal cost.

The scaling law it set out to revise

The paper it answered was published two years earlier by a team at OpenAI. Kaplan and colleagues' scaling-laws study established power-law relationships between a model's test loss and three quantities: parameter count, dataset size, and compute. Its practical recommendation, absorbed quickly across the field, was that when a lab acquired more compute it should spend most of it on a larger model and comparatively little on more data. In their estimate, model size should grow much faster than the number of training tokens as compute rose, so the counsel of the day was to train very large models and stop well short of using all available data.

That advice shaped a generation of systems. If the return on parameters dominates, the rational move is to push parameter counts upward, which is exactly what the frontier labs did through 2020 and 2021. Chinchilla's contribution begins as a disagreement with this specific quantitative conclusion, not with the idea of scaling laws itself. Hoffmann and colleagues accepted the framework and re-ran the measurement more carefully, and got a materially different answer about the exponents.

Three ways to draw the same frontier

The strength of the Chinchilla argument is that it estimated the optimal frontier three independent ways and found them to agree. The first approach fixed a set of model sizes and, for each, traced the training loss as a function of tokens processed, then read off, at each compute level, which model size had reached the lowest loss. The second, the IsoFLOP method, held total compute fixed at nine different budgets and varied model size within each budget, producing a set of U-shaped curves whose minima mark the best model size for that budget. The third fit a parametric function of the form loss equals an irreducible term plus one term that decays with parameters and one that decays with tokens, then solved for the compute-optimal allocation analytically.

All three pointed to roughly equal scaling: as compute grows, parameters and training tokens should grow at approximately the same rate, each scaling as about the square root of compute. Concretely, doubling the compute budget should mean training a model about 1.4 times larger on about 1.4 times more data, rather than pouring the increment almost entirely into size. To generate these estimates the authors trained more than 400 models spanning 70 million to over 16 billion parameters on 5 billion to 500 billion tokens, a deliberately dense sweep intended to pin the exponents down rather than infer them from a handful of runs.

Chinchilla versus Gopher, the controlled comparison

The headline experiment was designed to be legible. DeepMind had already trained Gopher, a 280-billion-parameter model trained on about 300 billion tokens, and knew its training compute exactly. Chinchilla was trained on that same compute budget, but the compute-optimal recipe reallocated it: quarter the parameters to 70 billion, and multiply the data by roughly a factor of four to about 1.4 trillion tokens. Because the two models cost the same to train and were built by the same lab on largely the same data pipeline, the comparison isolates the allocation decision from almost everything else.

Chinchilla won broadly. It outperformed Gopher, and also the larger GPT-3 at 175 billion parameters, Jurassic-1 at 178 billion, and Megatron-Turing NLG at 530 billion, across language modelling, reading comprehension, and question-answering suites. On the Massive Multitask Language Understanding benchmark, Chinchilla reached an average accuracy of 67.5 percent, an improvement of more than seven percentage points over Gopher. These are task-specific evaluation results on defined benchmarks, not measures of general intelligence, and the honest way to read them is as evidence that a smaller, better-fed model recovered capability that an undertrained larger model had left on the table.

There was a practical dividend the paper emphasized. Because Chinchilla has a quarter of Gopher's parameters, it is substantially cheaper to fine-tune and to run for inference. A better model that also costs less to serve is an unusual combination, and it made the result immediately attractive to anyone who had to deploy, not just publish, a language model.

Why the earlier law pointed the other way

A careful history has to ask why two competent groups measuring the same thing disagreed. Part of the answer is methodological. The Chinchilla authors noted that the shape of the learning-rate schedule matters: a cosine schedule should be set so that it decays to its minimum at the end of training, matched to the actual number of training steps. When a schedule tuned for a long run is applied to a shorter one, the intermediate loss values used to fit a scaling law are distorted, and the fitted exponents shift toward favouring larger models. The earlier work's setup did not vary the schedule with token count in this way, which biased its estimate of how much data a given model should see.

This reconciliation is more instructive than a simple "one was right, one was wrong." Both studies were doing real measurement inside a shared framework; the divergence came from training details that look minor and turn out to govern the answer, and which detail mattered most is still debated: a 2024 reanalysis by Pearce and Song traces much of the gap to Kaplan counting only non-embedding parameters at small scale. It is a recurring lesson in the scaling literature: the conclusions of an empirical law are only as trustworthy as the experimental hygiene of the runs it is fit to, and a small unmodeled confound can move a headline exponent.

The models of its era were undertrained

Chinchilla arrived into a moment defined by size. GPT-3 at 175 billion parameters was trained on roughly 300 billion tokens; Megatron-Turing NLG at 530 billion parameters saw a similar token count. By the compute-optimal ratio Chinchilla implied, all of these were data-starved, carrying far more parameters than their training data could properly exercise. The pattern continued even after Chinchilla's preprint: PaLM, announced by Google in April 2022, was a 540-billion-parameter model trained on about 780 billion tokens, well under two tokens per parameter, when the Chinchilla recipe called for around twenty.

The competing approach of the era, then, was not a rival architecture but a rival philosophy of allocation: keep making the model bigger, on the belief that scale itself was the dominant lever. Chinchilla did not refute that scale helps; it refuted the specific claim that parameters should soak up almost all incremental compute. Read against its contemporaries, the paper is best understood as a course correction inside the scaling program rather than a departure from it.

The "twenty tokens per parameter" rule and its downstream influence

Chinchilla's most portable legacy is a number. The ratio of its training tokens to its parameters, about 1.4 trillion divided by 70 billion, is roughly twenty, and "twenty tokens per parameter" became a widely quoted rule of thumb for compute-optimal training. The heuristic is a simplification of a law that actually depends on the compute budget, but it gave practitioners a quick sanity check on whether a planned run was balanced.

Its influence on open models was immediate and explicit. Meta's LLaMA models, released in early 2023, cited Chinchilla directly and pushed its logic further in a specific direction. LLaMA trained relatively small models on far more data than the compute-optimal frontier would strictly recommend: the 7-billion-parameter model was trained on a full trillion tokens. The stated reason is an important refinement of Chinchilla's frame. Chinchilla optimizes for training cost, but most of a deployed model's lifetime cost is inference. If a smaller model trained past the compute-optimal point can match a larger model's quality, it is cheaper to serve for the rest of its life, which justifies "overtraining" a small model on purpose. LLaMA showed that model quality kept improving well beyond the Chinchilla token count, so the smaller-and-longer strategy had real headroom.

Training-optimal is not deployment-optimal

That inference argument was formalized later. Sardana and colleagues' inference-aware analysis modified the Chinchilla laws to account for the compute spent serving a model, not just training it. Their conclusion is that a model expected to field heavy demand, on the order of a billion requests, should be smaller and trained longer than Chinchilla-optimal, because the training saving from a larger model is repaid many times over by the inference cost of running it. Across 47 models they found that quality continued to improve as the token-to-parameter ratio was pushed to extreme values, up to around 10,000 tokens per parameter, far beyond Chinchilla's twenty.

This is the cleanest statement of Chinchilla's principal limitation, and the paper itself flagged it. Compute-optimal training minimizes the loss achievable for a fixed training budget; it does not minimize the total lifetime cost of building and operating a model. Once serving is priced in, the optimum moves toward smaller models trained on more data, which is precisely the regime the open-model ecosystem gravitated to. Chinchilla answered the training-budget question correctly and left the deployment-budget question for its successors.

The replication that questioned the third method

Chinchilla's estimates have themselves been scrutinized. In 2024, Besiroglu and colleagues attempted to replicate the paper's third estimation procedure, the parametric fit of the loss function. Reconstructing the data from the published plots, they found that the reported parameters were inconsistent with the paper's own first two methods, did not actually fit the extracted data well, and carried confidence intervals so narrow that producing them would have required on the order of 600,000 experiments, whereas the authors likely ran fewer than 500. When the replicators re-derived the parametric law themselves, their result came out compatible with the first two methods after all.

The right way to read this is with care, because it cuts two ways. The critique lands on the reported uncertainty and the specific fit of one of the three methods, and it identifies a genuine error in the published numbers for that method. It does not overturn Chinchilla's central conclusion: the other two estimation approaches stood, and the replicators' corrected fit agreed with them. The episode is a reminder that a landmark result can contain a flawed sub-analysis while its main finding survives independent re-examination, and that the healthiest scaling laws are the ones that have been re-measured by people who did not produce them.

How to place Chinchilla in AI history

Chinchilla is best filed as a correction that changed practice, not as the birth of an idea. Scaling laws predate it, the transformer architecture it used predates it, and the models it beat were built on the same foundations. What it supplied was a more accurate map of a tradeoff everyone was already navigating, backed by an unusually dense and legible set of experiments and one decisive controlled comparison. The narrow documented claim is that, at fixed training compute, parameters and tokens should scale together; the broader historical judgment is that this reframed how the field spent its budgets and helped legitimize the smaller, longer-trained models that dominate open deployment today.

A reader placing the event on a timeline can navigate outward with three questions. Which prior result did it revise, and why did that prior get the exponents wrong? Which later projects adopted its ratio, and which deliberately departed from it once inference cost entered the picture? And which parts of its own analysis were later found wanting, without unseating its core result? Answered together, those questions show Chinchilla as a load-bearing measurement in the scaling era: influential precisely because it was quantitative, falsifiable, and quickly put to the test by both the models that followed it and the researchers who re-checked its math.

References

• Hoffmann et al. (2022). Training Compute-Optimal Large Language Models.
• Pearce and Song (2024). Reconciling Kaplan and Chinchilla Scaling Laws.
• Kaplan et al. (2020). Scaling Laws for Neural Language Models.
• Rae et al. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher.
• Brown et al. (2020). Language Models are Few-Shot Learners.
• Smith et al. (2022). Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model.
• Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways.
• Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models.
• Sardana et al. (2024). Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws.
• Besiroglu et al. (2024). Chinchilla Scaling: A replication attempt.