AlphaZero: Mastery through self-play

The event and the narrow claim

On 5 December 2017, a team led by David Silver at DeepMind posted a preprint describing AlphaZero, a single reinforcement-learning algorithm that learned to play chess, shogi (Japanese chess), and Go to a very high standard starting from nothing but the rules of each game. The preprint reports that the same method, with the same network architecture and essentially the same hyperparameters, reached a level that defeated the strongest existing program in all three games. A peer-reviewed version, with expanded matches and analysis, appeared the following year in Science.

The documented claim is narrower than the folklore around it, and the two are worth separating. AlphaZero did not "solve" chess, and it did not study human games. It learned by playing against itself, given only the legal moves and the eventual win, draw, or loss of a finished game, and it did so with one recipe applied to three games that had, until then, been attacked with three different toolkits. The broad historical judgment (that this was a step toward more general learning) rests on that generality across games, not on any single match score. A useful reading of the event keeps the specific result and the sweeping interpretation on separate shelves.

From Deep Blue to AlphaGo: the knowledge that had to be built in

The systems AlphaZero displaced had been engineered around deep human knowledge. IBM's Deep Blue, which beat Garry Kasparov in 1997, searched enormous game trees with alpha-beta pruning and a hand-tuned evaluation function whose terms were designed with grandmaster help and specialised chess hardware. The retrospective account by Campbell, Hoane, and Hsu describes an architecture purpose-built for one game. Its descendants, the open-source engine Stockfish in chess and Elmo in shogi, kept the same shape: a fast search over millions of positions per second, steered by evaluation heuristics that humans wrote and refined, plus opening books and endgame tables.

Go resisted that formula. Its branching factor is far larger and good positional evaluation is notoriously hard to hand-code, which is why Monte Carlo tree search, surveyed thoroughly by Browne and colleagues, became the dominant approach there instead of alpha-beta. DeepMind's AlphaGo, which beat Lee Sedol in 2016, combined that search with neural networks trained first on a database of human expert moves and then improved by self-play. It was a hybrid: learning plus search plus a substantial amount of human data and Go-specific design.

The constraint AlphaZero removed: generality across a class of games

The immediate predecessor was AlphaGo Zero, published in Nature in October 2017, which removed the human game data. It learned Go from scratch by self-play, used a single neural network rather than separate policy and value networks, and dropped the random rollouts that earlier versions had used to estimate a position's value. But AlphaGo Zero was still a Go program. It exploited Go's board symmetries to multiply its training data eightfold, and it optimised for the probability of winning in a game that has no draws.

AlphaZero's contribution was to generalise that self-play recipe from one game to a class of games. The same algorithm learned chess, shogi, and Go without game-specific engineering: it did not assume board symmetry (chess and shogi have none), and it optimised the expected outcome so that draws, which matter enormously in chess and shogi, were handled correctly. The only per-game differences the authors report are the input and output encodings dictated by each game's rules and a scaling of the exploration noise to the typical number of legal moves. Self-play learning itself was not new. Gerald Tesauro's TD-Gammon had reached near-expert backgammon in the early 1990s by training a neural network through temporal-difference learning on its own games. What AlphaZero showed was that a single modern recipe could reach the top of three very different perfect-information games at once.

How the algorithm actually works: one network, guided search, self-play

At the centre is a single deep neural network that takes a board position and outputs two things: a policy, a probability distribution over moves, and a value, a scalar estimate of the expected outcome from that position. That network does not choose moves on its own. It guides a Monte Carlo tree search that, for each move to be played, runs a fixed budget of simulations (800 in the reported experiments). Each simulation walks down the tree, choosing at every node the move that best balances the network's policy prior against the accumulated value estimates of what has been tried, until it reaches a position the search has not expanded before. There the network evaluates the position, and that estimate is propagated back up the branch. Crucially, and unlike the original AlphaGo, the AlphaZero search uses no random rollouts to the end of the game; it trusts the value network.

The learning loop is a form of policy iteration. The search, because it looks ahead, produces a better move distribution than the raw network policy does. AlphaZero plays complete games against itself using that search, records for each position the search's move distribution and the game's eventual result, and then trains the network so its policy moves toward the search distribution and its value moves toward the actual outcome. A stronger network makes a stronger search, which generates better training data, which makes a stronger network. The move-selection rule inside the tree is a variant of the "upper confidence bound applied to trees" idea from the Monte Carlo tree search literature, weighting each candidate by the network's prior and by how promising it has looked so far, so that exploration and exploitation stay balanced within the fixed simulation budget. AlphaGo Zero had used a gating step, promoting a new network only after it beat the current best in a tournament; AlphaZero drops that and simply maintains one network updated continuously. Training ran for 700,000 mini-batch updates from random initial weights, using roughly 5,000 first-generation tensor processing units to generate self-play games and a much smaller pool of second-generation units to train the network.

The numbers, and the conditions behind them

The headline results come from matches against the reigning computer champions. In the Science version, AlphaZero played Stockfish 1,000 games from the standard starting position and, as summarised in DeepMind's own write-up, won 155, drew 839, and lost only 6. Against Elmo in shogi it won 91.2 percent of games, and against a strong version of AlphaGo Zero in Go it won 61 percent. The earlier preprint had reported a first round of 100-game matches in which AlphaZero did not lose a single game to Stockfish, winning 28 and drawing 72. These are task-specific match outcomes against particular opponents under particular settings, not measurements of general ability, and they should be read that way.

The training curve is part of the claim. Starting from random play, AlphaZero surpassed Stockfish in chess after about four hours of training on that hardware, reached Elmo's level in shogi within roughly two hours, and overtook AlphaGo Zero in Go within a day or so of wall-clock time. The point was less the exact hours than that a from-scratch learner, given only the rules, could climb past decades of accumulated engineering in a single game within a training run.

Searching a thousand times less

The most analytically interesting number is not the score but the search rate. Because the neural network concentrates the search on plausible lines, AlphaZero examined far fewer positions than the engines it beat. DeepMind's account reports that in chess it evaluated roughly 60 thousand positions per second, against roughly 60 million for Stockfish, about a thousandfold difference, and a similar gap held in shogi against Elmo. This reframes what the network is doing. It is not out-calculating a classical engine; it is out-selecting it, spending its far smaller search budget on the moves that matter. That is a concrete demonstration that a learned evaluation and a learned sense of which moves to consider can substitute for brute-force breadth, at least in these games. It is also the property that made the result striking to chess players, who observed a positional, sometimes sacrificial style that looked less like a calculator and more like a strong intuition backed by verification.

The match-conditions debate

The comparison drew legitimate criticism, and it is part of an honest account. Reviewers and the computer-chess community objected that the 2017 preprint had run Stockfish under conditions that did not reflect competitive practice: a fixed one-minute-per-move control rather than a full game clock, a modest hash-table size, no opening book, and hardware choices that some argued handicapped the classical engine. The debate matters because a match result is only as strong as the fairness of the setup, and the earlier alpha-beta versus neural-search comparison had genuine confounds.

The peer-reviewed Science paper addressed several of these points. It ran additional matches across a range of time controls, gave Stockfish a large hash and more resources, and tested play from a wide set of standard human opening positions and from the balanced opening suite used in the Top Chess Engine Championship. AlphaZero continued to win the great majority of games under those variations. The honest summary is that the first announcement overstated the cleanliness of the comparison, and the follow-up work made it considerably more defensible without overturning the conclusion. Both facts belong in the record.

What it did not remove: the limits of a perfect simulator

AlphaZero's setting is narrow in ways that bound how far the result generalises. It applies to two-player, turn-based, perfect-information games that are deterministic and have known rules and a clear terminal reward. The self-play loop depends on a perfect simulator: to plan, the search must be able to apply the rules and roll positions forward exactly. That excludes imperfect-information games such as poker, environments with hidden state or randomness that the agent cannot model, and, more importantly, the open world, where the rules are not given, outcomes are noisy or delayed, and the objective itself is hard to specify. As the grounding for this event notes, mastering games with known rules and clean outcomes does not touch the harder problems of defining good objectives or exploring safely when mistakes are costly.

Two further caveats deserve stating. First, "one algorithm" means one recipe, not one trained agent: a separate network was trained for each game, so AlphaZero is general across a class of tasks rather than a single system that does everything. Second, the achievement is compute-intensive; the reported training used thousands of specialised accelerators, which is a resource few could match at the time. These are not reasons to discount the result, but they define its scope. A related point is that the impressive style attributed to AlphaZero, the willingness to sacrifice material for long-term positional pressure, is a qualitative human interpretation of its games rather than a measured quantity, and it should be reported as such: a description of how it played, not a claim about what it understood.

The descendants: MuZero, AlphaTensor, and open reproductions

The clearest measure of influence is the work that relaxed AlphaZero's assumptions in turn. MuZero, from largely the same team in 2019, removed the requirement for a perfect simulator by learning a model of the environment's dynamics, predicting only the reward, policy, and value that planning needs rather than the full rules. It matched AlphaZero in chess, shogi, and Go without being told the rules, and, importantly, reached state-of-the-art results on the visually complex Atari benchmark, a domain where model-based planning had previously struggled. If AlphaZero generalised AlphaGo Zero from one game to a class of games, MuZero generalised AlphaZero from games with given rules to environments whose rules must be learned.

The template also moved beyond play. AlphaTensor, published in Nature in 2022, cast the discovery of faster matrix-multiplication algorithms as a single-player game and used an AlphaZero-style search-and-learn agent to find new, provably correct algorithms, including one that improved on a result standing since 1969 for certain matrix sizes. A follow-up, AlphaDev, applied the same framing to discover faster low-level sorting routines that were later merged into a widely used standard library, again treating a computational task as a game to be searched. In the open-source world, the community project Leela Chess Zero reproduced the self-play approach and reached elite strength, and classical engines absorbed the lesson too: Stockfish later adopted an efficiently updatable neural network for evaluation, blending learned assessment with its alpha-beta search. The idea did not stay in a lab.

How to place AlphaZero in AI history

The safest placement follows three questions the timeline can be read through. Which earlier idea did it depend on? Self-play learning with a neural evaluator (TD-Gammon), Monte Carlo tree search, and the deep networks and self-play loop of the AlphaGo line, none of which AlphaZero invented. Which assumption did it remove? The need for game-specific human knowledge, shown by one recipe mastering three different games from the rules alone. Which limitation did it leave for others? The dependence on a known, perfect simulator and a clean reward, taken up by MuZero, and the confinement to closed games, only partly loosened by later work.

Read that way, AlphaZero is a genuine turning point in game-playing AI and in the study of planning combined with learning, and a vivid demonstration that learned search can outperform decades of hand-built domain expertise while examining a thousand times fewer positions. It is not evidence of general intelligence, and it did not make everything that followed inevitable. Its importance is that it collapsed three specialised engineering traditions into one learning procedure, and in doing so set the next research agenda: keep the self-play-and-search engine, and remove, one at a time, the assumptions that still tied it to a hand-drawn board.

References

• Silver et al. (2017). Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm.
• Silver et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.
• DeepMind (2018). AlphaZero: Shedding new light on chess, shogi, and Go.
• Silver et al. (2016). Mastering the game of Go with deep neural networks and tree search.
• Silver et al. (2017). Mastering the game of Go without human knowledge.
• Schrittwieser et al. (2020). Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model.
• Tesauro (1995). Temporal Difference Learning and TD-Gammon.
• Browne et al. (2012). A Survey of Monte Carlo Tree Search Methods.
• Campbell, Hoane and Hsu (2002). Deep Blue.
• Fawzi et al. (2022). Discovering faster matrix multiplication algorithms with reinforcement learning.