ChessBench: A New Era of Chess

This article was published in Spanish in the Cuaderno de Cultura Cientifica (CCC) under the title “ChessBench: una nueva era del ajedrez“, under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. The article was originally published in The Conversation (original article). Except for the English translation, no changes were made to the original article. Published with permission of CCC.

Authors: Francisco Herrera Triguero, Professor of Computer Science and Artificial Intelligence at the University of Granada, with the collaboration of José Luis Flórez—a mathematician and PhD in Economics, as well as an entrepreneur and executive with 30 years of experience in the fields of machine learning and AI..

Recently, a team of researchers from Google DeepMind updated the paper titled Amortized Planning with Large-Scale Transformers: A Case Study on Chess, which had previously gone relatively unnoticed. It presents an artificial intelligence (AI) system called ChessBench—based on large-scale Transformers (neural networks)—that plays chess.

Without Memory

To date, all chess programs have been based on complex processes of searching for the best move in the tree that combines next moves and countermoves, often with search depths greater than 20 future moves. This, together with the application of functions that allow them to evaluate the advantage or disadvantage of each possible decision, allowed them to find the best move.

While the fundamental thing for all AIs until now was memory, ChessBench marks the beginning of a new era, with machines trained in planning tasks in which memory is useless.

Leela Chess Zero and Stockfish are, today, the best chess engines, exponents of excellence in the game achieved with the first approach.

Both use neural network-based learning in different ways: Leela Chess Zero uses reinforcement learning through self-play, and Stockfish integrates a neural network to improve position evaluation and decision making.

But Google DeepMind’s ChessBench has just started a new model that can push all limits. As we noted above, it is an AI system based on Transformers (such as BERT and GPT) whose most advanced version has up to 270 million parameters.

Training

Para su entrenamiento y aprendizaje se han utilizado 10 millones de partidas de ajedrez extraídas de Lichess, plataforma de código abierto para jugar al ajedrez en línea a la que diariamente se conectan miles de usuarios humanos en todo el mundo. A partir de esas partidas se generan más de 15 000 millones de anotaciones evaluando cada posición y su mejor acción (movimiento). Esta evaluación la lleva a cabo Stockfish 16, cuyo motor de búsqueda determina el mejor movimiento posible en una posición concreta.

What Makes ChessBench Special

Training has enabled the system to learn to predict action values ​​for new positions with considerable accuracy. This represents a non-trivial generalization: it achieves this without explicitly searching for subsequent moves and counter-moves.

What makes the result remarkable is having an AI system capable of playing chess at such a high level of quality.

ChessBench plays without a search process to determine the best move, relying solely on predicting the next one. It is the dream of any chess player: to look at a position and recognize the best next move based on the configuration of the pieces.

This raises a recurring question in the debate surrounding large-scale Transformers and Large Language Models (LLMs)—such as the one powering ChatGPT: Are they merely vast memorization engines, or are they capable of actual reasoning?

Chess as a Testing Ground

Chess requires a combination of strategy, tactics, and foresight. These are key elements in the processes of planning, reasoning, and decision-making.

The model used by ChessBench has 270 million parameters (2.7×10¹¹), whereas the number of possible chess moves is vastly larger—approximately 10¹²⁰. In this game, memorization is of little use on a large scale; there is an immense variety of possible moves, compounded by the complexity of the strategies involved.

Chess offers a way to accurately evaluate the ability of AI models to generalize and adapt to new situations.

ChessBench has learned to recognize deep, complex patterns within the game of chess. This capability allows it to replicate its success even in situations not encountered during its training.

Its ability to identify and apply winning strategies in previously unseen positions reveals a level of reasoning that goes beyond simple memorization; it cannot be attributed merely to the encoding of the next best move within its parameter space.

ChessBench Breaks Records.

Unlike other disciplines, chess has an objective method for estimating a player’s strength: the Elo rating system. In the Blitz format (rapid chess, with games lasting between 3 and 5 minutes per player), only 15 people worldwide surpass the Elo rating of 2,895 achieved by ChessBench on the Lichess platform. The Top 200 Blitz list shows that the cutoff for the top 100 is an Elo rating of 2,784.

The Elo rating achieved by ChessBench represents a Grandmaster level. It solves challenging chess positions. Furthermore, it achieves this impressive rating by playing against human opponents on Lichess.

Surpassing AlphaZero

ChessBench is still far from competing with the AlphaZero AI, whose level surpasses that of the world’s best human player. It performs in-depth analysis combined with reinforcement learning, enabling it to learn by playing against itself.

Currently, ChessBench is an excellent approximation of Stockfish’s search-based algorithm. Perfect distillation based solely on position observation remains unreachable for an AI system that relies on position analysis without search processes.

Generalization in Learning

This makes ChessBench a suitable starting point for future research. For instance, it would be interesting to see how it performs in Fischer Chess—where the major pieces are randomly placed on the back rank. This setup involves a change in starting positions that demands a high capacity for generalization, as well as an understanding of the function of each piece and its legal moves.

A preliminary study was recently conducted using GPT-4 to swap knights and bishops in the starting position, revealing the limitations LLMs face in these novel scenarios. The study concludes that there is still significant progress to be made in these so-called counterfactual scenarios.

The Processes of Imagination

Judea Pearl, an AI researcher and Turing Award winner, links counterfactual scenarios—situations that have not occurred in the universe currently observable through human inquiry—to the human process of imagining. These are imagined worlds that could have been.

Large-scale Transformers are not solely focused on language; there is a growing number of applications across various fields. LLMs are already being integrated with intelligent robotics, playing a significant role.

However, LLM architecture alone will be insufficient to achieve a general AI that reaches the human cognitive level. Yet, the view held by those who claim these models rely merely on memorization—pejoratively labeling them “stochastic parrots”—does not seem accurate either. This is an exaggerated criticism.

The creative process is no longer exclusive to humans. Capabilities such as planning, continuous learning combined with the processing of robotic sensory inputs, mathematical reasoning, and language processing—all features of large-scale Transformers—will impact every aspect of the human experience.

It will be interesting to discover and understand the extent of ChessBench’s imaginative capabilities.