Teaching Chess with Language Models

I recently built a Chess Learning platform as part of my goal to achieve an Expert level FIDE ranking. The tool (open-source and local-first, meant for personal use) is straightforward to use: sync your games from Chess.com, run them through Stockfish, and identify patterns in your mistakes. In working on this project I came across an interesting problem: getting an LLM to explain why a particular move in chess is good or bad.

Modern engines (Stockfish, AlphaZero, Leela, etc.) have essentially solved chess evaluation and can tell you the ~best move in any position. As a semi-lazy but decent human player (~2000 rating on Chess.com), I find engines to be useful, but not quite as useful as a conceptual explanation. Chess books offer some of the guidance I seek but they're unpersonalized and much harder to apply without dedicated study. I'm also not particularly interested in finding a coach for logistical reasons.

Engines are fantastic for seeing whether a given move is favorable or not. One can explore alternatives and an opponent's responses, but this approach seems to lack the intuition-building which I suspect is more important for development at this level of play. So for a student seeking to understand what strategy is appropriate or not in a given situation, I tend to find engines rather lacking.

Perhaps naively, I believed that LLMs should be able to handle this task of applying and explaining chess concepts. But after doing some digging, I found broad agreement that LLMs cannot play chess well. Frontier models routinely make illegal moves and fail to correctly assess even basic positions. Benchmarks from 2025 found o3 making illegal moves in 87.5% of games and o4-mini in 94.3%, worse than GPT-3.5-turbo-instruct on raw legality. Larger "thinking" budgets don't appear to make much of a difference. I suppose the easiest way to beat an LLM at chess is to play delaying moves until the model simply becomes incoherent.

Research by Adam Karvonen shows that a 50M-parameter GPT trained on human games (from Lichess) can reach ~1500 Elo and develop a coherent internal board state, so perhaps these limitations are not entirely architectural.

But using an LLM to play chess strikes me as the wrong framing to begin with, given that evaluation is already solved. The interesting question then isn't whether LLMs can compete with purpose-built engines (they can't) but whether they can do something that engines are incapable of: explaining positions semantically.

The most obvious solution is to pair a language model with a specialist engine. In the simplest form, provide the engine's top line to the model and have it explain. This fails from being too narrow in that the model receives the what but not the how. It may know that e4 is +0.3 and what subsequent moves are to follow, but the model lacks visibility into all of the branches that the engine explored to reach that conclusion. It also has no sense of what state each continuation leads to, and no framework for translating an eval score into a piece of instruction which a student can learn from.

From my own testing, providing context via the most common chess notations (e.g. PGN, FEN) appears to be ineffective, although one might expect that these verbose / explicit game representations are easier to follow. In fact they still require the model to do significant inferential work to reconstruct what's on the board (e.g. whether pieces are pinned, a mating net is being set up) aside from reasoning about the position. This is a different failure mode than the model doesn't know chess.

A chess-trained model can learn to build a board representation on its own, whereas an off-the-shelf model needs us to hand it one. In my own exploration, these techniques noticeably improved how accurately LLMs reasoned about positions:

Each of these either translates the position into a language-native form, or offloads the part the model can't do reliably, before asking the model to reason about it.

The most relevant work I came across is Tang et al. (2026), which formalizes a similar approach into a training pipeline for their own model. Their core insight is something like: Stockfish knows the correct answer but can't articulate it; LLMs can articulate but can't evaluate. Their solution was to train a model with a technique called Feigned Discovery Prompting (reasoning as if the answer is unknown, rather than post-hoc justification). Feigned discovery forces the model to reason about its options until it arrives at the optimal move, resulting in a thorough and grounded chain of thought (rather than reasoning backwards from the solution and rationalizing incorrectly).

Their results were impressive but constrained only to puzzles which have a single correct continuation and required 40K annotated samples. Real game positions are significantly more complex, as there may not be an immediate tactical objective to pursue and the optimal strategy might only be marginally better than the alternatives. Still, an interesting approach!