26AI Research

Reconnaissance Blind Chess Recovery

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1
0.8957
±0.076
26672.8M265k$42.849.9h
2GPT-5.6
0.8186
±0.059
24037.3M123k$24.227.6h
3Kimi K3
0.4300
±0.228
30881.9M325k$32.1414.3h
4Grok 4.6
0.3936
±0.215
41798.7M866k$110.0412.3h
5GLM-5.3
0.3729
±0.111
31189.0M330k$33.1814.4h
6Muse Spark 1.2
0.3464
±0.082
17638.5M470k$8.405.6h
7Gemini 3.7 Flash
0.2121
±0.164
430151.1M658k$23.435.8h
8DeepSeek V4 Flash Exp
0.0707
±0.076
382138.8M427k$2.5911.6h
9Inkling
0.0664
±0.080
852.8M26k$2.781.4h
10Qwen3.8-Max
0.0650
±0.047
35659.5M728k$19.7313.5h

Background

Reconnaissance Blind Chess (RBC) is an imperfect-information variant of chess: a player never sees the opponent's moves or the true board, only a private 3×3 sense result each turn and the outcomes of its own moves. Playing well requires maintaining a belief distribution over thousands of possible board states, choosing sense actions that maximize information, and moving under genuine uncertainty — all classic hard problems in game AI.

This task is deliberately RBC-adjacent rather than standard RBC: the submitted bot is blind, but its opponents are sighted chess engines that receive the true board every turn. The question is how much of Stockfish's playing strength can be recovered through the limited RBC observation and action channels — under a fast 3+3 Fischer clock that rules out brute-force belief enumeration.

Task

Implement make_bot(game_id: str) in /app/blind_bot.py, returning a reconchess.Player. The bot receives the ordinary RBC callbacks — sense results and its own move outcomes — but never the true board or the opponent's hidden move, while its engine opponents see everything.

  • Games use a 3+3 Fischer clock; every callback must stay below three seconds, and a callback failure or timeout attributed to the bot counts as a loss.
  • game_id is opaque: the bot must not depend on an encoded opponent, color, schedule position, or random seed.
  • Matches are offline, the captured /appworkspace is read-only during a match, and writable scratch is fresh for every game — no cross-game state is available.
  • Additional modules may be added under /app and imported from the factory.

What The Agent Gets

A CPU-only workspace with the bot stub at /app/blind_bot.py, interface documentation in /app/README.md, and a development match runner at /app/run_dev_matches.py that plays seeded games against sighted Stockfish at three strengths (--level easy targeting roughly 1320 Elo, medium roughly 1700, hard roughly 2200) and reports wins, losses, callback failures, timing, and replay paths. Preinstalled: Stockfish 15.1 at /usr/games/stockfish, Python 3.11 with reconchess 1.6.9, python-chess, and NumPy. The sealed evaluation policies and their configurations are root-only and never visible. The machine is offline; a sandbox-timer CLI reports the remaining budget.

Verification

A root-only verifier in a separate no-network container plays an official 140-game evaluation: 10 color-balanced games against each of 14 sealed policies — two sighted-Stockfish policy families across seven strength bands from 800 to 2000. The reward is the mean game score (win 1, draw 0.5, loss 0) over all 140 games. The mixture score is uncalibrated and is not a human or classical Elo rating.

  • The submission runs isolated as the unprivileged agentuser inside a required cgroup-v2 / PID-namespace containment with a read-only root, private tmpfs scratch, seccomp restrictions, and trusted turn timing enforced outside the bot's process.
  • The scorer accepts only a complete, security-marked official schedule and attests every harness source file, the policy manifest, and the Stockfish binary against pinned SHA-256 digests.
  • Entrant failures count as losses, and games missing when the 4.5-hour suite deadline hits score zero — but trusted-opponent, engine, or harness failures invalidate the evaluation rather than becoming free wins.
  • Opponent randomness is drawn from a private per-run entropy key, so policy behavior cannot be replayed or predicted from public seeds.

Environment

Base imageubuntu:22.04
Tools availablePython 3
Compute6 CPUs · 24 GB RAM
Time limit15h

References

  • Stockfish (GPL-3.0): Version 15.1 (retained Debian package), baked at /usr/games/stockfish. Used sighted — granted the true board each turn — in the two sealed policy families; the submission never receives the true board or policy source.
  • reconchess (BSD-3-Clause): The Reconnaissance Blind Chess game framework by Johns Hopkins University Applied Physics Laboratory: the Player interface the submission implements and the game loop the harness drives (version 1.6.9).
  • python-chess (GPL-3.0): Board representation, move generation, and the UCI bridge to Stockfish (version 1.999, by Niklas Fiekas).
  • uv (Apache-2.0 OR MIT): Version 0.11.7; installs the retained CPython 3.11.15 and the hash-locked Python dependency set during image construction. Supporting libraries (NumPy, tqdm, click, loguru, and others) come from PyPI at image build. The task harness, scoring logic, and instructions are first-party; Scorca was considered as a reference opponent but excluded because of its CC-BY-ND 4.0 license.