| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.9443 ±0.023 | 1,853 | 819.9M | 1.7M | $451.30 | 15.8h |
| 2 | GPT-5.6 | 0.6671 ±0.113 | 1,430 | 525.2M | 644k | $532.83 | 9.3h |
| 3 | GLM-5.3 | 0.3947 ±0.172 | 1,923 | 914.2M | 1.9M | $250.66 | 14.3h |
| 4 | Grok 4.6 | 0.3262 ±0.142 | 1,319 | 342.9M | 2.2M | $346.56 | 10.1h |
| 5 | Kimi K3 | 0.2844 ±0.173 | 1,376 | 579.8M | 1.8M | $213.37 | 19.9h |
| 6 | Gemini 3.7 Flash | 0.1791 ±0.043 | 1,456 | 558.2M | 1.2M | $52.61 | 4.4h |
| 7 | DeepSeek V4 Flash Exp | 0.0751 ±0.046 | 2,861 | 1261.3M | 1.9M | $20.53 | 12.8h |
| 8 | Qwen3.8-Max | 0.0162 ±0.010 | 1,027 | 394.1M | 2.8M | $123.43 | 19.9h |
| 9 | Muse Spark 1.2 | 0.0035 ±0.008 | 356 | 90.6M | 672k | $30.14 | 3.7h |
| 10 | Inkling | 0.0000 ±0.000 | 157 | 7.3M | 29k | $7.06 | 0.3h |
The Game Boy Advance generates music through four programmable sound generator (PSG) channels driven directly by hardware registers (square waves, a programmable wavetable, and a noise channel), with every parameter change landing on an exact frame boundary. A step sequencer built on this hardware is a dense knot of state: pattern grids, per-channel parameter editors, sound banks, tempo logic, and a playback engine whose audible output depends on register-level synthesis details and frame-accurate timing.
This task turns a real, shipped GBA homebrew tracker into a clean-room reverse-engineering exercise. The reference ROM is never readable by the agent: it lives root-only inside the sandbox and can only be observed through a probe tool that runs keystroke scripts against it and returns what it drew and played. Everything the clone must reproduce has to be inferred from those observations.
The agent must reimplement the 4-channel PSG step sequencer from scratch as its own GBA ROM: the exact same screens, pixels, and controls, and above all the exact same sound and timing. The source lives in /app (starting from src/main.c) and make -C /app must produce /app/tracker.gba.
ref-probe <script.txt> <out_dir>, which captures screenshots (shot_*.png), audio traces (listen_*.npy), and frame-sequence recordings with an .mp4 preview for any keystroke script the agent writes../run-tests.sh, which builds the ROM and reports, per script, how close its frames and audio are to the reference.A ready-to-build GBA project at /app: a Makefile, a starter src/main.c, sample keystroke scripts in /app/scripts/, the ref-probe client, capture-and-compare tooling in /app/tools/ (the same capture code the verifier uses), and /app/README.md documenting the script format and workflow. The reference ROM itself is runnable but never readable; only its captured frames and audio are observable. The image preinstalls the devkitARM GBA toolchain, the mGBA emulator with Python bindings, and an audio-analysis stack (librosa, scipy, soundfile). The machine is offline.
A clean-room verifier in a separate container rebuilds the clone from captured source (make -C /app, run as the non-root agent user) and captures both the reference and the clone on a hidden suite of keystroke scripts: seeded, full-range variants of the public samples that sweep every scale, parameter, sound bank, tempo, and sustained 30-60 second songs. The reward is 0.25 × visual + 0.75 × audio, deliberately weighted toward the harder dimension.
tracker.gba is rejected if it is a symlink or resolves outside /app; the clone is fed to the emulator purely as data, with no agent process alive during capture.| Base image | devkitpro/devkitarm:latest |
| Tools available | FFmpeg, C/C++ toolchain, Python 3 |
| Compute | 8 CPUs · 8 GB RAM |
| Time limit | 20h |
ref-probe captures. No STEPPER or Bad Diode branding appears in any agent-visible surface, and no bytes of the ROM reach the graded deliverable.