| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 1.0000 ±0.000 | 194 | 64.4M | 384k | $41.54 | 3.2h |
| 2 | Claude Opus 5.5 | 1.0000 ±0.000 | 305 | 137.9M | 606k | $43.54 | 3.8h |
| 3 | Claude Sonnet 5.5 | 0.9968 ±0.007 | 511 | 211.1M | 1.0M | $55.43 | 8.0h |
| 4 | Gemini 4 Argon | 0.9923 ±0.011 | 163 | 62.9M | 294k | $42.42* | 2.6h |
| 5 | Claude Haiku 5.5 | 0.9905 ±0.009 | 426 | 173.2M | 1.0M | $11.39 | 4.6h |
| 6 | Claude Opus 5 | 0.9892 ±0.016 | 642 | 273.8M | 932k | $171.98 | 12.4h |
| 7 | GPT-6 Astra | 0.9801 ±0.001 | 240 | 54.3M | 260k | $103.38 | 2.1h |
| 8 | GPT-5.6 Sol | 0.9314 ±0.083 | 728 | 249.9M | 372k | $248.76 | 7.1h |
| 9 | Grok 4.7 | 0.7367 ±0.413 | 1,403 | 342.8M | 1.2M | $347.13 | 15.7h |
| 10 | Muse Spark 1.3 | 0.6273 ±0.240 | 1,107 | 487.7M | 943k | $350.43 | 19.9h |
| 11 | Claude Fable 5 | 0.5968 ±0.545 | 350 | 165.1M | 682k | $214.26 | 9.8h |
| 12 | Kimi K3 | 0.5930 ±0.276 | 1,025 | 446.5M | 1.3M | $163.81 | 19.9h |
| 13 | GLM-5.3 | 0.4429 ±0.402 | 905 | 428.9M | 827k | $126.63 | 19.0h |
| 14 | Gemini 3.7 Flash | 0.4029 ±0.493 | 1,471 | 586.8M | 1.3M | $75.03 | 16.0h |
| 15 | GLM-5.3 Flash | 0.3200 ±0.221 | 1,046 | 454.8M | 1.2M | $15.84 | 15.3h |
| 16 | DeepSeek V4 Flash Vision Exp | 0.1884 ±0.170 | 1,533 | 700.2M | 1.4M | $11.96 | 11.3h |
| 17 | Grok 4.6 | 0.0765 ±0.171 | 888 | 216.0M | 1.7M | $222.15 | 13.4h |
| 18 | Qwen3.8-Max-0902 | 0.0353 ±0.073 | 239 | 58.3M | 3.1M | $35.41 | 18.6h |
| 19 | Gemini 3.8 Flash | 0.0000 ±0.000 | 1,724 | 706.6M | 1.1M | $77.23 | 16.4h |
| 20 | Inkling | 0.0000 ±0.000 | 119 | 7.7M | 32k | $7.42 | 0.8h |
| 21 | Muse Spark 1.2 | 0.0000 ±0.000 | 292 | 92.4M | 660k | $21.30 | 3.9h |
| 22 | Qwen3.8-Max | 0.0000 ±0.000 | 156 | 24.7M | 1.1M | $13.28 | 20.0h |
*Gemini 4 Argon: Cache hit rate is much lower due to a non-production setting.
The Lean 4 kernel is the small trusted core of a major proof assistant: a dependent type theory with universes, inductive families, recursors, quotient types, and definitional equality (everything Mathlib's formalized mathematics ultimately rests on). Independent kernel re-implementations exist precisely so the kernel can be cross-checked, and this task asks for one: a standalone checker that reads exported Lean environments and decides, declaration by declaration, whether they are admissible.
The difficulty is twofold. Semantically, the checker must reproduce subtle kernel behavior: reduction, universe handling, the Nat and String literal extensions, and the admissibility rules for (possibly mutual and nested) inductives. Operationally, real inputs are large: exports range from a few hundred bytes to about 90 MB with over 1.5 million objects, so throughput is part of what is measured.
Implement the Lean 4 kernel's type checker in the Free Pascal project at /app/checker/, with sources only as Pascal files under src/ and the program entry in src/checker.pas. The documented fpc build line produces build/checker, invoked as checker <path-to-export-file>.
lean4export NDJSON file is well-formed, well-typed and admissible, and the file is structurally valid; exit non-zero otherwise. The exit code is the result.False is the most serious error the checker can make./app/README.md carries the CLI contract, the complete NDJSON wire-format specification (names, levels, expressions, and all six declaration kinds, adapted from lean4export's format document), and the build and test commands. /app/exports/ holds worked examples of both verdicts (every file under accept/ must exit 0 and every file under reject/ must not, with expected.tsv listing them all), and /app/run-tests.sh builds the checker and runs any subset. The Free Pascal compiler is preinstalled and the machine is offline.
A clean-room verifier, separate from the agent's environment, stages only the Pascal sources under src/, rebuilds them offline with fpc -O2as a non-root user, and runs a hidden corpus of 607 export files (~2.4 GB): genuine Lean, Mathlib, Batteries and Aesop environments to accept, plus 131 deliberately corrupted or False-grafted exports to reject, each labeled offline by an independent reference kernel (nanoda_lib). Cases are staged under content-hash names with uniform timestamps, so neither path nor order reveals a verdict; the checker's exit status on each file is the verdict.
False is a hard zero.src/checker.pas entry point zero the result.| Base image | ubuntu:24.04 |
| Tools available | Free Pascal, C/C++ toolchain, Python 3 |
| Compute | 4 CPUs · 8 GB RAM |
| Time limit | 20h |
v4.33.0-rc1 toolchain compiled the modules whose environments were exported; the corpus also exports declarations from its own Init, Lean and Std libraries. Not shipped in the image..ndjson case (format version 3.1.0); its format document is the source of the wire-format section of /app/README.md, adapted under the same license. Not shipped in the image.False) and must not be mistaken for valid Mathlib output. Not shipped in the image./app.