| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 1.0000 ±0.000 | 194 | 64.4M | 384k | $41.54 | 3.2h |
| 2 | GPT-5.6 | 0.9314 ±0.083 | 728 | 249.9M | 372k | $248.76 | 7.1h |
| 3 | Kimi K3 | 0.5930 ±0.276 | 1,025 | 446.5M | 1.3M | $163.81 | 20.0h |
| 4 | GLM-5.3 | 0.4429 ±0.402 | 905 | 428.9M | 827k | $126.63 | 19.0h |
| 5 | Gemini 3.7 Flash | 0.4029 ±0.493 | 1,471 | 586.8M | 1.3M | $75.03 | 16.0h |
| 6 | DeepSeek V4 Flash Exp | 0.1884 ±0.170 | 1,533 | 700.2M | 1.4M | $11.96 | 11.3h |
| 7 | Grok 4.6 | 0.0765 ±0.171 | 888 | 216.0M | 1.7M | $222.15 | 13.4h |
| 8 | Inkling | 0.0000 ±0.000 | 119 | 7.7M | 32k | $7.42 | 0.8h |
| 9 | Muse Spark 1.2 | 0.0000 ±0.000 | 292 | 92.4M | 660k | $21.30 | 3.9h |
| 10 | Qwen3.8-Max | 0.0000 ±0.000 | 156 | 24.7M | 1.1M | $13.28 | 20.0h |
The Lean 4 kernel is the small trusted core of a major proof assistant: a dependent type theory with universes, inductive families, recursors, quotient types, and definitional equality (everything Mathlib's formalized mathematics ultimately rests on). Independent kernel re-implementations exist precisely so the kernel can be cross-checked, and this task asks for one: a standalone checker that reads exported Lean environments and decides, declaration by declaration, whether they are admissible.
The difficulty is twofold. Semantically, the checker must reproduce subtle kernel behavior: reduction, universe handling, the Nat and String literal extensions, and the admissibility rules for (possibly mutual and nested) inductives. Operationally, real inputs are large: exports range from a few hundred bytes to about 90 MB with over 1.5 million objects, so throughput is part of what is measured.
Implement the Lean 4 kernel's type checker in the Free Pascal project at /app/checker/, with sources only as Pascal files under src/ and the program entry in src/checker.pas. The documented fpc build line produces build/checker, invoked as checker <path-to-export-file>.
lean4export NDJSON file is well-formed, well-typed and admissible, and the file is structurally valid; exit non-zero otherwise. The exit code is the result.False is the most serious error the checker can make./app/README.md carries the CLI contract, the complete NDJSON wire-format specification (names, levels, expressions, and all six declaration kinds, adapted from lean4export's format document), and the build and test commands. /app/exports/ holds worked examples of both verdicts (every file under accept/ must exit 0 and every file under reject/ must not, with expected.tsv listing them all), and /app/run-tests.sh builds the checker and runs any subset. The Free Pascal compiler is preinstalled and the machine is offline.
A clean-room verifier, separate from the agent's environment, stages only the Pascal sources under src/, rebuilds them offline with fpc -O2as a non-root user, and runs a hidden corpus of 607 export files (~2.4 GB): genuine Lean, Mathlib, Batteries and Aesop environments to accept, plus 131 deliberately corrupted or False-grafted exports to reject, each labeled offline by an independent reference kernel (nanoda_lib). Cases are staged under content-hash names with uniform timestamps, so neither path nor order reveals a verdict; the checker's exit status on each file is the verdict.
False is a hard zero.src/checker.pas entry point zero the result.| Base image | ubuntu:24.04 |
| Tools available | Free Pascal, C/C++ toolchain, Python 3 |
| Compute | 4 CPUs · 8 GB RAM |
| Time limit | 20h |
v4.33.0-rc1 toolchain compiled the modules whose environments were exported; the corpus also exports declarations from its own Init, Lean and Std libraries. Not shipped in the image..ndjson case (format version 3.1.0); its format document is the source of the wire-format section of /app/README.md, adapted under the same license. Not shipped in the image.False) and must not be mistaken for valid Mathlib output. Not shipped in the image./app.