| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.2340 ±0.143 | 935 | 366.5M | 719k | $200.53 | 16.3h |
| 2 | Kimi K3 | 0.0243 ±0.028 | 1,043 | 402.4M | 884k | $138.87 | 18.9h |
| 3 | Gemini 3.7 Flash | 0.0220 ±0.033 | 587 | 221.0M | 245k | $31.50 | 5.4h |
| 4 | GLM-5.3 | 0.0079 ±0.007 | 725 | 298.7M | 452k | $88.78 | 12.5h |
| 5 | DeepSeek V4 Flash Exp | 0.0075 ±0.010 | 835 | 327.5M | 469k | $5.38 | 8.1h |
| 6 | GPT-5.6 | 0.0035 ±0.008 | 523 | 253.5M | 231k | $264.94 | 7.7h |
| 7 | Qwen3.8-Max | 0.0023 ±0.005 | 504 | 116.7M | 2.1M | $42.60 | 19.3h |
| 8 | Grok 4.6 | 0.0000 ±0.000 | 573 | 140.2M | 1.0M | $147.90 | 9.9h |
| 9 | Inkling | 0.0000 ±0.000 | 120 | 11.7M | 18k | $11.20 | 1.0h |
| 10 | Muse Spark 1.2 | 0.0000 ±0.000 | 445 | 202.3M | 460k | $53.21 | 8.5h |
Cranelift is the optimizing compiler backend inside Wasmtime. Its design point is fast compilation, so the machine code it emits leaves measurable performance on the table compared to LLVM-class backends: in the mid-end rewrite rules (written in the ISLE DSL), the e-graph optimizer and its cost model, the x86-64 instruction selector, and the vendored regalloc2 register allocator. Improving generated code without materially slowing the compiler is exactly the trade-off production compiler engineers navigate.
Correctness makes this unforgiving: WebAssembly semantics pin down exact float results and NaN propagation, integer division/remainder edges, shift-count masking, trap conditions and their ordering; a rewrite rule that is almost right miscompiles real programs.
The agent must make the native code Cranelift generates run faster, without making the compiler itself do materially more work to compile. What counts is the machine code emitted for a WebAssembly module, not how fast Wasmtime itself runs. All changes are confined to the Wasmtime source tree at /app/wasmtime/, written as .rs and .isle sources.
/app/perf-check models the full performance requirement: it rebuilds the compiler, measures the generated code for the visible workloads against the unmodified compiler, and separately reports compile work on the biggest modules as a ratio to the baseline.Cargo.toml, Cargo.lock, cargo config, and build.rs files are fixed, so no new dependencies can be added.The full pinned Wasmtime source tree with a warm target/ for incremental rebuilds, regalloc2 vendored as an editable path dependency, and the Wasm spec and misc test suites. Ten visible benchmark workloads ship as pre-built .wasmwith inputs and the unmodified compiler's recorded outputs: an interpreter, compression codecs, SQLite, crypto, and loop/recursion micro-kernels drawn largely from the Sightglass corpus.
/app/perf-check is the self-check loop: rebuild, compile every workload, measure the generated code deterministically, and compare against the baseline (with --wast for the spec suites and wasmtime-baseline on PATH for manual A/B). The measurement stack in /app/performance/ (per-address attribution plus per-opcode pricing) and the baseline measurements are all visible. The machine is offline with a pinned Rust toolchain preinstalled.
A clean-room verifier in a separate container reconstructs the scored tree from pristine Wasmtime sources plus only the agent's .rs/.isle edits, rebuilds it, and scores ten held-out workloads: same corpus families as the visible set (SIMD, compression, regex, crypto, jump-table and integer kernels) but never seen by the agent.
| Base image | ubuntu:24.04 |
| Tools available | Rust, CMake, C/C++ toolchain, Python 3 |
| Compute | 8 CPUs · 128 GB RAM |
| Time limit | 20h |
This task is an improved revision of Cranelift Codegen Optimization from FrontierSWE V1.
.rs/.isle diffs, rejects modified sources that reach an external code generator, auto-zeroes implausible single-workload jumps above 2.5x, and adds must-pass/must-fail canary suites and edge-case binaries alongside the spec suites./app/.timer daemon files were replaced by the sandbox-timer CLI under the new harbor schema./app/wasmtime/as the codebase the agent optimizes; its WebAssembly spec and misc test suites are also baked into the verifier's pristine assets./app/wasmtime/vendor/regalloc2/ as a modifiable local path dependency of the Cranelift build..wasmworkloads and inputs derived from open-source projects including brotli, SQLite, SpiderMonkey, Lua, zstd, serde-json, bzip2, meshoptimizer, pulldown-cmark, rust regex, libsodium, BLAKE3, intgemm, LLVM test-suite and PolyBench/C kernels, and Computer Language Shootout micro-benchmarks, plus their bundled text input corpora (CommonMark spec, Learn X in Y Minutes, bzip2 documentation). Reference outputs are re-recorded from the task's own baseline compiler at image build.