02Performance Optimisation

Cranelift Codegen Optimization

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1
0.2340
±0.143
935366.5M719k$200.5316.3h
2Kimi K3
0.0243
±0.028
1,043402.4M884k$138.8718.9h
3Gemini 3.7 Flash
0.0220
±0.033
587221.0M245k$31.505.4h
4GLM-5.3
0.0079
±0.007
725298.7M452k$88.7812.5h
5DeepSeek V4 Flash Exp
0.0075
±0.010
835327.5M469k$5.388.1h
6GPT-5.6
0.0035
±0.008
523253.5M231k$264.947.7h
7Qwen3.8-Max
0.0023
±0.005
504116.7M2.1M$42.6019.3h
8Grok 4.6
0.0000
±0.000
573140.2M1.0M$147.909.9h
9Inkling
0.0000
±0.000
12011.7M18k$11.201.0h
10Muse Spark 1.2
0.0000
±0.000
445202.3M460k$53.218.5h

Background

Cranelift is the optimizing compiler backend inside Wasmtime. Its design point is fast compilation, so the machine code it emits leaves measurable performance on the table compared to LLVM-class backends: in the mid-end rewrite rules (written in the ISLE DSL), the e-graph optimizer and its cost model, the x86-64 instruction selector, and the vendored regalloc2 register allocator. Improving generated code without materially slowing the compiler is exactly the trade-off production compiler engineers navigate.

Correctness makes this unforgiving: WebAssembly semantics pin down exact float results and NaN propagation, integer division/remainder edges, shift-count masking, trap conditions and their ordering; a rewrite rule that is almost right miscompiles real programs.

Task

The agent must make the native code Cranelift generates run faster, without making the compiler itself do materially more work to compile. What counts is the machine code emitted for a WebAssembly module, not how fast Wasmtime itself runs. All changes are confined to the Wasmtime source tree at /app/wasmtime/, written as .rs and .isle sources.

  • /app/perf-check models the full performance requirement: it rebuilds the compiler, measures the generated code for the visible workloads against the unmodified compiler, and separately reports compile work on the biggest modules as a ratio to the baseline.
  • Compile time is a constraint, not free: the guidance is to stay within about +10% of baseline compile work; a runtime win bought by a much slower compile does not count.
  • Every workload's reference output and every spec test that passes today must still pass; one regression is as bad as making no change.
  • Cargo is offline with a fully vendored dependency graph: Cargo.toml, Cargo.lock, cargo config, and build.rs files are fixed, so no new dependencies can be added.

What The Agent Gets

The full pinned Wasmtime source tree with a warm target/ for incremental rebuilds, regalloc2 vendored as an editable path dependency, and the Wasm spec and misc test suites. Ten visible benchmark workloads ship as pre-built .wasmwith inputs and the unmodified compiler's recorded outputs: an interpreter, compression codecs, SQLite, crypto, and loop/recursion micro-kernels drawn largely from the Sightglass corpus.

/app/perf-check is the self-check loop: rebuild, compile every workload, measure the generated code deterministically, and compare against the baseline (with --wast for the spec suites and wasmtime-baseline on PATH for manual A/B). The measurement stack in /app/performance/ (per-address attribution plus per-opcode pricing) and the baseline measurements are all visible. The machine is offline with a pinned Rust toolchain preinstalled.

Verification

A clean-room verifier in a separate container reconstructs the scored tree from pristine Wasmtime sources plus only the agent's .rs/.isle edits, rebuilds it, and scores ten held-out workloads: same corpus families as the visible set (SIMD, compression, regex, crypto, jump-table and integer kernels) but never seen by the agent.

  • The metric is deterministic simulated work, not wall-clock: per-address execution counts are restricted to the code Cranelift generated (excluding Wasmtime startup), and each opcode is priced by a reciprocal-throughput cost table derived from uops.info, with codegen pinned to x86-64-v3.
  • Correctness gates run first: must-pass and must-fail canary suites, two passes over the full spec suites (nothing that passes on the reference may regress), edge-case binaries, and every workload's output compared against the reference compiler's build. Any failure zeroes the result.
  • The scored speedup is outlier-dampened (held down to the geometric mean that survives dropping the single best workload), then mapped on a convex curve with a 0.5% deadband reaching full credit at a 1.20x geomean. A one-sided compile-work penalty multiplies the result: free within +10% of baseline, linear to zero at +20%.
  • A single workload "improving" by more than 2.5x is treated as implausible and zeroed for review; modified sources that reach an external code generator are rejected; a compiler that fails to build or to run a workload the baseline handles scores zero.

Environment

Base imageubuntu:24.04
Tools availableRust, CMake, C/C++ toolchain, Python 3
Compute8 CPUs · 128 GB RAM
Time limit20h

Changes From V1

This task is an improved revision of Cranelift Codegen Optimization from FrontierSWE V1.

  • The metric moved from wall-clock timings in a benchmark runner to deterministic simulated work: per-address execution counts restricted to Cranelift-generated code, priced per opcode from a uops.info-derived cost table, with codegen pinned to x86-64-v3. One measurement per change replaces iteration counts and host noise.
  • Scoring moved from the visible tiered benchmark suite to ten held-out workloads from the same corpus families that the agent never sees, replacing the weighted-harmonic-mean formula with an outlier-dampened geometric mean (held to what survives dropping the single best workload) on a convex curve reaching full credit at 1.20x.
  • The compile-time constraint became an explicit modeled penalty: compile work on the biggest modules is measured against the baseline and multiplies the reward (free within +10%, linear to zero at +20%) instead of V1's unquantified "compile time must not regress".
  • Anti-cheat hardened: the verifier reconstructs the scored tree from pristine sources plus only the agent's .rs/.isle diffs, rejects modified sources that reach an external code generator, auto-zeroes implausible single-workload jumps above 2.5x, and adds must-pass/must-fail canary suites and edge-case binaries alongside the spec suites.
  • Verification became a separate clean-room container with its timeout cut from 24 hours to 2.5 hours, and the /app/.timer daemon files were replaced by the sandbox-timer CLI under the new harbor schema.

References

  • Wasmtime / Cranelift (Apache-2.0 WITH LLVM-exception): The full source tree at a pinned commit, fetched at image build to /app/wasmtime/as the codebase the agent optimizes; its WebAssembly spec and misc test suites are also baked into the verifier's pristine assets.
  • regalloc2 (Apache-2.0 WITH LLVM-exception): v0.15.0, cloned at image build into /app/wasmtime/vendor/regalloc2/ as a modifiable local path dependency of the Cranelift build.
  • uops.info instruction data (Cited per authors' request; format via XED-to-XML (Apache-2.0)): The per-instruction cost table is a derived aggregate of reciprocal-throughput measurements from Abel & Reineke's uops.info (ASPLOS 2019), generated once offline, not a copy of the dataset.
  • Sightglass benchmark corpus (Apache-2.0 OR MIT (corpus; members under their own licenses)): Pre-compiled .wasmworkloads and inputs derived from open-source projects including brotli, SQLite, SpiderMonkey, Lua, zstd, serde-json, bzip2, meshoptimizer, pulldown-cmark, rust regex, libsodium, BLAKE3, intgemm, LLVM test-suite and PolyBench/C kernels, and Computer Language Shootout micro-benchmarks, plus their bundled text input corpora (CommonMark spec, Learn X in Y Minutes, bzip2 documentation). Reference outputs are re-recorded from the task's own baseline compiler at image build.