| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.9026 ±0.148 | 237 | 80.4M | 420k | $50.45 | 3.8h |
| 2 | GLM-5.3 | 0.7811 ±0.437 | 1,778 | 808.0M | 1.3M | $221.13 | 15.6h |
| 3 | Gemini 3.7 Flash | 0.5943 ±0.151 | 984 | 396.0M | 1.0M | $43.73 | 4.8h |
| 4 | Kimi K3 | 0.5250 ±0.401 | 1,262 | 560.6M | 1.3M | $195.62 | 19.7h |
| 5 | DeepSeek V4 Flash Exp | 0.2358 ±0.424 | 2,475 | 1151.0M | 2.1M | $19.30 | 13.0h |
| 6 | Grok 4.6 | 0.1277 ±0.285 | 1,113 | 288.4M | 2.1M | $297.13 | 10.7h |
| 7 | GPT-5.6 | 0.0000 ±0.000 | 224 | 50.9M | 157k | $43.12 | 1.5h |
| 8 | Inkling | 0.0000 ±0.000 | 257 | 24.1M | 74k | $23.20 | 0.8h |
| 9 | Muse Spark 1.2 | 0.0000 ±0.000 | 237 | 71.5M | 370k | $18.96 | 2.5h |
| 10 | Qwen3.8-Max | 0.0000 ±0.000 | 740 | 278.2M | 2.0M | $82.46 | 18.3h |
Compiling a dynamic language ahead of time means replacing its interpreter loop with generated machine code: every arithmetic operation needs type-tag dispatch with metamethod fallbacks, and closures, upvalues, coroutines, and the garbage collector all have to keep working through native call frames. This task provides Lua's parser and full runtime as static libraries; the code generator, the native replacement for the bytecode dispatch loop, is the deliverable.
The compiler must also be genuinely retargetable: it emits code for three instruction sets (x86-64, aarch64, and riscv64), and the cross-architecture binaries are exercised under QEMU user-mode emulation. Correctness is defined differentially: each emitted binary must reproduce stock Lua 5.4's stdout byte for byte.
Build luanatc, an ahead-of-time compiler written in Go in the project at /app/lua-native-compiler/ (go build -o luanatc .), invoked as luanatc program.lua -o out --target <x86_64|aarch64|riscv64>.
as/ld and their aarch64/riscv64 cross variants) or construct the ELF directly. A C compiler may be used only as a link driver over object files; handing it generated C source is a disclosed boundary violation./reference/lua-src/<arch>/liblua-runtime.a (GC, tables, strings, metamethods, coroutines, stdlib, VM helpers); the compiler itself can link liblua-compile.a to parse Lua source into bytecode.//go:embed is not allowed.A Go scaffold at /app/lua-native-compiler/ and an /app/README.md documenting the CLI, the linking contract, and the per-target toolchain table. Example Lua programs live under /app/tests/programs/ with their exact expected stdout in /app/tests/expected/, and /app/run-tests.sh compiles and checks every program on every target (cross targets under qemu-<arch>-static). The image provides Lua 5.4 headers, the host parser archive, the three per-target runtime archives, native and cross binutils, and QEMU user-mode emulators. No Lua interpreter ships in the image: expected outputs were baked at image build and the interpreter deleted. The machine is offline and a sandbox-timer CLI reports the remaining budget.
A clean-room verifier, separate from the agent's environment, scans the sources for disclosed-boundary violations (references to verifier internals, or invoking a C compiler on C source), reconstructs the project while dropping any compiled objects, archives, or ELF files, rebuilds it from Go source as a non-root user, and then scores a hidden corpus: perturbed twins of the public test chunks, each mutated and its expected output re-baked with the real reference interpreter at image build, so memorising the visible corpus does not transfer.
luaV_execute, or use of the C embedding API beyond one-time init. Build failures, a missing go.mod, and reset or scan violations zero the result.| Base image | ubuntu:24.04 |
| Tools available | Go, QEMU, C/C++ toolchain, Python 3 |
| Compute | 8 CPUs · 32 GB RAM |
| Time limit | 20h |
This task is an improved revision of Lua Native Compiler from FrontierSWE V1.
/reference/lua and luac during the run and deleted them before verification; V2 bakes expected outputs at image build and removes the interpreter from the image entirely.luanatc binary) to a single Go module rebuilt from committed source after a reset that drops all compiled objects, archives, and ELF files.gcc used as a link driver.sandbox-timer CLI replacing /app/.timer files.liblua-compile.a, liblua-runtime.a) compiled from upstream sources with task-authored stubs replacing the dispatch loop, parser, and bytecode loader. The built interpreter is used only at image build to bake expected outputs, then deleted; no interpreter binary ships.