15Implementation

Lua Native Compiler

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1
0.9026
±0.148
23780.4M420k$50.453.8h
2GLM-5.3
0.7811
±0.437
1,778808.0M1.3M$221.1315.6h
3Gemini 3.7 Flash
0.5943
±0.151
984396.0M1.0M$43.734.8h
4Kimi K3
0.5250
±0.401
1,262560.6M1.3M$195.6219.7h
5DeepSeek V4 Flash Exp
0.2358
±0.424
2,4751151.0M2.1M$19.3013.0h
6Grok 4.6
0.1277
±0.285
1,113288.4M2.1M$297.1310.7h
7GPT-5.6
0.0000
±0.000
22450.9M157k$43.121.5h
8Inkling
0.0000
±0.000
25724.1M74k$23.200.8h
9Muse Spark 1.2
0.0000
±0.000
23771.5M370k$18.962.5h
10Qwen3.8-Max
0.0000
±0.000
740278.2M2.0M$82.4618.3h

Background

Compiling a dynamic language ahead of time means replacing its interpreter loop with generated machine code: every arithmetic operation needs type-tag dispatch with metamethod fallbacks, and closures, upvalues, coroutines, and the garbage collector all have to keep working through native call frames. This task provides Lua's parser and full runtime as static libraries; the code generator, the native replacement for the bytecode dispatch loop, is the deliverable.

The compiler must also be genuinely retargetable: it emits code for three instruction sets (x86-64, aarch64, and riscv64), and the cross-architecture binaries are exercised under QEMU user-mode emulation. Correctness is defined differentially: each emitted binary must reproduce stock Lua 5.4's stdout byte for byte.

Task

Build luanatc, an ahead-of-time compiler written in Go in the project at /app/lua-native-compiler/ (go build -o luanatc .), invoked as luanatc program.lua -o out --target <x86_64|aarch64|riscv64>.

  • Each emitted binary must be a standalone native ELF for the requested target that reproduces stock Lua 5.4's stdout byte for byte and exits 0, not an interpreter that reads embedded bytecode at runtime.
  • Emit textual assembly and assemble/link it (as/ld and their aarch64/riscv64 cross variants) or construct the ELF directly. A C compiler may be used only as a link driver over object files; handing it generated C source is a disclosed boundary violation.
  • Emitted binaries link the provided per-target runtime archive /reference/lua-src/<arch>/liblua-runtime.a (GC, tables, strings, metamethods, coroutines, stdlib, VM helpers); the compiler itself can link liblua-compile.a to parse Lua source into bytecode.
  • The project must build purely from committed Go source; embedding prebuilt object files, archives, or binaries via //go:embed is not allowed.

What The Agent Gets

A Go scaffold at /app/lua-native-compiler/ and an /app/README.md documenting the CLI, the linking contract, and the per-target toolchain table. Example Lua programs live under /app/tests/programs/ with their exact expected stdout in /app/tests/expected/, and /app/run-tests.sh compiles and checks every program on every target (cross targets under qemu-<arch>-static). The image provides Lua 5.4 headers, the host parser archive, the three per-target runtime archives, native and cross binutils, and QEMU user-mode emulators. No Lua interpreter ships in the image: expected outputs were baked at image build and the interpreter deleted. The machine is offline and a sandbox-timer CLI reports the remaining budget.

Verification

A clean-room verifier, separate from the agent's environment, scans the sources for disclosed-boundary violations (references to verifier internals, or invoking a C compiler on C source), reconstructs the project while dropping any compiled objects, archives, or ELF files, rebuilds it from Go source as a non-root user, and then scores a hidden corpus: perturbed twins of the public test chunks, each mutated and its expected output re-baked with the real reference interpreter at image build, so memorising the visible corpus does not transfer.

  • The reward is passed over attempted across (program × target) cells: every scored chunk is compiled and run for each of the three architectures, so a compiler that only handles one tops out near a third of the score. Per-invocation compile and run caps are 60 seconds each under a 9000-second suite deadline.
  • Each emitted binary is snapshotted to a root-owned immutable copy before checking and running, so the inspected bytes are exactly the executed bytes; runs go through a hidden ptrace no-exec launcher that kills any binary attempting to exec a second program.
  • Structural gates reject smuggled interpreters: embedded precompiled Lua chunks, real parser/code-generator/loader symbols beyond the provided error stubs, an oversized luaV_execute, or use of the C embedding API beyond one-time init. Build failures, a missing go.mod, and reset or scan violations zero the result.

Environment

Base imageubuntu:24.04
Tools availableGo, QEMU, C/C++ toolchain, Python 3
Compute8 CPUs · 32 GB RAM
Time limit20h

Changes From V1

This task is an improved revision of Lua Native Compiler from FrontierSWE V1.

  • The compiler is now multi-target: V1 scored x86-64 only, while this QEMU-based revision compiles and runs every program for x86-64, aarch64, and riscv64 (cross targets under QEMU user-mode emulation) and scores (program × target) cells; a single-architecture compiler tops out near a third of the reward.
  • The test corpus changed from a bespoke hidden suite of authored programs to the official Lua 5.4.7 test suite, split into self-contained chunks and bake-filtered against the reference. The full corpus is now agent-visible with expected outputs, and grading uses root-only perturbed twins whose expected outputs are re-baked by the real interpreter: anti-memorisation by mutation instead of hiding the tests.
  • The reference interpreter no longer ships: V1 gave the agent /reference/lua and luac during the run and deleted them before verification; V2 bakes expected outputs at image build and removes the interpreter from the image entirely.
  • The build contract tightened from five accepted build systems (or even a committed pre-built luanatc binary) to a single Go module rebuilt from committed source after a reset that drops all compiled objects, archives, and ELF files.
  • Anti-cheat moved from static heuristics to runtime enforcement: a hidden ptrace no-exec launcher kills emitted binaries that exec a second program, every binary is snapshotted to a root-owned immutable copy so the checked and executed bytes cannot diverge, and symbol-size gates verify the parser/VM-engine stubs stayed stubs. The C-compiler scan was also refined to reject only compiling C source, no longer false-flagging gcc used as a link driver.
  • Infrastructure standardized: a clean-room separate verifier with hash-pinned agent and verifier images, the agent budget grown from 8 to 20 hours, the verifier budget from 1 hour to 12000 seconds to cover three targets under emulation, and the standard sandbox-timer CLI replacing /app/.timer files.

References

  • Lua (MIT): Lua 5.4.7, fetched at image build from the official distribution (nothing vendored). The image ships the public and internal headers and two derived static libraries (liblua-compile.a, liblua-runtime.a) compiled from upstream sources with task-authored stubs replacing the dispatch loop, parser, and bytecode loader. The built interpreter is used only at image build to bake expected outputs, then deleted; no interpreter binary ships.
  • Lua 5.4 test suite (MIT): The official Lua 5.4 behavioural suite (version-matched to the runtime), fetched at image build. Its files are adapted to run standalone, split into small self-contained chunks, and bake-filtered against the real reference; the whole corpus ships to the agent un-mutated, while a perturbed twin of every chunk (the graded material) is held root-only.