05 Implementation

Lua Native Compiler

Results

#ModelSuccess RateAvgBest
1
GPT-5.4
Codex
0/569%88%
2
Claude Opus 4.6
Claude Code
0/549%85%
3
Gemini 3.1 Pro
Gemini CLI
0/517%48%
4
Kimi K2.5
Kimi CLI
0/50.5%2.8%
5
Qwen3.6-Plus
Qwen Code
0/50%0%

Background

Lua 5.4 is implemented as a bytecode virtual machine with runtime semantics around tables, closures, coroutines, metatables, and garbage collection. This task asks for those programs to be compiled into standalone x86-64 ELF binaries rather than executed through the stock VM.

The provided workspace splits parsing support from runtime support, so the task sits across bytecode decoding, code generation, calling conventions, and runtime integration. The compiler must preserve Lua behavior closely enough that hidden programs produce the same output as the reference interpreter.

Task

The agent starts from a workspace that contains Lua 5.4 headers, a parser-capable compile library, and a separate runtime library. The deliverable is a compiler binary named luanatc that accepts Lua source files and emits standalone x86-64 ELF executables.

In practice that means parsing Lua into bytecode, translating Lua VM instructions into native code, linking against the provided runtime helpers, and producing binaries whose stdout exactly matches the reference interpreter on hidden programs.

  • Use the parser in liblua-compile.a to recover Lua prototypes.
  • Generate machine code or assembly instead of dispatching through the VM.
  • Link output programs against liblua-runtime.a for tables, GC, strings, metamethods, and standard libraries.
  • Emit real ELF binaries, not scripts or wrappers around a hidden interpreter.

Evaluation

The verifier scores hidden test pass rate on 182 Lua programs after build, anti-cheat, and compiler-presence checks. Any hard-fail gate forces the result to zero before pass rate is applied.

  • Source scans try to block direct access to verifier-only paths.
  • Output binaries are inspected for forbidden lua_* and luaL_* symbols.
  • A compiler that passes 91 of 182 hidden programs scores exactly 0.500.

No model was able to complete this task successfully, so we used overall test pass rate as a partial reward to rank models.

Environment And Constraints

Agents run offline on an 8 CPU, 32 GB RAM container with a one-hour verifier budget. The runtime and compile libraries are deliberately split so the compiler can inspect Lua programs while the generated binaries cannot fall back to the stock VM.

  • No internet access and no verifier-visible reference binaries during scoring.
  • Output binaries must link against liblua-runtime.a; linking them against liblua-compile.a or a full liblua.a hard-fails the verifier.
  • Output binaries must not rely on the lua_*/luaL_* C embedding API beyond one-time init and teardown.
  • The compiler must not invoke gcc, clang, or cc to compile generated C, and it must not wrap the reference interpreter, which is deleted before verification.
  • Assembler and linker based code generation is allowed.