| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.5105 ±0.030 | 1,997 | 859.9M | 2.2M | $483.24 | 19.1h |
| 2 | GPT-5.6 | 0.2658 ±0.058 | 2,244 | 1037.2M | 806k | $1044.53 | 19.3h |
| 3 | Kimi K3 | 0.1633 ±0.012 | 1,609 | 711.5M | 1.1M | $237.57 | 19.8h |
| 4 | Grok 4.6 | 0.1490 ±0.018 | 2,237 | 595.3M | 2.1M | $593.06 | 17.9h |
| 5 | GLM-5.3 | 0.1415 ±0.014 | 1,737 | 777.5M | 807k | $214.26 | 19.8h |
| 6 | Gemini 3.7 Flash | 0.1195 ±0.070 | 2,189 | 998.3M | 1.6M | $97.22 | 19.9h |
| 7 | DeepSeek V4 Flash Exp | 0.1156 ±0.012 | 3,369 | 1556.6M | 2.0M | $24.95 | 19.1h |
| 8 | Qwen3.8-Max | 0.1038 ±0.011 | 1,752 | 702.5M | 1.4M | $186.05 | 19.6h |
| 9 | Inkling | 0.0324 ±0.066 | 191 | 12.6M | 24k | $12.07 | 1.1h |
| 10 | Muse Spark 1.2 | 0.0037 ±0.008 | 344 | 130.4M | 382k | $22.49 | 2.3h |
git is one of the most widely deployed systems programs in existence: an object database, a content-addressed storage format, an index, refs, diff and merge machinery, and dozens of porcelain commands, all with two decades of precisely specified behaviour down to exact output strings and exit codes. Reimplementing it is a marathon of format compatibility (SHA-1 object encoding, pack files, the index format) layered under a very large CLI surface.
In this task the agent is handed tests, not source. git's C code is not in the workspace and the real gitbinary is locked root-only, so the behaviour must be reconstructed from git's own behavioural test scripts, which serve as the executable specification.
Implement git's version control functionality in Zig inside the pre-scaffolded project at /app/zig-git/. The build scaffold is fixed: zig build compiles src/ into zig-out/bin/git, and the agent's sources may only be .zig files under src/.
/app/tests/t/./app/run_tests.py (the full set), /app/run_tests.py --sample (a quick representative slice), or by naming individual scripts..zig sources under /app/zig-git/src/; the verifier discards everything else.A ready-to-build Zig scaffold at /app/zig-git/ (build.zig, build.zig.zon, src/) and an agent-readable copy of git v2.47.0's own behavioural test suite at /app/tests/: the test-lib.sh harness, prebuilt test-helper binaries, and a visible subset of the t/scripts, with git's C source and git's own binaries removed. A provided /app/run_tests.pyrunner drives the scripts against the agent's binary. Zig 0.14.0 is preinstalled, the machine is offline, and a sandbox-timer CLI reports the remaining time budget. The real git is installed but root-only: unreadable and unrunnable by the agent.
A clean-room verifier, separate from the agent's environment, first reconstructs the scored project from a pristine baked scaffold plus only the agent's .zigsources (symlinks and caches excluded), so tampering with the build configuration or stashing binaries in the workspace cannot survive. It then builds the candidate as a non-root user and runs 743 scored scripts from a trusted, root-only copy of git's test suite against the resulting binary.
| Base image | ubuntu:24.04 |
| Tools available | Zig, C/C++ toolchain, Python 3 |
| Compute | 4 CPUs · 16 GB RAM |
| Time limit | 20h |
This task is an improved revision of Git to Zig from FrontierSWE V1.
/app/git-src/and a working system git to test against; V2 removes both. The agent now gets only git's behavioural test suite as the specification, and the real git binary is locked root-only.environment_mode = "separate")..zig sources, so build-config tampering, stashed binaries, and wrapper tricks are discarded before scoring.--sample mode for quick iteration), and the ad-hoc /app/.timer file interface was replaced by the standard sandbox-timer CLI.t/ scripts and test-lib.shharness) and its build outputs, cloned and built once at image build from the pinned tag. The full built tree is baked root-only as the verifier's trusted corpus; the agent-readable copy keeps the harness, prebuilt test-helpers, and a visible subset of the scripts, with git's C source and git's own binaries removed; the agent is handed tests, not source./app/tests.