09Implementation

Git to Zig

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1
0.5105
±0.030
1,997859.9M2.2M$483.2419.1h
2GPT-5.6
0.2658
±0.058
2,2441037.2M806k$1044.5319.3h
3Kimi K3
0.1633
±0.012
1,609711.5M1.1M$237.5719.8h
4Grok 4.6
0.1490
±0.018
2,237595.3M2.1M$593.0617.9h
5GLM-5.3
0.1415
±0.014
1,737777.5M807k$214.2619.8h
6Gemini 3.7 Flash
0.1195
±0.070
2,189998.3M1.6M$97.2219.9h
7DeepSeek V4 Flash Exp
0.1156
±0.012
3,3691556.6M2.0M$24.9519.1h
8Qwen3.8-Max
0.1038
±0.011
1,752702.5M1.4M$186.0519.6h
9Inkling
0.0324
±0.066
19112.6M24k$12.071.1h
10Muse Spark 1.2
0.0037
±0.008
344130.4M382k$22.492.3h

Background

git is one of the most widely deployed systems programs in existence: an object database, a content-addressed storage format, an index, refs, diff and merge machinery, and dozens of porcelain commands, all with two decades of precisely specified behaviour down to exact output strings and exit codes. Reimplementing it is a marathon of format compatibility (SHA-1 object encoding, pack files, the index format) layered under a very large CLI surface.

In this task the agent is handed tests, not source. git's C code is not in the workspace and the real gitbinary is locked root-only, so the behaviour must be reconstructed from git's own behavioural test scripts, which serve as the executable specification.

Task

Implement git's version control functionality in Zig inside the pre-scaffolded project at /app/zig-git/. The build scaffold is fixed: zig build compiles src/ into zig-out/bin/git, and the agent's sources may only be .zig files under src/.

  • Match real git's behaviour (its subcommands, output formats, and exit codes) as pinned down by the behavioural test scripts in /app/tests/t/.
  • Run the tests with /app/run_tests.py (the full set), /app/run_tests.py --sample (a quick representative slice), or by naming individual scripts.
  • The implementation must be Zig itself: delegating to an existing git binary or linking libgit2 zeroes the result.
  • All changes must stay inside .zig sources under /app/zig-git/src/; the verifier discards everything else.

What The Agent Gets

A ready-to-build Zig scaffold at /app/zig-git/ (build.zig, build.zig.zon, src/) and an agent-readable copy of git v2.47.0's own behavioural test suite at /app/tests/: the test-lib.sh harness, prebuilt test-helper binaries, and a visible subset of the t/scripts, with git's C source and git's own binaries removed. A provided /app/run_tests.pyrunner drives the scripts against the agent's binary. Zig 0.14.0 is preinstalled, the machine is offline, and a sandbox-timer CLI reports the remaining time budget. The real git is installed but root-only: unreadable and unrunnable by the agent.

Verification

A clean-room verifier, separate from the agent's environment, first reconstructs the scored project from a pristine baked scaffold plus only the agent's .zigsources (symlinks and caches excluded), so tampering with the build configuration or stashing binaries in the workspace cannot survive. It then builds the candidate as a non-root user and runs 743 scored scripts from a trusted, root-only copy of git's test suite against the resulting binary.

  • Scoring is a weighted mean of per-script completion fractions: each script's TAP passes are capped at the reference implementation's count, a per-script floor subtracts everything the empty scaffold already passes (a no-op submission scores exactly 0), and a contribution cap keeps any single script from outweighing roughly 1/133 of the total.
  • TAP parsing is plan-bounded and per-script results are captured by the root harness into a nonce-named root-only directory, so forged "ok" lines cannot inflate the score.
  • Build failure, a missing or non-ELF binary, linking libgit2, or a failed workspace reset zeroes the result; per-script time caps scaled from baked reference durations and a global suite deadline make hangs score ~0 instead of erroring.

Environment

Base imageubuntu:24.04
Tools availableZig, C/C++ toolchain, Python 3
Compute4 CPUs · 16 GB RAM
Time limit20h

Changes From V1

This task is an improved revision of Git to Zig from FrontierSWE V1.

  • The V1 workspace shipped git's full C source at /app/git-src/and a working system git to test against; V2 removes both. The agent now gets only git's behavioural test suite as the specification, and the real git binary is locked root-only.
  • Verification moved from a bash script running inside the agent's own container to a clean-room verifier in a separate, hash-pinned image (environment_mode = "separate").
  • Anti-cheat was rebuilt from detection to construction: V1 relied on grep and strace heuristics to catch wrappers and embedded C; V2 reconstructs the scored project from a pristine baked scaffold plus only the agent's .zig sources, so build-config tampering, stashed binaries, and wrapper tricks are discarded before scoring.
  • Scoring gates hardened: V1 divided raw TAP passes by a fixed oracle total across the whole suite; V2 scores 743 curated scripts against a baked per-script reference, subtracts an empty-scaffold floor so a no-op scores exactly 0, caps credit at the reference ceiling, and bounds any single script's contribution, with plan-bounded TAP parsing against forged output.
  • The verifier now runs the agent's build and tests as a non-root user with the reward directory locked before any agent code executes; its timeout dropped from 24 hours to 2.5 hours via per-script time caps scaled from baked reference durations.
  • The brief was rewritten around the runnable test suite (with a --sample mode for quick iteration), and the ad-hoc /app/.timer file interface was replaced by the standard sandbox-timer CLI.

References

  • git (GPL-2.0): git v2.47.0's own behavioural test suite (the t/ scripts and test-lib.shharness) and its build outputs, cloned and built once at image build from the pinned tag. The full built tree is baked root-only as the verifier's trusted corpus; the agent-readable copy keeps the harness, prebuilt test-helpers, and a visible subset of the scripts, with git's C source and git's own binaries removed; the agent is handed tests, not source.
  • sha1collisiondetection (MIT): git's bundled SHA-1 collision-detection library (pinned by git v2.47.0's submodule config), needed to compile the suite's test-helpers at image build. Its C source is stripped from the agent-readable /app/tests.