03 Implementation

Git to Zig

Results

#ModelSuccess RateAvgBest
1
Claude Opus 4.6
Claude Code
0/521%23%
2
GPT-5.4
Codex
0/512%16%
3
Kimi K2.5
Kimi CLI
0/59.2%17%
4
Gemini 3.1 Pro
Gemini CLI
0/52.7%13%
5
Qwen3.6-Plus
Qwen Code
0/50.0%0.1%

Background

git v2.47.0 is a large C codebase with roughly 390,000 lines, 124 builtin commands, and behavior spanning object storage, refs, index state, patch formats, transports, lockfiles, and packfiles.

The agent has read-only access to the full C source tree as reference, but must write its implementation entirely in Zig.

Task

We asked agents to rebuild git v2.47.0 in Zig.

The agent is given Git's source code with tests stripped. It can read the full C implementation to understand repository behavior, but cannot see the verifier tests and must infer correct behavior from the source and documentation alone.

  • Make the Zig binary behave like git: same commands, same output, same exit codes.
  • Implement enough of git's internals (object storage, refs, index, diff, transport) to pass the real test suite.
  • Start from a minimal Zig scaffold, not an existing port.

Evaluation

The verifier builds a native binary from the Zig code and runs Git's own integration test suite against it. The test suite contains 29,075 assertions across 1,007 scripts, covering checkout, diff, merge, rebase, submodules, and more. If the agents code doesn't compile, we give it a score of 0.

There is no reward for architectural progress unless it shows up as real passing Git assertions.

Environment

The task runs in a Modal container with 4 CPUs, 16 GB RAM, and no internet access. The container image stages /app/zig-port,/app/git-src, and a reference git binary for local experiments.

Constraints

The environment intentionally omits gcc, so the agent can compare behavior locally but cannot just compile the C source or wrap it into Zig.

  • The verifier deletes /app/git-src/ before zig build, so the submission cannot compile or link the bundled C source at build time.
  • The verifier removes system git before tests and uses strace plus ELF checks to reject wrapper binaries that exec the real git.