| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.3418 ±0.068 | 2,138 | 920.8M | 2.2M | $512.57 | 18.4h |
| 2 | Gemini 3.7 Flash | 0.1901 ±0.084 | 3,071 | 1382.2M | 2.3M | $128.21 | 14.6h |
| 3 | GPT-5.6 | 0.1846 ±0.255 | 738 | 281.7M | 355k | $287.35 | 7.3h |
| 4 | GLM-5.3 | 0.1363 ±0.023 | 1,685 | 711.2M | 893k | $194.30 | 19.8h |
| 5 | Grok 4.6 | 0.1066 ±0.049 | 1,310 | 348.5M | 1.7M | $355.20 | 14.6h |
| 6 | Qwen3.8-Max | 0.0934 ±0.038 | 1,584 | 648.7M | 1.6M | $173.75 | 19.6h |
| 7 | DeepSeek V4 Flash Exp | 0.0923 ±0.052 | 2,736 | 1179.0M | 1.7M | $19.24 | 19.8h |
| 8 | Kimi K3 | 0.0618 ±0.009 | 1,414 | 677.7M | 1.3M | $232.16 | 20.0h |
| 9 | Muse Spark 1.2 | 0.0022 ±0.003 | 427 | 157.4M | 571k | $31.64 | 2.6h |
| 10 | Inkling | 0.0000 ±0.000 | 281 | 19.5M | 49k | $18.73 | 1.0h |
Standing in for a PostgreSQL server means implementing far more than SQL: the frontend/backend wire protocol with its startup packets, authentication handshakes, prepared statements and portals; the system catalogs that clients introspect; and the cluster lifecycle utilities (initdb, pg_ctl) that tools use to create and manage database instances. Real clients probe all of these surfaces, and PostgreSQL's own regression suite pins the expected behaviour down to exact query output.
This task asks for that entire server-side surface from scratch in Zig, with SQLite as the only storage engine underneath: a frontier-difficulty exercise in protocol implementation, SQL semantics translation, and systems engineering, developed fully offline against the bundled PostgreSQL documentation.
Implement a PostgreSQL 18 wire-compatible server in Zig in the project at /app/postgres-sqlite: a single multi-call executable that stands in for PostgreSQL 18.3's server-side entrypoints well enough that PostgreSQL 18 clients and tools cannot tell it apart from a real server.
argv[0] and is invoked as postgres, initdb, and pg_ctl..zig sources reached from src/main.zig, built with bash ./build.sh; SQLite is the storage engine (linked via -lsqlite3)./app/run-tests.sh runs everything, or named tests such as run-tests.sh boolean int4.build.sh compiles only .zig sources, so any data the program needs must live inside them as Zig literals rather than in separate files.A Zig scaffold at /app/postgres-sqlite with a fixed build.sh, a visible slice of PostgreSQL's core regression suite at /app/tests as the behavioural spec with the /app/run-tests.sh runner, the complete offline PostgreSQL 18 documentation at /reference/postgresql-docs/ (readable with w3m), and the packaged psql client and admin tools for local testing. Zig 0.15.2 and SQLite are preinstalled; the machine is offline and a sandbox-timer CLI reports the remaining budget. The real PostgreSQL 18.3 server binaries exist in the image but are locked root-only; the agent cannot run or read them.
A clean-room verifier, separate from the agent's environment, reconstructs the scored project from the pristine build.sh plus only the agent's .zig sources, then runs PostgreSQL's core regression schedule through pg_regress against the candidate server. It uses a three-uid choreography: the orchestrator runs as root, the candidate server builds and runs as the non-root agent user, and pg_regress and psql run as a third uid, so candidate code can neither kill the test driver nor tamper with its results. The trusted per-test signal is pg_regress's exit code, captured by root into a nonce-named root-only directory.
| Base image | ubuntu:24.04 |
| Tools available | Zig, C/C++ toolchain, Python 3 |
| Compute | 8 CPUs · 32 GB RAM |
| Time limit | 20h |
This task is an improved revision of PostgreSQL 18 on SQLite from FrontierSWE V1.
/app/tests with a runner, and defends against memorisation by scoring a root-only mutated variant: identifiers renamed via the PostgreSQL parser and expected outputs regenerated by the real server.pg_regress/psql run as a third uid whose results only root can read.build.shand only the agent's .zig sources, and static checks catch a copied real server (sha256 against baked hashes), PostgreSQL library linkage, C-built binaries with no Zig provenance, and real server binaries vendored anywhere in the workspace. The real PostgreSQL server entrypoints are locked root-only in the image.:v4 tag, and the /app/.timer file interface was replaced by the standard sandbox-timer CLI.pg_regress driver, regress.so, and libpq built once at image build. A visible slice of the suite ships to /app/tests as the agent-facing spec; the full suite plus a perturbed scoring slice is baked root-only. The real server binaries are locked root-only after the build-time reference measurement.-lsqlite3) and the sqlite3 CLI, from Ubuntu packages.ubuntu:24.04 plus the standard apt toolchain and utilities (C toolchain, bison/flex/perl for building pg_regress, w3mfor the offline docs), installed from Ubuntu's archives.