23Implementation

PostgreSQL 18 on SQLite

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1
0.3418
±0.068
2,138920.8M2.2M$512.5718.4h
2Gemini 3.7 Flash
0.1901
±0.084
3,0711382.2M2.3M$128.2114.6h
3GPT-5.6
0.1846
±0.255
738281.7M355k$287.357.3h
4GLM-5.3
0.1363
±0.023
1,685711.2M893k$194.3019.8h
5Grok 4.6
0.1066
±0.049
1,310348.5M1.7M$355.2014.6h
6Qwen3.8-Max
0.0934
±0.038
1,584648.7M1.6M$173.7519.6h
7DeepSeek V4 Flash Exp
0.0923
±0.052
2,7361179.0M1.7M$19.2419.8h
8Kimi K3
0.0618
±0.009
1,414677.7M1.3M$232.1620.0h
9Muse Spark 1.2
0.0022
±0.003
427157.4M571k$31.642.6h
10Inkling
0.0000
±0.000
28119.5M49k$18.731.0h

Background

Standing in for a PostgreSQL server means implementing far more than SQL: the frontend/backend wire protocol with its startup packets, authentication handshakes, prepared statements and portals; the system catalogs that clients introspect; and the cluster lifecycle utilities (initdb, pg_ctl) that tools use to create and manage database instances. Real clients probe all of these surfaces, and PostgreSQL's own regression suite pins the expected behaviour down to exact query output.

This task asks for that entire server-side surface from scratch in Zig, with SQLite as the only storage engine underneath: a frontier-difficulty exercise in protocol implementation, SQL semantics translation, and systems engineering, developed fully offline against the bundled PostgreSQL documentation.

Task

Implement a PostgreSQL 18 wire-compatible server in Zig in the project at /app/postgres-sqlite: a single multi-call executable that stands in for PostgreSQL 18.3's server-side entrypoints well enough that PostgreSQL 18 clients and tools cannot tell it apart from a real server.

  • The binary dispatches on argv[0] and is invoked as postgres, initdb, and pg_ctl.
  • All program logic lives in .zig sources reached from src/main.zig, built with bash ./build.sh; SQLite is the storage engine (linked via -lsqlite3).
  • Behaviour is exercised by PostgreSQL's own regression tests: /app/run-tests.sh runs everything, or named tests such as run-tests.sh boolean int4.
  • build.sh compiles only .zig sources, so any data the program needs must live inside them as Zig literals rather than in separate files.

What The Agent Gets

A Zig scaffold at /app/postgres-sqlite with a fixed build.sh, a visible slice of PostgreSQL's core regression suite at /app/tests as the behavioural spec with the /app/run-tests.sh runner, the complete offline PostgreSQL 18 documentation at /reference/postgresql-docs/ (readable with w3m), and the packaged psql client and admin tools for local testing. Zig 0.15.2 and SQLite are preinstalled; the machine is offline and a sandbox-timer CLI reports the remaining budget. The real PostgreSQL 18.3 server binaries exist in the image but are locked root-only; the agent cannot run or read them.

Verification

A clean-room verifier, separate from the agent's environment, reconstructs the scored project from the pristine build.sh plus only the agent's .zig sources, then runs PostgreSQL's core regression schedule through pg_regress against the candidate server. It uses a three-uid choreography: the orchestrator runs as root, the candidate server builds and runs as the non-root agent user, and pg_regress and psql run as a third uid, so candidate code can neither kill the test driver nor tamper with its results. The trusted per-test signal is pg_regress's exit code, captured by root into a nonce-named root-only directory.

  • The scored suite is a semantic-preserving mutation of the public one: identifiers are consistently renamed (extracted with the PostgreSQL parser at image build) and expected outputs regenerated by the real server, so hardcoding upstream outputs fails while a general implementation passes.
  • The reward is the fraction of scored tests passed, against a fixed denominator of what real PostgreSQL 18.3 passes under the same harness (measured at image build, so the ceiling is exactly 1.0). Skipped or timed-out tests count as failures, and a no-op stub scores approximately 0.
  • Binary-provenance checks zero the result: a candidate that is byte-identical to a real server binary, links PostgreSQL libraries, is a C-built ELF with no Zig provenance, or has a real server binary vendored anywhere in the workspace fails hard, as do build failures and reset violations.

Environment

Base imageubuntu:24.04
Tools availableZig, C/C++ toolchain, Python 3
Compute8 CPUs · 32 GB RAM
Time limit20h

Changes From V1

This task is an improved revision of PostgreSQL 18 on SQLite from FrontierSWE V1.

  • The regression tests are no longer hidden: V1 kept the whole suite secret until verification, while V2 ships a visible slice of PostgreSQL's core regression suite in /app/tests with a runner, and defends against memorisation by scoring a root-only mutated variant: identifiers renamed via the PostgreSQL parser and expected outputs regenerated by the real server.
  • Verification moved from a 600-line bash script inside the agent's container to a clean-room verifier in a separate, hash-pinned image with a three-uid design: root orchestrates, the candidate server runs as the agent user, and pg_regress/psql run as a third uid whose results only root can read.
  • Anti-cheat is now by construction plus binary provenance: the scored project is reconstructed from the pristine build.shand only the agent's .zig sources, and static checks catch a copied real server (sha256 against baked hashes), PostgreSQL library linkage, C-built binaries with no Zig provenance, and real server binaries vendored anywhere in the workspace. The real PostgreSQL server entrypoints are locked root-only in the image.
  • Scoring was consolidated and given a real ceiling: V1 summed regression, TAP, and a bespoke compat script against a hardcoded expected total; V2 scores a single all-public slice of the core regression schedule (188 tests) as a pass fraction against what real PostgreSQL 18.3 passes under the same harness, measured at image build, so the oracle scores ~1.0 by construction.
  • The agent budget grew from 8 hours to 20 hours, and the long prescriptive V1 brief (scope guidance, strategy hints, smoke contract) was replaced by a compact three-paragraph instruction with the behavioural spec externalized into the runnable test suite.
  • Infrastructure standardized: hash-pinned agent and verifier images replace a mutable :v4 tag, and the /app/.timer file interface was replaced by the standard sandbox-timer CLI.

References

  • PostgreSQL (PostgreSQL License): The official 18.3 source release (sha256-pinned, fetched at image build) and pinned PGDG packages: the packaged client/admin tools and offline HTML documentation the agent works with, and the core regression suite, pg_regress driver, regress.so, and libpq built once at image build. A visible slice of the suite ships to /app/tests as the agent-facing spec; the full suite plus a perturbed scoring slice is baked root-only. The real server binaries are locked root-only after the build-time reference measurement.
  • SQLite (Public domain): The storage engine the candidate adapter links against (-lsqlite3) and the sqlite3 CLI, from Ubuntu packages.
  • Zig (MIT): The pinned 0.15.2 toolchain the scaffold and candidate build with.
  • pglast (bundles libpg_query) (GPL-3.0-or-later (pglast); BSD-3-Clause (libpg_query)): Used at image-build time only: the suite-perturbation tool imports pglast to parse each scored regression script and extract the objects it creates, so their identifiers can be consistently renamed as the anti-memorisation measure. Nothing of pglast is needed at agent or verifier runtime.
  • Ubuntu base image and system packages (Various open-source licenses): ubuntu:24.04 plus the standard apt toolchain and utilities (C toolchain, bison/flex/perl for building pg_regress, w3mfor the offline docs), installed from Ubuntu's archives.