| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.9219 ±0.010 | 975 | 408.9M | 856k | $225.25 | 8.8h |
| 2 | GPT-5.6 | 0.8503 ±0.024 | 657 | 272.9M | 321k | $279.87 | 6.6h |
| 3 | GLM-5.3 | 0.7884 ±0.029 | 1,925 | 871.2M | 1.2M | $239.03 | 19.8h |
| 4 | Kimi K3 | 0.7240 ±0.079 | 1,509 | 659.9M | 1.4M | $229.52 | 19.9h |
| 5 | Grok 4.6 | 0.7195 ±0.124 | 1,015 | 261.0M | 1.9M | $272.94 | 12.2h |
| 6 | Gemini 3.7 Flash | 0.6863 ±0.210 | 1,491 | 609.9M | 1.4M | $69.60 | 19.4h |
| 7 | Qwen3.8-Max | 0.4154 ±0.204 | 1,116 | 482.3M | 1.5M | $130.78 | 19.8h |
| 8 | DeepSeek V4 Flash Exp | 0.3979 ±0.100 | 3,046 | 1410.5M | 1.8M | $22.57 | 19.1h |
| 9 | Muse Spark 1.2 | 0.0188 ±0.019 | 262 | 69.8M | 391k | $13.15 | 1.4h |
| 10 | Inkling | 0.0003 ±0.001 | 173 | 10.1M | 31k | $9.68 | 1.4h |
Simulating Verilog means implementing a language and a scheduler at once: four-state logic on arbitrary bit widths, the standard's intricate self-determination rules for expression width and signedness, blocking versus non-blocking assignment semantics, event-driven scheduling with delays, and the system tasks ($display and friends) through which a design reports its own behavior. Corner cases in width, sign extension, and part-selects are exactly where independent simulators disagree, and where a correct one must not.
The task is cross-domain by design: hardware-description-language semantics, implemented in Swift, judged against Icarus Verilog (an established open-source simulator) running live on the same designs.
Implement a Verilog simulator in the Swift package at /app, writing sources only as .swift files under Sources/vsim/ (standard library plus Foundation only, with Package.swift left as provided), and building offline with swift build -c release into /app/.build/release/vsim.
vsim <design.v>, it reads a self-contained, self-driving Verilog-2005 design, runs it to completion, prints its output to stdout and diagnostics to stderr, and exits 0.ivtest/ (listed in ivtest/manifest.tsv) define the scope, from full behavioral regression tests to generated operator/width/sign-corner expression testbenches.A ready-to-build Swift package at /app and the ivtest/ corpus: 1,528 designs listed in manifest.tsv, drawn from the Icarus Verilog regression suite (ivltests/) and from VlogHammer expression modules wrapped as self-driving testbenches (vlh/), each paired with its expected output under ivtest/goldens/. build_and_test.sh builds the simulator and checks it against the corpus. The Swift toolchain is preinstalled and the machine is offline; Icarus Verilog itself is locked root-only, unreachable by the agent.
A clean-room verifier, separate from the agent's environment, reconstructs the project from a pristine scaffold plus only the agent's .swift sources, rebuilds it offline as a non-root user (600-second cap), and grades it differentially against live Icarus Verilog: for every graded design, the root-only iverilog/vvptoolchain computes the expected output at verify time, so there are no stored answers to replay, and the candidate's output must match it under the whitespace/diagnostic-insensitive comparator.
| Base image | ubuntu:24.04 |
| Tools available | Swift, Icarus Verilog, C/C++ toolchain, Python 3 |
| Compute | 2 CPUs · 4 GB RAM |
| Time limit | 20h |
ivltests/, cloned from a pinned upstream commit at image build into both the root-only graded corpus and the agent-visible copy, from which the held-out graded designs are removed. No upstream golden outputs are used; expected output is computed live from iverilog at verify time.$display testbenches; the synthesis-equivalence flow is not used.oss-cad-suite release provides iverilog and vvp, which serve both as the differential oracle computing live goldens and as the reference candidate on the oracle run; root/verifier-only, never reachable by the agent.