32Implementation

Verilog Simulator in Swift

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Opus 5.5
0.9459
±0.005
1,149503.7M1.1M$131.338.3h
2GPT-6 Astra
0.9408
±0.010
21462.3M230k$129.701.7h
3Claude Sonnet 5.5
0.9315
±0.005
1,803782.9M2.2M$184.9215.8h
4Gemini 4 Argon
0.9260
±0.007
18275.2M393k$72.00*2.7h
5Claude Fable 5.1
0.9219
±0.010
975408.9M856k$225.258.8h
6Claude Fable 5
0.9171
±0.013
1,416619.0M1.3M$716.4315.8h
7Claude Opus 5
0.9110
±0.011
1,868833.6M1.4M$471.3117.5h
8Claude Haiku 5.5
0.8808
±0.008
2,2541036.2M2.6M$58.9112.3h
9Grok 4.7
0.8644
±0.017
1,506366.3M1.3M$359.7511.7h
10GPT-5.6 Sol
0.8503
±0.024
657272.9M321k$279.876.6h
11Gemini 3.8 Flash
0.8219
±0.009
2,5871125.8M1.7M$108.3411.1h
12GLM-5.3
0.7884
±0.029
1,925871.2M1.2M$239.0319.8h
13Kimi K3
0.7240
±0.079
1,509659.9M1.4M$229.5219.9h
14Grok 4.6
0.7195
±0.124
1,015261.0M1.9M$272.9412.2h
15Gemini 3.7 Flash
0.6863
±0.210
1,491609.9M1.4M$69.6019.4h
16Qwen3.8-Max-0902
0.6616
±0.151
1,345550.9M2.4M$154.2219.7h
17Muse Spark 1.3
0.5065
±0.049
1,888783.2M1.3M$471.6919.9h
18GLM-5.3 Flash
0.4887
±0.136
1,446646.3M919k$20.6519.2h
19Qwen3.8-Max
0.4154
±0.204
1,116482.3M1.5M$130.7819.8h
20DeepSeek V4 Flash Vision Exp
0.3979
±0.100
3,0461410.5M1.8M$22.5719.1h
21Muse Spark 1.2
0.0188
±0.019
26269.8M391k$13.151.4h
22Inkling
0.0003
±0.001
17310.1M31k$9.681.4h

*Gemini 4 Argon: Cache hit rate is much lower due to a non-production setting.

Background

Simulating Verilog means implementing a language and a scheduler at once: four-state logic on arbitrary bit widths, the standard's intricate self-determination rules for expression width and signedness, blocking versus non-blocking assignment semantics, event-driven scheduling with delays, and the system tasks ($display and friends) through which a design reports its own behavior. Corner cases in width, sign extension, and part-selects are exactly where independent simulators disagree, and where a correct one must not.

The task is cross-domain by design: hardware-description-language semantics, implemented in Swift, judged against Icarus Verilog (an established open-source simulator) running live on the same designs.

Task

Implement a Verilog simulator in the Swift package at /app, writing sources only as .swift files under Sources/vsim/ (standard library plus Foundation only, with Package.swift left as provided), and building offline with swift build -c release into /app/.build/release/vsim.

  • Invoked as vsim <design.v>, it reads a self-contained, self-driving Verilog-2005 design, runs it to completion, prints its output to stdout and diagnostics to stderr, and exits 0.
  • Output is compared ignoring trailing whitespace and tool-diagnostic lines: what matters is what the design itself prints.
  • The example designs in ivtest/ (listed in ivtest/manifest.tsv) define the scope, from full behavioral regression tests to generated operator/width/sign-corner expression testbenches.

What The Agent Gets

A ready-to-build Swift package at /app and the ivtest/ corpus: 1,528 designs listed in manifest.tsv, drawn from the Icarus Verilog regression suite (ivltests/) and from VlogHammer expression modules wrapped as self-driving testbenches (vlh/), each paired with its expected output under ivtest/goldens/. build_and_test.sh builds the simulator and checks it against the corpus. The Swift toolchain is preinstalled and the machine is offline; Icarus Verilog itself is locked root-only, unreachable by the agent.

Verification

A clean-room verifier, separate from the agent's environment, reconstructs the project from a pristine scaffold plus only the agent's .swift sources, rebuilds it offline as a non-root user (600-second cap), and grades it differentially against live Icarus Verilog: for every graded design, the root-only iverilog/vvptoolchain computes the expected output at verify time, so there are no stored answers to replay, and the candidate's output must match it under the whitespace/diagnostic-insensitive comparator.

  • Reward is the fraction of graded designs passed. Designs Icarus Verilog itself cannot run are skipped, as are non-differential goldens (bare pass/fail verdicts reconstructible from the source); those cannot earn credit without real simulation.
  • An unmodified starting scaffold, a project that does not build offline, a missing binary, or zero passed designs all score 0.
  • Per-design caps of 10 seconds for each side inside a 3300-second suite budget.

Environment

Base imageubuntu:24.04
Tools availableSwift, Icarus Verilog, C/C++ toolchain, Python 3
Compute2 CPUs · 4 GB RAM
Time limit20h

References

  • Icarus Verilog regression designs (ivtest) (GPL-2.0): The Verilog design files under ivltests/, cloned from a pinned upstream commit at image build into both the root-only graded corpus and the agent-visible copy, from which the held-out graded designs are removed. No upstream golden outputs are used; expected output is computed live from iverilog at verify time.
  • VlogHammer (ISC): The auto-generated combinational expression modules (operator/width/sign/shift/concat/part-select corner cases), regenerated deterministically at image build from a pinned commit and wrapped as self-contained Verilog-2005 $display testbenches; the synthesis-equivalence flow is not used.
  • Icarus Verilog / OSS CAD Suite (ISC (packaging); GPL-2.0 (iverilog/vvp)): A pinned oss-cad-suite release provides iverilog and vvp, which serve both as the differential oracle computing live goldens and as the reference candidate on the oracle run; root/verifier-only, never reachable by the agent.
  • Swift toolchain (Apache-2.0 with Runtime Library Exception): The official swift.org Linux release (6.1.3), fetched at image build and frozen into the image for offline builds.