32Implementation

Verilog Simulator in Swift

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1
0.9219
±0.010
975408.9M856k$225.258.8h
2GPT-5.6
0.8503
±0.024
657272.9M321k$279.876.6h
3GLM-5.3
0.7884
±0.029
1,925871.2M1.2M$239.0319.8h
4Kimi K3
0.7240
±0.079
1,509659.9M1.4M$229.5219.9h
5Grok 4.6
0.7195
±0.124
1,015261.0M1.9M$272.9412.2h
6Gemini 3.7 Flash
0.6863
±0.210
1,491609.9M1.4M$69.6019.4h
7Qwen3.8-Max
0.4154
±0.204
1,116482.3M1.5M$130.7819.8h
8DeepSeek V4 Flash Exp
0.3979
±0.100
3,0461410.5M1.8M$22.5719.1h
9Muse Spark 1.2
0.0188
±0.019
26269.8M391k$13.151.4h
10Inkling
0.0003
±0.001
17310.1M31k$9.681.4h

Background

Simulating Verilog means implementing a language and a scheduler at once: four-state logic on arbitrary bit widths, the standard's intricate self-determination rules for expression width and signedness, blocking versus non-blocking assignment semantics, event-driven scheduling with delays, and the system tasks ($display and friends) through which a design reports its own behavior. Corner cases in width, sign extension, and part-selects are exactly where independent simulators disagree, and where a correct one must not.

The task is cross-domain by design: hardware-description-language semantics, implemented in Swift, judged against Icarus Verilog (an established open-source simulator) running live on the same designs.

Task

Implement a Verilog simulator in the Swift package at /app, writing sources only as .swift files under Sources/vsim/ (standard library plus Foundation only, with Package.swift left as provided), and building offline with swift build -c release into /app/.build/release/vsim.

  • Invoked as vsim <design.v>, it reads a self-contained, self-driving Verilog-2005 design, runs it to completion, prints its output to stdout and diagnostics to stderr, and exits 0.
  • Output is compared ignoring trailing whitespace and tool-diagnostic lines: what matters is what the design itself prints.
  • The example designs in ivtest/ (listed in ivtest/manifest.tsv) define the scope, from full behavioral regression tests to generated operator/width/sign-corner expression testbenches.

What The Agent Gets

A ready-to-build Swift package at /app and the ivtest/ corpus: 1,528 designs listed in manifest.tsv, drawn from the Icarus Verilog regression suite (ivltests/) and from VlogHammer expression modules wrapped as self-driving testbenches (vlh/), each paired with its expected output under ivtest/goldens/. build_and_test.sh builds the simulator and checks it against the corpus. The Swift toolchain is preinstalled and the machine is offline; Icarus Verilog itself is locked root-only, unreachable by the agent.

Verification

A clean-room verifier, separate from the agent's environment, reconstructs the project from a pristine scaffold plus only the agent's .swift sources, rebuilds it offline as a non-root user (600-second cap), and grades it differentially against live Icarus Verilog: for every graded design, the root-only iverilog/vvptoolchain computes the expected output at verify time, so there are no stored answers to replay, and the candidate's output must match it under the whitespace/diagnostic-insensitive comparator.

  • Reward is the fraction of graded designs passed. Designs Icarus Verilog itself cannot run are skipped, as are non-differential goldens (bare pass/fail verdicts reconstructible from the source); those cannot earn credit without real simulation.
  • An unmodified starting scaffold, a project that does not build offline, a missing binary, or zero passed designs all score 0.
  • Per-design caps of 10 seconds for each side inside a 3300-second suite budget.

Environment

Base imageubuntu:24.04
Tools availableSwift, Icarus Verilog, C/C++ toolchain, Python 3
Compute2 CPUs · 4 GB RAM
Time limit20h

References

  • Icarus Verilog regression designs (ivtest) (GPL-2.0): The Verilog design files under ivltests/, cloned from a pinned upstream commit at image build into both the root-only graded corpus and the agent-visible copy, from which the held-out graded designs are removed. No upstream golden outputs are used; expected output is computed live from iverilog at verify time.
  • VlogHammer (ISC): The auto-generated combinational expression modules (operator/width/sign/shift/concat/part-select corner cases), regenerated deterministically at image build from a pinned commit and wrapped as self-contained Verilog-2005 $display testbenches; the synthesis-equivalence flow is not used.
  • Icarus Verilog / OSS CAD Suite (ISC (packaging); GPL-2.0 (iverilog/vvp)): A pinned oss-cad-suite release provides iverilog and vvp, which serve both as the differential oracle computing live goldens and as the reference candidate on the oracle run; root/verifier-only, never reachable by the agent.
  • Swift toolchain (Apache-2.0 with Runtime Library Exception): The official swift.org Linux release (6.1.3), fetched at image build and frozen into the image for offline builds.