29Implementation

SPICE Circuit Simulator in Rust

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1
0.7394
±0.076
423166.3M869k$104.285.0h
2Grok 4.6
0.4990
±0.046
2,486694.8M2.4M$699.2216.3h
3Gemini 3.7 Flash
0.4889
±0.094
2,4711014.2M1.8M$100.888.3h
4GLM-5.3
0.4404
±0.085
1,479660.6M1.5M$181.8616.6h
5GPT-5.6
0.3697
±0.072
797316.5M409k$325.294.6h
6Qwen3.8-Max
0.3192
±0.040
999434.6M1.8M$121.2719.3h
7Muse Spark 1.2
0.0727
±0.035
30097.8M465k$17.961.8h
8DeepSeek V4 Flash Exp
0.0000
±0.000
2,4131113.2M2.0M$18.5915.7h
9Inkling
0.0000
±0.000
1107.3M25k$7.010.8h
10Kimi K3
0.0000
±0.000
1,317545.4M1.8M$198.3519.8h

Background

SPICE is the lingua franca of analog circuit simulation: a netlist describes devices and their connections, and the simulator builds and solves the nonlinear system they imply: operating points via Newton iteration, transient responses via adaptive time-stepping, small-signal AC, pole-zero, noise, and sensitivity analyses on top. Getting the numbers right means simultaneously getting the device physics right (down to the BSIM3-SOI MOSFET model, one of the more intricate compact models in the classic Berkeley lineage), the numerical methods right, and the netlist language right.

This task grades the simulator differentially against a live ngspice: correctness is defined as producing the same numbers the real tool produces on decks drawn from ngspice's own regression suite, not as matching any stored text.

Task

Implement a SPICE simulator in the Cargo project at /app, with sources only as .rs files under src/. cargo build --release compiles them into /app/target/release/spice-sim, offline. Run from a netlist's own directory as spice-sim --batch <netlist.cir>, it executes the analyses the netlist requests, prints results to stdout and diagnostics to stderr, and exits 0.

  • Support the standard analyses: .op, .dc, .tran, .ac, .pz, .noise, and .sens.
  • Cover the BSIM3-SOI MOSFET model and the classic Berkeley SPICE3 device set; modern compact models (BSIM4, HiSIM, HICUM, BSIMSOI) are out of scope.
  • Match ngspice numerically, not byte-for-byte; table layout does not matter.
  • Standard-library Rust only, with serde/serde_json available as vendored crates; no external tools.

What The Agent Gets

A ready-to-build Cargo project at /app and the scope-defining suite under suite/: 114 netlists curated from the ngspice-42 regression tree, each paired with a .gold reference generated by the real ngspice at image build. ./build_and_test.sh (all decks) or ./build_and_test.sh <name>(one) builds and self-checks. The machine is offline; Rust 1.83.0 and the vendored crates are preinstalled, and ngspice itself is relocated off the agent's PATH and locked root-only.

Verification

A clean-room verifier, separate from the agent's environment, reconstructs the project from a pristine scaffold plus only the agent's src/*.rs sources (and Cargo.toml), rebuilds it offline as a non-root user, and grades it differentially against the live, root-only ngspice. Every graded deck is first numerically mutated with a hidden fixed seed (same topology, different component values), and its expected output regenerated by ngspice at verify time, so there are no stored answers to replay (four decks that cannot be safely mutated are graded on nominal goldens).

  • The comparator matches computed values, not text: waveforms under calibrated RMS and pointwise tolerances, scalars under relative tolerance.
  • Reward is the fraction of graded decks whose numbers match; build failure, a missing binary, or matching zero decks all score 0.
  • The candidate binary is copied out of /app, which is locked unreadable during grading, and runs as the nobody user, defense-in-depth against reading the baked goldens.
  • Per-deck caps of 60 s (ngspice) and 90 s (candidate) inside a 3000-second suite budget; the rebuild is capped at 600 s.

Environment

Base imageubuntu:24.04
Tools availableRust, C/C++ toolchain, Python 3
Compute2 CPUs · 4 GB RAM
Time limit20h

References

  • ngspice regression test suite (BSD-3-Clause (project license; mixed open-source components)): The .cir netlists and support files from the tests/ tree of the ngspice-42 release, cloned at image build from the pinned tag and curated down to the graded device families. No upstream golden outputs are used; references are generated live by ngspice.
  • ngspice (BSD-3-Clause (project license; mixed open-source components)): The Ubuntu 24.04 package (ngspice=42+ds-3build1) is the differential oracle: relocated to a root-only path off the agent's PATH so it can be neither discovered nor executed, and run by the verifier to produce live golden output for every graded netlist.
  • Rust toolchain and vendored crates (MIT OR Apache-2.0 (plus permissive licenses on transitive deps)): Rust 1.83.0 via rustup, with serde and serde_json(and transitive dependencies) baked into an offline crate mirror so the agent's cargo build --release works without network access.