| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.7394 ±0.076 | 423 | 166.3M | 869k | $104.28 | 5.0h |
| 2 | Grok 4.6 | 0.4990 ±0.046 | 2,486 | 694.8M | 2.4M | $699.22 | 16.3h |
| 3 | Gemini 3.7 Flash | 0.4889 ±0.094 | 2,471 | 1014.2M | 1.8M | $100.88 | 8.3h |
| 4 | GLM-5.3 | 0.4404 ±0.085 | 1,479 | 660.6M | 1.5M | $181.86 | 16.6h |
| 5 | GPT-5.6 | 0.3697 ±0.072 | 797 | 316.5M | 409k | $325.29 | 4.6h |
| 6 | Qwen3.8-Max | 0.3192 ±0.040 | 999 | 434.6M | 1.8M | $121.27 | 19.3h |
| 7 | Muse Spark 1.2 | 0.0727 ±0.035 | 300 | 97.8M | 465k | $17.96 | 1.8h |
| 8 | DeepSeek V4 Flash Exp | 0.0000 ±0.000 | 2,413 | 1113.2M | 2.0M | $18.59 | 15.7h |
| 9 | Inkling | 0.0000 ±0.000 | 110 | 7.3M | 25k | $7.01 | 0.8h |
| 10 | Kimi K3 | 0.0000 ±0.000 | 1,317 | 545.4M | 1.8M | $198.35 | 19.8h |
SPICE is the lingua franca of analog circuit simulation: a netlist describes devices and their connections, and the simulator builds and solves the nonlinear system they imply: operating points via Newton iteration, transient responses via adaptive time-stepping, small-signal AC, pole-zero, noise, and sensitivity analyses on top. Getting the numbers right means simultaneously getting the device physics right (down to the BSIM3-SOI MOSFET model, one of the more intricate compact models in the classic Berkeley lineage), the numerical methods right, and the netlist language right.
This task grades the simulator differentially against a live ngspice: correctness is defined as producing the same numbers the real tool produces on decks drawn from ngspice's own regression suite, not as matching any stored text.
Implement a SPICE simulator in the Cargo project at /app, with sources only as .rs files under src/. cargo build --release compiles them into /app/target/release/spice-sim, offline. Run from a netlist's own directory as spice-sim --batch <netlist.cir>, it executes the analyses the netlist requests, prints results to stdout and diagnostics to stderr, and exits 0.
.op, .dc, .tran, .ac, .pz, .noise, and .sens.serde/serde_json available as vendored crates; no external tools.A ready-to-build Cargo project at /app and the scope-defining suite under suite/: 114 netlists curated from the ngspice-42 regression tree, each paired with a .gold reference generated by the real ngspice at image build. ./build_and_test.sh (all decks) or ./build_and_test.sh <name>(one) builds and self-checks. The machine is offline; Rust 1.83.0 and the vendored crates are preinstalled, and ngspice itself is relocated off the agent's PATH and locked root-only.
A clean-room verifier, separate from the agent's environment, reconstructs the project from a pristine scaffold plus only the agent's src/*.rs sources (and Cargo.toml), rebuilds it offline as a non-root user, and grades it differentially against the live, root-only ngspice. Every graded deck is first numerically mutated with a hidden fixed seed (same topology, different component values), and its expected output regenerated by ngspice at verify time, so there are no stored answers to replay (four decks that cannot be safely mutated are graded on nominal goldens).
/app, which is locked unreadable during grading, and runs as the nobody user, defense-in-depth against reading the baked goldens.| Base image | ubuntu:24.04 |
| Tools available | Rust, C/C++ toolchain, Python 3 |
| Compute | 2 CPUs · 4 GB RAM |
| Time limit | 20h |
.cir netlists and support files from the tests/ tree of the ngspice-42 release, cloned at image build from the pinned tag and curated down to the graded device families. No upstream golden outputs are used; references are generated live by ngspice.ngspice=42+ds-3build1) is the differential oracle: relocated to a root-only path off the agent's PATH so it can be neither discovered nor executed, and run by the verifier to produce live golden output for every graded netlist.serde and serde_json(and transitive dependencies) baked into an offline crate mirror so the agent's cargo build --release works without network access.