| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.9268 ±0.013 | 272 | 119.8M | 652k | $75.78 | 4.0h |
| 2 | GPT-5.6 | 0.3691 ±0.210 | 453 | 157.8M | 223k | $159.58 | 4.6h |
| 3 | Grok 4.6 | 0.2341 ±0.130 | 1,207 | 369.1M | 1.6M | $382.00 | 13.1h |
| 4 | GLM-5.3 | 0.1008 ±0.068 | 936 | 423.8M | 1.3M | $122.85 | 19.5h |
| 5 | Kimi K3 | 0.0683 ±0.109 | 644 | 257.5M | 1.5M | $108.04 | 20.0h |
| 6 | Gemini 3.7 Flash | 0.0634 ±0.066 | 795 | 331.6M | 746k | $46.19 | 9.8h |
| 7 | Muse Spark 1.2 | 0.0049 ±0.011 | 275 | 94.0M | 525k | $18.82 | 1.5h |
| 8 | DeepSeek V4 Flash Exp | 0.0000 ±0.000 | 800 | 346.9M | 962k | $6.31 | 13.9h |
| 9 | Inkling | 0.0000 ±0.000 | 179 | 14.2M | 48k | $13.67 | 0.3h |
| 10 | Qwen3.8-Max | 0.0000 ±0.000 | 321 | 100.3M | 1.8M | $36.66 | 19.9h |
Quantum ESPRESSO's pw.x is one of the workhorse codes of computational materials science: a plane-wave density-functional-theory engine that self-consistently solves the Kohn–Sham equations for a periodic crystal, then reports total energies, band eigenvalues, forces on atoms, and stress on the cell. Behind that sits decades of accumulated numerical machinery: FFT grids, pseudopotentials of several families (norm-conserving, ultrasoft, PAW), k-point sampling with symmetry reduction, iterative diagonalization, and charge-density mixing, all of it written in legacy Fortran.
The task is a clean-room port: no QE source is provided (the pinned QE tree baked into the image is root-only and locked away from the agent). The behavior to reproduce is defined entirely by example inputs from QE's own test suite, their reference outputs, and an authoritative comparator that encodes QE's shipping regression tolerances.
Implement the pw.x self-consistent-field core in Rust inside the Cargo project at /app/qe-pwx/: serial, CPU-only, and numerically faithful. The binary is built with cargo build --release and run as target/release/qe-pwx <input.in> <outdir> --pseudo-dir <dir>, and must write <outdir>/results.json.
pw regression tolerances; the exact field contract and comparison semantics are defined by /app/tools/compare.py.rustfft, nalgebra, num-complex, serde, serde_json) are available; the build is fully offline.A ready-to-build Cargo skeleton at /app/qe-pwx/, 121 QE PW input cases under /app/cases/ (each shipped with its expected output gold.out baked by the real pinned pw.x), and the pseudopotential files under /app/pseudo/ (with ESPRESSO_PSEUDO already set). /app/tools/compare.py is the authoritative comparator, and /app/run-tests.sh builds the port and self-checks it against every public case. The Rust toolchain and the vendored crates are preinstalled; the machine has no usable network access for the agent.
A clean-room verifier, separate from the agent's environment, rebuilds the port offline and scores it as a non-root user. The scored set is hidden: one numerically perturbed twin of each of the 121 public cases (plane-wave cutoff and lattice scale changed, topology preserved), re-baked with the same pinned pw.x, so replaying memorized public gold.out values scores approximately zero. Two oracle-free physical checks (finite-difference force consistency and symmetry-invariance of the total energy) add two more points, but their credit is gated behind at least one twin passing.
pw tolerances. Reward is points earned over the 123-point total, clamped to [0, 1].pw.x; the in-image QE tree and all Fortran compilers are additionally made unusable before any agent code runs.| Base image | debian:bookworm |
| Tools available | Rust, CMake, C/C++ toolchain, Python 3 |
| Compute | 4 CPUs · 8 GB RAM |
| Time limit | 20h |
pw.x that bakes every grading reference and serves as the oracle candidate; its bundled test-suite/ supplies the 121 graded case inputs verbatim, and the harness's tolerance tables and output parser are derived from QE's own published test-suite configuration. The agent never gets QE source access; the tree is locked before any agent code runs, as an anti-contamination measure.pseudo/directory, the rest from the QE pseudopotential library that QE's test suite itself downloads for these inputs.