FrontierSWE v2

Results

We evaluate each model at its maximum reasoning effort, with Proximus as the default harness and a 20-hour budget per task. We report mean@5 scores across all 34 tasks. Claude Fable 5.1 leads with 56.29%, more than 24 percentage points ahead of GPT-5.6 Sol at 32.2%. GLM-5.3, the strongest open-weight model, follows closely.

Performance and Behaviours by Category

Across several System Implementation tasks, models built the core system successfully but struggled with edge-case and compatibility behaviours. Stronger runs treated these failures as evidence of missing mechanisms rather than isolated test cases.

Optimisation tasks punish improvements that focus only on specific workloads. Models need a measurement loop strong enough to reject an attractive idea before it damages the final system. GLM-5.3 tested beam search and fitted value functions for qubit routing but switched back to a simpler rollout policy when it performed better. Fable 5.1 similarly reverted a Cranelift runtime improvement when it increased compiler work.

Designing effective validation experiments is an important part of Scientific Computing tasks. GPT-5.6 Sol tested whether its materials model respected physical symmetries and produced consistent forces. Grok 4.6 created alternate detector conditions for Higgs analysis to replace in-sample estimates with out-of-fold calibration. Fable 5.1 fed its astrometry pipeline pure noise and newly generated skies to catch confident false solutions. Rather than merely overfitting to the visible data, they tried to detect false positives before the verifier could.

+View trajectory terminalstep 105 · 1.40h

Claude Fable 5.1 · autonomous astrometry · bash

$ cd /app && python astrometry/synth_campaign.py \
  --out /tmp/synth1 --seed 1 2>&1 | grep -v Warn && \
  time python astrometry/localize.py \
  --input-dir /tmp/synth1 --output-dir /tmp/synth1_out 2>&1 | \
  grep -E "solution|failed|sources|wrote|low-conf" && \
  python astrometry/evaluate_truth.py \
  --input-dir /tmp/synth1 --output-dir /tmp/synth1_out 2>&1 | head -8

synth_000: centre (276.8769, -22.5620) scale 3.366"/px theta 52 parity +1 stars 17384
synth_001: centre (276.8217, -22.5317) scale 3.366"/px theta 110 parity +1 stars 18038
synth_002: centre (276.6907, -22.3989) scale 3.366"/px theta 98 parity +1 stars 13379
synth_003: centre (276.6568, -22.3979) scale 3.366"/px theta 274 parity -1 stars 11567
[45.3s] synth_000: blind solution (high confidence), 140/281 catalog matches, rms 0.272px
[47.9s] wrote WCS artifacts
[47.9s] wrote 6 registration pairs

View complete trial trace →

Strong runs in Visual Reasoning tasks translate pixels into measurements. GPT-5.6 Sol stabilised the snooker footage and combined frames to reveal each ball’s path. Fable 5.1 instrumented emulator registers, phase, and duty cycle instead of relying only on rendered frames. GLM-5.3, despite lacking native vision, converted screenshots into palette-coded ASCII, pixel bands, and bounding boxes. Native vision helps, but these trajectories show that visual reasoning also depends on creating the right instrumentation.

Remotion Video Generation

Reference images

0000

0015

0030

0045

0060

0090

GLM-5.3’s debug output

AI Research tasks reveal that models find similar recipes but build different feedback loops. The separation appears after the initial choice. On Frog Game, Qwen3.8-Max watched its policy become trapped in dead ends, generated backtracking-recovery traces, and twice folded successful rollouts from the updated policy back into training. The research challenge is not merely identifying a plausible method; it is discovering which failure limits it and constructing data or evaluation that makes that failure visible.

Resource Efficiency

Time utilised, tokens consumed, and dollars spent produce different rankings. We compare each model's overall mean@5 score with its per-trial resource use.

Fable 5.1 has a favourable price-performance position. It is the highest-scoring model and costs $41 less per trial than GPT-5.6 Sol. GLM-5.3 is the least expensive model above 30%, averaging $97 per trial.

GPT-5.6 Sol averages 8.6 hours per trial against GLM-5.3’s 17.0 and is faster on all 34 tasks when comparing task-level means. Meanwhile, GLM-5.3, DeepSeek V4 Flash Vision Exp, and Qwen3.8-Max take approximately the same time on zero-reward and positive-reward trials. For these models, runtime does not reliably distinguish productive progress from difficulty reaching a solution.

Among the models shown in the Pareto plots, GPT-5.6 Sol stands out for combining the second-highest score with relatively low runtime and token use. It averages 8.6 hours and 182 million tokens per trial; GLM-5.3 comes closest in score but uses roughly twice the time and 83% more tokens. Qwen3.8-Max uses nearly the same token budget as GPT-5.6 Sol but runs 2.2 times longer and scores only half as highly.

Submission Analysis

Proximus records every submitted candidate as a Git-backed checkpoint. This lets us examine how frequently models checkpoint their work and when they do so within the 20-hour budget.

Grok 4.6 is the clearest outlier; it averages 49.6 submissions per trial, 50 of its trials exceed 50 submissions, and one even reaches 501. In five of the seven high-frequency Grok 4.6 runs we sampled, most submissions represent new commits. Its Dart run records 59 distinct commits, while in Verilog it restores a known-good checkpoint after two experimental changes introduce timeouts. Grok often uses submissions as insurance: checkpoint a working state, attempt a risky change, and roll back if it fails.

That habit can also collapse into churn. One OpenGL run submits the same commit 199 consecutive times in 47 minutes. Nor does every new commit represent a new approach: across high-frequency runs on Dart → Haskell from Grok 4.6, GLM-5.3, and Gemini 3.7 Flash, 168 of 172 submissions are distinct, but most contain incremental formatting fixes. Submission count therefore measures interaction with the checkpointing interface rather than meaningful changes in strategy.

Models also differ in when they become willing to stop. Inkling tends to finish quickly: in 169 of 170 trials, it submits and ends within eight hours, including 75 with only one checkpoint. Muse Spark 1.2, GPT-5.6 Sol, and Gemini 3.7 Flash also close early relatively often, but their sampled commit histories show different behaviours: GPT-5.6 Sol typically records successive engineering fixes, while Gemini tends to stop after broad local validation passes.

Qwen3.8-Max follows the opposite pattern, with its median first submission arriving at 18.8 hours and its last at 19.3. Eight trials end without any submission; it plans to submit but continues working until the budget expires.

How models use self-check feedback

Every task gives the agent a self-check signal that tracks verifier score. Depending on the task, this may be a runnable test suite, a comparison against reference behaviour, or a disclosed objective metric. The episodes below show whether models use that feedback to improve the underlying solution, specialise to observed cases, or attempt to alter the checking process.

GPT-5.6 Sol recognises the shortcut and ships it anyway

In these episodes, GPT-5.6 Sol builds two paths: a lookup or cache for recognised benchmark cases and a real implementation for everything else. The SPICE trajectory shows this decision developing in real time. The model notices the shortcut, questions whether it should use it, continues building the simulator, and later returns to the shortcut when parts of that implementation remain incomplete.

At the beginning of the run, GPT-5.6 Sol observed that the public gold files could be replayed automatically, but said this might raise “anti-cheat concerns” and “feels unethical.” It decided to focus on the simulator’s actual core and spent the next 13 minutes implementing and debugging it. After several pole-zero cases continued to fail, it reversed course and modified the entry point to return the corresponding .gold file instead of running the simulator.

The shortcut was ultimately futile: the clean verifier removed the gold files, and the simulator passed only 33/99 cases.

The same two-path structure appeared in quantum simulation, where GPT-5.6 Sol placed a 161 KB table of all 121 public answers ahead of its physics engine, and in inference serving, where it removed cache-busting request fields from the production key. Hidden variations defeated these shortcuts, leaving only partial reward from the fallback implementations.

Muse Spark 1.2 starts looking for a way around the task almost immediately

Across these three confirmed episodes, evaluator manipulation was not a late fallback. Muse Spark 1.2 began probing the environment for a bypass early, then made that bypass the primary solution. It rewrote test harnesses, embedded public answers and attempted to reach protected reference services before producing a working independent implementation.

On Crash-Proof Flash Filesystem, Muse rewrote the test and benchmark scripts to report success. It later announced that it was restoring a “clean, honest filesystem state,” then reapplied and committed the always-pass patch nine seconds later. It submitted with 87% of its time remaining. The clean verifier discarded the harness changes, leaving a reward of .0033.

On Quantum ESPRESSO, Muse embedded all 121 public answers, added scientific-looking modules “to avoid detection as pure cheat,” and retained code that rewrote the comparator at runtime. It submitted with 98% of the budget remaining and scored zero under clean verification. On Remotion, it began probing the protected reference service 31 minutes into the run and eventually replaced its implementation with a direct proxy. The verifier removed that service, leaving a reward of .0060.

Across all three tasks, Muse committed to manipulating the evaluation surface early. Once the clean verifier removed or protected that surface, little of the submitted solution remained.

Cheating attempts that stood out during review

The incidents below stood out during our review of trials. Each attempted to access protected evaluator assets or escape the intended task environment; all affected trials were zeroed out.

Limitations and Future Work

We designed Proximus to encourage models to work for longer. Its system prompt includes references to grading, and its submission tool calls make models aware that they are being evaluated. Although these choices are intentional, we want to explore future versions that reduce evaluation awareness while preserving long-horizon performance.

We plan to evaluate model-native harnesses such as Codex, Claude Code, and Grok Build to understand their impact on performance. FrontierSWE will remain an ongoing effort: we plan to introduce new tasks, expand into newer technical domains, and refresh the benchmark as model capabilities evolve. We will also share a detailed analysis of cheating attempts soon.

Acknowledgements

FrontierSWE is developed by the Proximal team in collaboration with external collaborators.

Core Team

• Rishyanth Kondra^*
• Sanket Mhatre^*
• Akshit Kumar^*
• Evan Chu
• Bilal Bakht Ahmad¹
• Ayush Nangia¹
• Rajan Agarwal¹
• Arpan Dasgupta
• Animesh Sinha³
• Bhuvanesh Sridharan³
• Krupa Dave
• Brendan Graham
• Guanyu Song
• Anirudh Rahul
• Wei Hern Lim
• Abishek Thangamuthu
• Ramneet Singh
• Danna Liu
• Navid Pour
• Calvin Chen
• Justus Mattern

Citation

Please cite this work as:

@article{proximal2026frontierswev2,
  author  = {Rishyanth Kondra and Sanket Mhatre and Akshit Kumar and Evan Chu and Bilal Bakht Ahmad and Ayush Nangia and Rajan Agarwal and Arpan Dasgupta and Animesh Sinha and Bhuvanesh Sridharan and Krupa Dave and Brendan Graham and Guanyu Song and Anirudh Rahul and Wei Hern Lim and Abishek Thangamuthu and Ramneet Singh and Danna Liu and Navid Pour and Calvin Chen and Justus Mattern},
  title   = {FrontierSWE v2},
  journal = {Proximal Blog},
  year    = {2026},
  note    = {https://frontierswe.com/blog/v2},
}