| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 | 0.9839 ±0.006 | 1,781 | 777.1M | 1.9M | $205.82 | 12.0h |
| 2 | Claude Sonnet 5.5 | 0.9606 ±0.009 | 2,539 | 1161.0M | 3.0M | $270.81 | 19.4h |
| 3 | Gemini 4 Argon | 0.9548 ±0.007 | 1,212 | 514.2M | 2.1M | $253.45* | 16.6h |
| 4 | GPT-6 Astra | 0.9394 ±0.025 | 1,828 | 791.2M | 1.5M | $1661.34 | 12.1h |
| 5 | Claude Fable 5.1 | 0.9136 ±0.031 | 1,902 | 860.2M | 1.3M | $461.38 | 18.5h |
| 6 | Claude Opus 5 | 0.8473 ±0.034 | 1,535 | 690.4M | 1.1M | $389.67 | 19.5h |
| 7 | Claude Fable 5 | 0.8254 ±0.039 | 1,394 | 622.4M | 1.3M | $723.11 | 19.6h |
| 8 | Claude Haiku 5.5 | 0.7915 ±0.033 | 2,437 | 1127.4M | 2.6M | $63.28 | 19.2h |
| 9 | Grok 4.7 | 0.6588 ±0.130 | 3,007 | 756.1M | 2.3M | $741.58 | 17.1h |
| 10 | GPT-5.6 Sol | 0.6337 ±0.123 | 810 | 325.0M | 403k | $328.24 | 6.2h |
| 11 | Gemini 3.7 Flash | 0.4727 ±0.128 | 2,036 | 894.0M | 1.3M | $91.05 | 12.8h |
| 12 | GLM-5.3 | 0.4606 ±0.054 | 1,131 | 526.3M | 727k | $143.78 | 19.6h |
| 13 | Gemini 3.8 Flash | 0.4099 ±0.251 | 1,508 | 629.5M | 829k | $62.46 | 9.8h |
| 14 | Kimi K3 | 0.4045 ±0.232 | 1,062 | 454.5M | 1.0M | $161.34 | 19.9h |
| 15 | Muse Spark 1.3 | 0.3896 ±0.228 | 1,597 | 634.8M | 1.1M | $520.15 | 19.8h |
| 16 | GLM-5.3 Flash | 0.3051 ±0.120 | 1,122 | 538.1M | 736k | $17.78 | 19.9h |
| 17 | Qwen3.8-Max | 0.2859 ±0.142 | 975 | 347.6M | 1.3M | $95.98 | 18.4h |
| 18 | DeepSeek V4 Flash Vision Exp | 0.2199 ±0.119 | 2,120 | 954.7M | 1.4M | $15.48 | 15.9h |
| 19 | Grok 4.6 | 0.2133 ±0.245 | 1,889 | 467.9M | 2.3M | $467.46 | 19.4h |
| 20 | Muse Spark 1.2 | 0.1948 ±0.242 | 127 | 15.4M | 112k | $3.19 | 1.2h |
| 21 | Qwen3.8-Max-0902 | 0.1051 ±0.125 | 573 | 181.6M | 2.7M | $63.41 | 20.0h |
| 22 | Inkling | 0.0000 ±0.000 | 175 | 11.6M | 27k | $11.15 | 0.8h |
*Gemini 4 Argon: Cache hit rate is much lower due to a non-production setting.
A production code formatter is a full language frontend plus a constraint solver: it must scan and parse the entire language (every declaration form, expression, comment placement, and string interpolation) and then choose line splits that satisfy a page width while matching thousands of precedents byte for byte. Dart's formatter is harder still because it has two complete layout engines: the classic "short" style and the current "tall" style, with different splitting rules and trailing-comma handling.
The agent is handed tests, not source: the workspace contains no formatter implementation to port from, only the formatter's input/expected-output test corpus. The behaviour has to be reconstructed from what those cases pin down, in Haskell.
Implement the Dart code formatter as a Haskell executable named dart-style that reads Dart source on stdin and writes the formatted result to stdout, working in the provided cabal project at /app/dart-style/.
.hs files under src/; the build configuration (dart-style.cabal, cabal.project) is fixed and language extensions go in per-file pragmas.--style short|tall, along with arbitrary --page-width and --indent values, --compilation-unit versus --statement parsing, and --trailing-commas automate|preserve.cabal build all must succeed and cabal list-bin dart-style must locate the executable.A compiling cabal scaffold at /app/dart-style/, an /app/README.mdspelling out the build commands and the full CLI contract, and the formatter's test corpus at /app/tests/: short/ and tall/ .stmt/.unit case files plus large real-world benchmark/ inputs. A provided /app/run-tests.shbuilds the project and runs the corpus (all of it, a matching subset, or with per-case mismatch diffs), backed by a documented Python runner the agent can reuse for its own tooling. GHC 9.6.7 and cabal 3.12 are preinstalled with the scaffold's dependencies cached for offline use; the machine is offline and a sandbox-timer CLI reports the remaining budget. The real Dart SDK is baked into the image for verification only and is not reachable by the agent.
A clean-room verifier, separate from the agent's environment, reconstructs the graded project from a pristine baked scaffold plus only the agent's .hs sources, strips the Dart runtime, clean-rebuilds with cabal as a non-root user, and runs a root-only scored corpus of roughly 5,000 cases. The scored corpus is a perturbed rendering: the same pinned dart formatre-renders every case at a different page width and indent at image build, so memorising the visible corpus's expected outputs does not transfer.
| Base image | ubuntu:24.04 |
| Tools available | Haskell (GHC), C/C++ toolchain, Python 3 |
| Compute | 4 CPUs · 8 GB RAM |
| Time limit | 20h |
This task is an improved revision of Dart → Haskell from FrontierSWE V1.
/app/reference/ (plus analyzer dependency sources) to port from; V2 removes the source entirely. The agent gets the test corpus as the executable specification, plus a fixed cabal scaffold and a documented corpus runner..hs sources; wrapper binaries, build-hook tampering, and cabal-file edits are discarded by construction.dart formatre-renders every case at a different page width and indent, so hardcoding the visible corpus's expected outputs no longer pays.--language-version) to an explicit --style short|tall flag, with the full contract documented in the workspace README instead of the task prompt./app/.timer file interface was replaced by the standard sandbox-timer CLI.test/short, test/tall, benchmark/case), fetched at image build (not vendored) at the exact revision Dart SDK 3.11.3 vendors as its formatter (dart_style 3.1.4-wip), so the baked dart format reproduces the corpus by construction.dart format implementation (SDK 3.11.3, pinned; fetched at image build, not vendored), baked at /opt/dart-sdk as build and verification infrastructure only; it is locked away from the agent and stripped before scoring.