| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.9136 ±0.031 | 1,902 | 860.2M | 1.3M | $461.38 | 18.5h |
| 2 | GPT-5.6 | 0.6337 ±0.123 | 810 | 325.0M | 403k | $328.24 | 6.2h |
| 3 | Gemini 3.7 Flash | 0.4727 ±0.128 | 2,036 | 894.0M | 1.3M | $91.05 | 12.8h |
| 4 | GLM-5.3 | 0.4606 ±0.054 | 1,131 | 526.3M | 727k | $143.78 | 19.6h |
| 5 | Kimi K3 | 0.4045 ±0.232 | 1,062 | 454.5M | 1.0M | $161.34 | 19.9h |
| 6 | Qwen3.8-Max | 0.2859 ±0.142 | 975 | 347.6M | 1.3M | $95.98 | 18.4h |
| 7 | DeepSeek V4 Flash Exp | 0.2199 ±0.119 | 2,120 | 954.7M | 1.4M | $15.48 | 15.9h |
| 8 | Grok 4.6 | 0.2133 ±0.245 | 1,889 | 467.9M | 2.3M | $467.46 | 19.4h |
| 9 | Muse Spark 1.2 | 0.1948 ±0.242 | 127 | 15.4M | 112k | $3.19 | 1.2h |
| 10 | Inkling | 0.0000 ±0.000 | 175 | 11.6M | 27k | $11.15 | 0.8h |
A production code formatter is a full language frontend plus a constraint solver: it must scan and parse the entire language (every declaration form, expression, comment placement, and string interpolation) and then choose line splits that satisfy a page width while matching thousands of precedents byte for byte. Dart's formatter is harder still because it has two complete layout engines: the classic "short" style and the current "tall" style, with different splitting rules and trailing-comma handling.
The agent is handed tests, not source: the workspace contains no formatter implementation to port from, only the formatter's input/expected-output test corpus. The behaviour has to be reconstructed from what those cases pin down, in Haskell.
Implement the Dart code formatter as a Haskell executable named dart-style that reads Dart source on stdin and writes the formatted result to stdout, working in the provided cabal project at /app/dart-style/.
.hs files under src/; the build configuration (dart-style.cabal, cabal.project) is fixed and language extensions go in per-file pragmas.--style short|tall, along with arbitrary --page-width and --indent values, --compilation-unit versus --statement parsing, and --trailing-commas automate|preserve.cabal build all must succeed and cabal list-bin dart-style must locate the executable.A compiling cabal scaffold at /app/dart-style/, an /app/README.mdspelling out the build commands and the full CLI contract, and the formatter's test corpus at /app/tests/: short/ and tall/ .stmt/.unit case files plus large real-world benchmark/ inputs. A provided /app/run-tests.shbuilds the project and runs the corpus (all of it, a matching subset, or with per-case mismatch diffs), backed by a documented Python runner the agent can reuse for its own tooling. GHC 9.6.7 and cabal 3.12 are preinstalled with the scaffold's dependencies cached for offline use; the machine is offline and a sandbox-timer CLI reports the remaining budget. The real Dart SDK is baked into the image for verification only and is not reachable by the agent.
A clean-room verifier, separate from the agent's environment, reconstructs the graded project from a pristine baked scaffold plus only the agent's .hs sources, strips the Dart runtime, clean-rebuilds with cabal as a non-root user, and runs a root-only scored corpus of roughly 5,000 cases. The scored corpus is a perturbed rendering: the same pinned dart formatre-renders every case at a different page width and indent at image build, so memorising the visible corpus's expected outputs does not transfer.
| Base image | ubuntu:24.04 |
| Tools available | Haskell (GHC), C/C++ toolchain, Python 3 |
| Compute | 4 CPUs · 8 GB RAM |
| Time limit | 20h |
This task is an improved revision of Dart → Haskell from FrontierSWE V1.
/app/reference/ (plus analyzer dependency sources) to port from; V2 removes the source entirely. The agent gets the test corpus as the executable specification, plus a fixed cabal scaffold and a documented corpus runner..hs sources; wrapper binaries, build-hook tampering, and cabal-file edits are discarded by construction.dart formatre-renders every case at a different page width and indent, so hardcoding the visible corpus's expected outputs no longer pays.--language-version) to an explicit --style short|tall flag, with the full contract documented in the workspace README instead of the task prompt./app/.timer file interface was replaced by the standard sandbox-timer CLI.test/short, test/tall, benchmark/case), fetched at image build (not vendored) at the exact revision Dart SDK 3.11.3 vendors as its formatter (dart_style 3.1.4-wip), so the baked dart format reproduces the corpus by construction.dart format implementation (SDK 3.11.3, pinned; fetched at image build, not vendored), baked at /opt/dart-sdk as build and verification infrastructure only; it is locked away from the agent and stripped before scoring.