04Implementation

Dart Style in Haskell

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Opus 5.5
0.9839
±0.006
1,781777.1M1.9M$205.8212.0h
2Claude Sonnet 5.5
0.9606
±0.009
2,5391161.0M3.0M$270.8119.4h
3Gemini 4 Argon
0.9548
±0.007
1,212514.2M2.1M$253.45*16.6h
4GPT-6 Astra
0.9394
±0.025
1,828791.2M1.5M$1661.3412.1h
5Claude Fable 5.1
0.9136
±0.031
1,902860.2M1.3M$461.3818.5h
6Claude Opus 5
0.8473
±0.034
1,535690.4M1.1M$389.6719.5h
7Claude Fable 5
0.8254
±0.039
1,394622.4M1.3M$723.1119.6h
8Claude Haiku 5.5
0.7915
±0.033
2,4371127.4M2.6M$63.2819.2h
9Grok 4.7
0.6588
±0.130
3,007756.1M2.3M$741.5817.1h
10GPT-5.6 Sol
0.6337
±0.123
810325.0M403k$328.246.2h
11Gemini 3.7 Flash
0.4727
±0.128
2,036894.0M1.3M$91.0512.8h
12GLM-5.3
0.4606
±0.054
1,131526.3M727k$143.7819.6h
13Gemini 3.8 Flash
0.4099
±0.251
1,508629.5M829k$62.469.8h
14Kimi K3
0.4045
±0.232
1,062454.5M1.0M$161.3419.9h
15Muse Spark 1.3
0.3896
±0.228
1,597634.8M1.1M$520.1519.8h
16GLM-5.3 Flash
0.3051
±0.120
1,122538.1M736k$17.7819.9h
17Qwen3.8-Max
0.2859
±0.142
975347.6M1.3M$95.9818.4h
18DeepSeek V4 Flash Vision Exp
0.2199
±0.119
2,120954.7M1.4M$15.4815.9h
19Grok 4.6
0.2133
±0.245
1,889467.9M2.3M$467.4619.4h
20Muse Spark 1.2
0.1948
±0.242
12715.4M112k$3.191.2h
21Qwen3.8-Max-0902
0.1051
±0.125
573181.6M2.7M$63.4120.0h
22Inkling
0.0000
±0.000
17511.6M27k$11.150.8h

*Gemini 4 Argon: Cache hit rate is much lower due to a non-production setting.

Background

A production code formatter is a full language frontend plus a constraint solver: it must scan and parse the entire language (every declaration form, expression, comment placement, and string interpolation) and then choose line splits that satisfy a page width while matching thousands of precedents byte for byte. Dart's formatter is harder still because it has two complete layout engines: the classic "short" style and the current "tall" style, with different splitting rules and trailing-comma handling.

The agent is handed tests, not source: the workspace contains no formatter implementation to port from, only the formatter's input/expected-output test corpus. The behaviour has to be reconstructed from what those cases pin down, in Haskell.

Task

Implement the Dart code formatter as a Haskell executable named dart-style that reads Dart source on stdin and writes the formatted result to stdout, working in the provided cabal project at /app/dart-style/.

  • Changes are confined to .hs files under src/; the build configuration (dart-style.cabal, cabal.project) is fixed and language extensions go in per-file pragmas.
  • Both formatting styles must work, selected by --style short|tall, along with arbitrary --page-width and --indent values, --compilation-unit versus --statement parsing, and --trailing-commas automate|preserve.
  • Correctness is byte-exact: each corpus case feeds input on stdin and compares stdout against the expected output byte for byte.
  • cabal build all must succeed and cabal list-bin dart-style must locate the executable.

What The Agent Gets

A compiling cabal scaffold at /app/dart-style/, an /app/README.mdspelling out the build commands and the full CLI contract, and the formatter's test corpus at /app/tests/: short/ and tall/ .stmt/.unit case files plus large real-world benchmark/ inputs. A provided /app/run-tests.shbuilds the project and runs the corpus (all of it, a matching subset, or with per-case mismatch diffs), backed by a documented Python runner the agent can reuse for its own tooling. GHC 9.6.7 and cabal 3.12 are preinstalled with the scaffold's dependencies cached for offline use; the machine is offline and a sandbox-timer CLI reports the remaining budget. The real Dart SDK is baked into the image for verification only and is not reachable by the agent.

Verification

A clean-room verifier, separate from the agent's environment, reconstructs the graded project from a pristine baked scaffold plus only the agent's .hs sources, strips the Dart runtime, clean-rebuilds with cabal as a non-root user, and runs a root-only scored corpus of roughly 5,000 cases. The scored corpus is a perturbed rendering: the same pinned dart formatre-renders every case at a different page width and indent at image build, so memorising the visible corpus's expected outputs does not transfer.

  • The reward is byte-exact passes divided by scored cases, where scored cases are the changed-input cases the pinned reference formatter passed at image build. Unchanged-input cases (about 11% of the corpus) are excluded from both numerator and denominator, so copying stdin to stdout scores 0.
  • Hard gates: a failed build or missing executable, a reachable Dart runtime, or submitting the unmodified starting scaffold zeroes the result.
  • Each case runs under a fixed 30-second timeout and the whole suite under a fixed 4600-second deadline; tripping the deadline records an invalid run rather than a partial score.

Environment

Base imageubuntu:24.04
Tools availableHaskell (GHC), C/C++ toolchain, Python 3
Compute4 CPUs · 8 GB RAM
Time limit20h

Changes From V1

This task is an improved revision of Dart → Haskell from FrontierSWE V1.

  • The V1 workspace shipped dart_style's full Dart source at /app/reference/ (plus analyzer dependency sources) to port from; V2 removes the source entirely. The agent gets the test corpus as the executable specification, plus a fixed cabal scaffold and a documented corpus runner.
  • Verification moved from a bash script hardening the agent's own container in place (PATH locking, system-binary checksums, strace probes) to a clean-room verifier in a separate, hash-pinned image that reconstructs the project from a pristine scaffold plus only the agent's .hs sources; wrapper binaries, build-hook tampering, and cabal-file edits are discarded by construction.
  • The scored corpus is now perturbed at image build: the pinned dart formatre-renders every case at a different page width and indent, so hardcoding the visible corpus's expected outputs no longer pays.
  • Scoring gates tightened: unchanged-input cases (about 11%) are excluded from both sides of the fraction so a copy-stdin-to-stdout submission scores exactly 0, the denominator is fixed to the cases the reference passed at bake, and submitting the unmodified scaffold is an explicit anti-cheat failure.
  • The CLI contract was reworked from version-based pipeline selection (--language-version) to an explicit --style short|tall flag, with the full contract documented in the workspace README instead of the task prompt.
  • The verifier timeout dropped from 24 hours to 2 hours, with a fixed 30-second per-case timeout and a fixed 4600-second suite deadline that records an invalid run when tripped; the ad-hoc /app/.timer file interface was replaced by the standard sandbox-timer CLI.

References

  • dart_style (BSD-3-Clause): The test corpus (test/short, test/tall, benchmark/case), fetched at image build (not vendored) at the exact revision Dart SDK 3.11.3 vendors as its formatter (dart_style 3.1.4-wip), so the baked dart format reproduces the corpus by construction.
  • Dart SDK (BSD-3-Clause): The dart format implementation (SDK 3.11.3, pinned; fetched at image build, not vendored), baked at /opt/dart-sdk as build and verification infrastructure only; it is locked away from the agent and stripped before scoring.
  • Flutter (BSD-3-Clause): A few benchmark and regression corpus cases derive from Flutter framework test code via dart_style's own benchmark suite, used solely as formatter inputs and expected formatted outputs.