04Implementation

Dart Style in Haskell

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1
0.9136
±0.031
1,902860.2M1.3M$461.3818.5h
2GPT-5.6
0.6337
±0.123
810325.0M403k$328.246.2h
3Gemini 3.7 Flash
0.4727
±0.128
2,036894.0M1.3M$91.0512.8h
4GLM-5.3
0.4606
±0.054
1,131526.3M727k$143.7819.6h
5Kimi K3
0.4045
±0.232
1,062454.5M1.0M$161.3419.9h
6Qwen3.8-Max
0.2859
±0.142
975347.6M1.3M$95.9818.4h
7DeepSeek V4 Flash Exp
0.2199
±0.119
2,120954.7M1.4M$15.4815.9h
8Grok 4.6
0.2133
±0.245
1,889467.9M2.3M$467.4619.4h
9Muse Spark 1.2
0.1948
±0.242
12715.4M112k$3.191.2h
10Inkling
0.0000
±0.000
17511.6M27k$11.150.8h

Background

A production code formatter is a full language frontend plus a constraint solver: it must scan and parse the entire language (every declaration form, expression, comment placement, and string interpolation) and then choose line splits that satisfy a page width while matching thousands of precedents byte for byte. Dart's formatter is harder still because it has two complete layout engines: the classic "short" style and the current "tall" style, with different splitting rules and trailing-comma handling.

The agent is handed tests, not source: the workspace contains no formatter implementation to port from, only the formatter's input/expected-output test corpus. The behaviour has to be reconstructed from what those cases pin down, in Haskell.

Task

Implement the Dart code formatter as a Haskell executable named dart-style that reads Dart source on stdin and writes the formatted result to stdout, working in the provided cabal project at /app/dart-style/.

  • Changes are confined to .hs files under src/; the build configuration (dart-style.cabal, cabal.project) is fixed and language extensions go in per-file pragmas.
  • Both formatting styles must work, selected by --style short|tall, along with arbitrary --page-width and --indent values, --compilation-unit versus --statement parsing, and --trailing-commas automate|preserve.
  • Correctness is byte-exact: each corpus case feeds input on stdin and compares stdout against the expected output byte for byte.
  • cabal build all must succeed and cabal list-bin dart-style must locate the executable.

What The Agent Gets

A compiling cabal scaffold at /app/dart-style/, an /app/README.mdspelling out the build commands and the full CLI contract, and the formatter's test corpus at /app/tests/: short/ and tall/ .stmt/.unit case files plus large real-world benchmark/ inputs. A provided /app/run-tests.shbuilds the project and runs the corpus (all of it, a matching subset, or with per-case mismatch diffs), backed by a documented Python runner the agent can reuse for its own tooling. GHC 9.6.7 and cabal 3.12 are preinstalled with the scaffold's dependencies cached for offline use; the machine is offline and a sandbox-timer CLI reports the remaining budget. The real Dart SDK is baked into the image for verification only and is not reachable by the agent.

Verification

A clean-room verifier, separate from the agent's environment, reconstructs the graded project from a pristine baked scaffold plus only the agent's .hs sources, strips the Dart runtime, clean-rebuilds with cabal as a non-root user, and runs a root-only scored corpus of roughly 5,000 cases. The scored corpus is a perturbed rendering: the same pinned dart formatre-renders every case at a different page width and indent at image build, so memorising the visible corpus's expected outputs does not transfer.

  • The reward is byte-exact passes divided by scored cases, where scored cases are the changed-input cases the pinned reference formatter passed at image build. Unchanged-input cases (about 11% of the corpus) are excluded from both numerator and denominator, so copying stdin to stdout scores 0.
  • Hard gates: a failed build or missing executable, a reachable Dart runtime, or submitting the unmodified starting scaffold zeroes the result.
  • Each case runs under a fixed 30-second timeout and the whole suite under a fixed 4600-second deadline; tripping the deadline records an invalid run rather than a partial score.

Environment

Base imageubuntu:24.04
Tools availableHaskell (GHC), C/C++ toolchain, Python 3
Compute4 CPUs · 8 GB RAM
Time limit20h

Changes From V1

This task is an improved revision of Dart → Haskell from FrontierSWE V1.

  • The V1 workspace shipped dart_style's full Dart source at /app/reference/ (plus analyzer dependency sources) to port from; V2 removes the source entirely. The agent gets the test corpus as the executable specification, plus a fixed cabal scaffold and a documented corpus runner.
  • Verification moved from a bash script hardening the agent's own container in place (PATH locking, system-binary checksums, strace probes) to a clean-room verifier in a separate, hash-pinned image that reconstructs the project from a pristine scaffold plus only the agent's .hs sources; wrapper binaries, build-hook tampering, and cabal-file edits are discarded by construction.
  • The scored corpus is now perturbed at image build: the pinned dart formatre-renders every case at a different page width and indent, so hardcoding the visible corpus's expected outputs no longer pays.
  • Scoring gates tightened: unchanged-input cases (about 11%) are excluded from both sides of the fraction so a copy-stdin-to-stdout submission scores exactly 0, the denominator is fixed to the cases the reference passed at bake, and submitting the unmodified scaffold is an explicit anti-cheat failure.
  • The CLI contract was reworked from version-based pipeline selection (--language-version) to an explicit --style short|tall flag, with the full contract documented in the workspace README instead of the task prompt.
  • The verifier timeout dropped from 24 hours to 2 hours, with a fixed 30-second per-case timeout and a fixed 4600-second suite deadline that records an invalid run when tripped; the ad-hoc /app/.timer file interface was replaced by the standard sandbox-timer CLI.

References

  • dart_style (BSD-3-Clause): The test corpus (test/short, test/tall, benchmark/case), fetched at image build (not vendored) at the exact revision Dart SDK 3.11.3 vendors as its formatter (dart_style 3.1.4-wip), so the baked dart format reproduces the corpus by construction.
  • Dart SDK (BSD-3-Clause): The dart format implementation (SDK 3.11.3, pinned; fetched at image build, not vendored), baked at /opt/dart-sdk as build and verification infrastructure only; it is locked away from the agent and stripped before scoring.
  • Flutter (BSD-3-Clause): A few benchmark and regression corpus cases derive from Flutter framework test code via dart_style's own benchmark suite, used solely as formatter inputs and expected formatted outputs.