| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.7779 ±0.428 | 249 | 95.4M | 448k | $58.57 | 3.7h |
| 2 | Grok 4.6 | 0.0209 ±0.024 | 1,452 | 393.7M | 2.0M | $402.94 | 14.4h |
| 3 | GPT-5.6 | 0.0204 ±0.003 | 311 | 92.1M | 204k | $94.52 | 2.4h |
| 4 | Kimi K3 | 0.0130 ±0.017 | 1,170 | 513.6M | 1.6M | $188.95 | 19.9h |
| 5 | GLM-5.3 | 0.0109 ±0.006 | 1,338 | 596.2M | 1.2M | $163.96 | 19.0h |
| 6 | Gemini 3.7 Flash | 0.0082 ±0.008 | 1,878 | 756.6M | 1.9M | $87.06 | 20.0h |
| 7 | DeepSeek V4 Flash Exp | 0.0062 ±0.003 | 2,894 | 1331.7M | 2.3M | $22.22 | 19.3h |
| 8 | Muse Spark 1.2 | 0.0033 ±0.002 | 415 | 184.7M | 495k | $31.67 | 2.7h |
| 9 | Qwen3.8-Max | 0.0003 ±0.001 | 142 | 16.9M | 1.4M | $12.66 | 19.3h |
| 10 | Inkling | 0.0000 ±0.000 | 167 | 13.6M | 26k | $13.03 | 0.7h |
Filesystems for NOR flash live under constraints that desktop filesystems never face: writes only flip bits one way until a whole block is erased, blocks wear out after a bounded number of erase cycles, and embedded devices can lose power at any instant. A correct design must guarantee that power loss between any two operations never corrupts the filesystem, while spreading erases across the device and keeping RAM usage tiny.
The task workspace is derived from a production embedded filesystem, de-branded so the agent cannot simply recall the implementation: the test and bench suites define the behavior, and the agent has to reconstruct the design from what they pin down.
Implement the filesystem in Zig inside /app/flash-fs/, exporting C-compatible symbols (export fn, callconv(.c)) that match what the test suites call. Using @cImport to wrap existing C code is not allowed; the filesystem itself must be written in Zig.
tests/ suites are white-box: they read raw on-disk metadata directly, so correctness means reproducing the exact on-disk block-device state they pin down, not merely passing at the API surface.benches/ suites measure bytes read, bytes programmed, and blocks erased; lower is better.A ready-to-run Zig project at /app/flash-fs/: the correctness suites in tests/, the flash-I/O efficiency suites in benches/, an emulated block device, the test runner, and a build system driven by ./build_and_test.sh (all suites) or ./build_and_test.sh <suite> (one). The machine is offline; everything needed is preinstalled, including the Zig toolchain.
A clean-room verifier, separate from the agent's environment, rebuilds the submission and runs the hidden versions of the suites. The scored answer key is the reference implementation's per-permutation block-device checksums across four flash geometries, so credit requires a genuinely working filesystem that produces the exact on-disk state; a recalled description of the original design is not enough.
| Base image | ubuntu:24.04 |
| Tools available | Zig, C/C++ toolchain, Python 3 |
| Compute | 4 CPUs · 4 GB RAM |
| Time limit | 20h |