| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.9246 ±0.007 | 279 | 96.7M | 337k | $56.47 | 7.4h |
| 2 | GPT-5.6 | 0.9017 ±0.021 | 281 | 73.9M | 127k | $65.97 | 4.3h |
| 3 | Kimi K3 | 0.7866 ±0.281 | 343 | 99.3M | 417k | $39.10 | 9.2h |
| 4 | Grok 4.6 | 0.6431 ±0.359 | 452 | 108.5M | 1.1M | $119.46 | 12.8h |
| 5 | GLM-5.3 | 0.5702 ±0.467 | 450 | 155.9M | 420k | $54.84 | 17.5h |
| 6 | DeepSeek V4 Flash Exp | 0.3511 ±0.089 | 552 | 213.1M | 423k | $3.66 | 17.8h |
| 7 | Qwen3.8-Max | 0.2491 ±0.207 | 443 | 109.6M | 1.4M | $40.44 | 19.7h |
| 8 | Gemini 3.7 Flash | 0.0331 ±0.002 | 520 | 185.3M | 609k | $31.93 | 8.5h |
| 9 | Inkling | 0.0228 ±0.021 | 161 | 14.9M | 29k | $14.28 | 1.3h |
| 10 | Muse Spark 1.2 | 0.0115 ±0.016 | 297 | 94.1M | 425k | $21.15 | 3.3h |
Wan 2.1 T2V-1.3B is a text-to-video diffusion model (a diffusion transformer paired with a 3D VAE and a text encoder) whose inference pipeline spans text encoding, iterative denoising, and video decoding on the GPU. Porting it off PyTorch means re-expressing every model component on a different ML stack: Modular's MAX graph and inference APIs, with custom Mojo kernels where the built-in operations fall short.
The port is measured against the original: the agent's pipeline must reproduce the PyTorch reference's output frames to high pixel fidelity, working entirely offline from the reference source, the pretrained weights, and the bundled MAX API documentation. PyTorch itself is not installed; the target stack is the only stack available.
Implement Wan 2.1 T2V-1.3B text-to-video generation on Modular's MAX/Mojo stack. The deliverable is the Python package at /app/wan21_max/, whose entrypoint module wan_pipeline.py must export generate_video(prompt, height, width, num_frames, num_steps, seed) returning a list of PIL Images, one per frame.
.py modules (plus .mojo kernels for custom ops) under /app/wan21_max/./app/reference/ and the .py files under /app/weights/ are removed before scoring.torch, transformers, and diffusers are hard library constraints: they are not installed and must not be used.The package scaffold at /app/wan21_max/ and an /app/README.md spelling out the exact API, the workload family, what counts as correct output, and the library constraints. For porting: the Wan 2.1 PyTorch reference source at /app/reference/ (read-only), the pretrained T2V-1.3B weights in diffusers format at /app/weights/, and the Modular MAX API documentation at /app/max_docs/. For self-checking: sample workloads with baked reference frames under /app/examples/ and /app/verify_correctness.py, which generates each sample workload, compares output against the reference frames, and enforces the per-generation time budget. The machine has one H100 GPU, is offline, ships the MAX SDK and Mojo toolchain with warmed compile caches, and exposes a sandbox-timer CLI for the remaining budget.
A clean-room verifier runs in a fresh container on the captured workspace. It reconstructs the scored package from a pristine scaffold plus only the agent's wan21_max/ files into a root-owned, read-only directory, strips the reference source and stray Python files as disclosed, then runs a gate sequence: a source-level anti-cheat verdict (banned imports, verifier-internals references, shell or dynamic-import reach, and a required MAX SDK import), an import check, and the disclosed smoke gate. Only then does it generate frames for nine hidden workloads as a non-root user. Root then scores the saved frames against root-only reference frames; no candidate code runs during scoring.
| Base image | nvidia/cuda:12.8.1-devel-ubuntu22.04 |
| Tools available | CUDA, MAX SDK, Mojo, CMake, C/C++ toolchain, Python 3 |
| Compute | 8 CPUs · 128 GB RAM |
| GPU | 1× H100 |
| Time limit | 20h |
This task is an improved revision of Wan 2.1 on MAX/Mojo from FrontierSWE V1.
.py files for banned imports; V2 uninstalls torch, transformers, and diffusers from the image entirely (verified in preflight), so the constraint cannot be routed around at runtime.wan21_max/ files into a root-owned read-only directory, candidate code runs de-rooted, reference frames stay root-only, and the scorer rejects symlinked frames./app/submission/candidate_pipeline.py into the /app/wan21_max/ package with the full contract moved into /app/README.md, and the /app/.timer file interface was replaced by the standard sandbox-timer CLI./app/reference/ for the agent to read while porting the pipeline to MAX./app/weights. The same weights, run through the diffusers WanPipeline, generated the baked reference frames used for scoring and the visible examples.WanPipeline stack the port is measured against; it generated the baked reference frames out-of-band at pinned versions. The stack is not installed in the task image: the deliverable must not use it, and the ban is structural (absent, with no egress to fetch it)./app/max_docs/.