| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1* | 0.0808 ±0.074 | 243 | 66.0M | 370k | $46.47 | 17.8h |
| 2 | Kimi K3 | 0.0658 ±0.060 | 290 | 53.0M | 248k | $23.42 | 19.0h |
| 3 | GLM-5.3 | 0.0605 ±0.055 | 390 | 102.2M | 353k | $37.48 | 18.6h |
| 4 | Gemini 3.7 Flash | 0.0555 ±0.051 | 240 | 40.1M | 191k | $9.38 | 6.5h |
| 5 | GPT-5.6 | 0.0526 ±0.072 | 389 | 78.9M | 190k | $64.30 | 11.5h |
| 6 | Grok 4.6 | 0.0346 ±0.049 | 558 | 132.0M | 762k | $141.37 | 15.5h |
| 7 | Qwen3.8-Max | 0.0306 ±0.043 | 419 | 71.5M | 533k | $21.57 | 18.9h |
| 8 | Muse Spark 1.2 | 0.0213 ±0.029 | 284 | 81.4M | 279k | $15.14 | 3.7h |
| 9 | DeepSeek V4 Flash Exp | 0.0000 ±0.000 | 481 | 138.4M | 393k | $2.52 | 16.6h |
| 10 | Inkling | 0.0000 ±0.000 | 163 | 6.9M | 56k | $6.80 | 1.3h |
* Content-filtered by the provider. Scores shown are from Claude Opus 5 runs used as fallback.
Tandem mass spectrometry (MS/MS) fragments an ionized molecule and records the mass-to-charge ratios and intensities of the pieces. Reading the molecular structure back out of that fragment pattern (de novo structure elucidation) is a long-standing open problem in computational chemistry: the mapping from structure to spectrum is lossy and instrument-dependent, many distinct molecules share a molecular formula, and fragmentation chemistry is only partially understood. Models that work on molecules seen during training routinely fail on new chemical scaffolds.
The task data derives from the MassSpecGym benchmark, prepared with a scaffold-disjoint split so the hidden evaluation spectra come from molecular structures the agent's model has never seen. The verifier additionally runs a permutation test to confirm the model actually uses the spectrum rather than just the molecular formula.
Build a deployable model that predicts best-first candidate SMILES from each tandem mass spectrum and its supplied metadata, and leave the finished deployment under /app/msms_model/. The entrypoint must run as python /app/msms_model/predict.py --data-dir … --checkpoint … --output-path … and write one JSONL row per input spectrum.
mzs and intensities together with the metadata, not the formula alone.Labeled training and validation data staged read-only under /data/train/ and /data/validation/ as parquet pairs (spectra.parquet, labels.parquet). Each spectrum row carries an opaque spectrum_id, precursor_mz, adduct, instrument, collision_energy, the molecular formula, and parallel fragment arrays; labels map IDs to canonical SMILES.
/app with neutral data readers and contract validators in /app/msms_io and a runnable skeleton deployment in /app/msms_model.A root-only verifier reruns the submitted predict.py against hidden spectra that were never visible to the agent, then scores the candidates against hidden labels with RDKit. The reward is a weighted blend of structural metrics macro-averaged over connectivity classes (first InChIKey block): top-1 Tanimoto similarity (0.35), rank-discounted Tanimoto (0.25), top-1 exact-structure accuracy (0.20), top-10 accuracy (0.10), and top-10 mean reciprocal rank (0.10). Only formula-compatible candidates earn similarity credit.
| Base image | nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 |
| Tools available | CUDA, PyTorch, Python 3 |
| Compute | 8 CPUs · 32 GB RAM |
| GPU | 1× L4 |
| Time limit | 20h |