19Scientific Computing

MS/MS De Novo Generation

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1*
0.0808
±0.074
24366.0M370k$46.4717.8h
2Kimi K3
0.0658
±0.060
29053.0M248k$23.4219.0h
3GLM-5.3
0.0605
±0.055
390102.2M353k$37.4818.6h
4Gemini 3.7 Flash
0.0555
±0.051
24040.1M191k$9.386.5h
5GPT-5.6
0.0526
±0.072
38978.9M190k$64.3011.5h
6Grok 4.6
0.0346
±0.049
558132.0M762k$141.3715.5h
7Qwen3.8-Max
0.0306
±0.043
41971.5M533k$21.5718.9h
8Muse Spark 1.2
0.0213
±0.029
28481.4M279k$15.143.7h
9DeepSeek V4 Flash Exp
0.0000
±0.000
481138.4M393k$2.5216.6h
10Inkling
0.0000
±0.000
1636.9M56k$6.801.3h

* Content-filtered by the provider. Scores shown are from Claude Opus 5 runs used as fallback.

Background

Tandem mass spectrometry (MS/MS) fragments an ionized molecule and records the mass-to-charge ratios and intensities of the pieces. Reading the molecular structure back out of that fragment pattern (de novo structure elucidation) is a long-standing open problem in computational chemistry: the mapping from structure to spectrum is lossy and instrument-dependent, many distinct molecules share a molecular formula, and fragmentation chemistry is only partially understood. Models that work on molecules seen during training routinely fail on new chemical scaffolds.

The task data derives from the MassSpecGym benchmark, prepared with a scaffold-disjoint split so the hidden evaluation spectra come from molecular structures the agent's model has never seen. The verifier additionally runs a permutation test to confirm the model actually uses the spectrum rather than just the molecular formula.

Task

Build a deployable model that predicts best-first candidate SMILES from each tandem mass spectrum and its supplied metadata, and leave the finished deployment under /app/msms_model/. The entrypoint must run as python /app/msms_model/predict.py --data-dir … --checkpoint … --output-path … and write one JSONL row per input spectrum.

  • Each row carries 1–10 candidate SMILES ordered best-first. Invalid or formula-incompatible candidates score zero at their rank and never promote later candidates.
  • Candidates should be chemically valid and compatible with the supplied molecular formula (heavy-atom counts must match; hydrogen is ignored to avoid adduct artifacts).
  • Output must be deterministic for the same input and checkpoint, and predictions should use the measured mzs and intensities together with the metadata, not the formula alone.
  • The deployment must be self-contained and replayable offline: the checkpoint may occupy at most 2.5 GB and a prediction batch must finish within 30 minutes.

What The Agent Gets

Labeled training and validation data staged read-only under /data/train/ and /data/validation/ as parquet pairs (spectra.parquet, labels.parquet). Each spectrum row carries an opaque spectrum_id, precursor_mz, adduct, instrument, collision_energy, the molecular formula, and parallel fragment arrays; labels map IDs to canonical SMILES.

  • A starter workspace at /app with neutral data readers and contract validators in /app/msms_io and a runnable skeleton deployment in /app/msms_model.
  • A CUDA 12.4 Python stack preinstalled from a hash-pinned lock: PyTorch 2.6, RDKit, matchms, scikit-learn, SciPy, pandas, and pyarrow.
  • The machine is offline (Hugging Face and Transformers caches are forced offline); everything needed is already present.

Verification

A root-only verifier reruns the submitted predict.py against hidden spectra that were never visible to the agent, then scores the candidates against hidden labels with RDKit. The reward is a weighted blend of structural metrics macro-averaged over connectivity classes (first InChIKey block): top-1 Tanimoto similarity (0.35), rank-discounted Tanimoto (0.25), top-1 exact-structure accuracy (0.20), top-10 accuracy (0.10), and top-10 mean reciprocal rank (0.10). Only formula-compatible candidates earn similarity credit.

  • Five prediction runs are captured: the primary hidden set, two determinism runs on a fixed subset that must agree exactly, and noise/swap probes with corrupted or shuffled fragment peaks.
  • A spectral-advantage permutation test (999 permutations) conditionally permutes predictions between structures sharing the same formula; if the model shows no statistically significant association with the actual spectra (p ≤ 0.10 with a minimum effect size), the reward is zero.
  • Zeroed outright: a missing or oversized (>2.5 GB) checkpoint, candidate-run failures or timeouts, fewer than 50% chemically valid candidates, nondeterministic output, missing or duplicate predictions, and submission code that references scoring internals.

Environment

Base imagenvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
Tools availableCUDA, PyTorch, Python 3
Compute8 CPUs · 32 GB RAM
GPU1× L4
Time limit20h

References

  • MassSpecGym (MIT): The spectra and molecular labels (dataset release on Polaris Hub; paper: NeurIPS 2024, arXiv:2410.23326). MassSpecGym aggregates spectra from the MoNA, MassBank, and GNPS community libraries, whose attribution metadata is retained in the prepared data manifest. The scaffold-disjoint split preparation, task scaffolding, and verification code are original to this task.