19Scientific Computing

MS/MS De Novo Generation

#ModelScoreStepsIn tokOut tokAvg costAvg time
1GPT-6 Astra
0.1627
±0.010
937377.4M809k$789.9012.4h
2Muse Spark 1.3
0.0890
±0.050
33349.0M254k$53.2018.9h
3Claude Sonnet 5.5
0.0823
±0.075
505234.9M826k$57.3916.3h
4Gemini 4 Argon
0.0816
±0.075
18357.2M455k$57.01*9.1h
5Grok 4.7
0.0814
±0.046
23151.1M291k$60.557.7h
6Claude Fable 5*
0.0808
±0.074
24366.0M370k$46.4717.8h
7Claude Fable 5.1*
0.0808
±0.074
24366.0M370k$46.4717.8h
8Claude Opus 5
0.0808
±0.074
24366.0M370k$46.4717.8h
9Claude Haiku 5.5
0.0723
±0.066
296118.8M581k$7.5415.1h
10Kimi K3
0.0658
±0.060
29053.0M248k$23.4219.0h
11GLM-5.3
0.0605
±0.055
390102.2M353k$37.4818.6h
12Claude Opus 5.5
0.0558
±0.076
359115.2M413k$33.8617.2h
13Gemini 3.7 Flash
0.0555
±0.051
24040.1M191k$9.386.5h
14GPT-5.6 Sol
0.0526
±0.072
38978.9M190k$64.3011.5h
15Gemini 3.8 Flash
0.0434
±0.060
24442.0M184k$6.622.9h
16Grok 4.6
0.0346
±0.049
558132.0M762k$141.3715.5h
17Qwen3.8-Max
0.0306
±0.043
41971.5M533k$21.5718.9h
18Muse Spark 1.2
0.0213
±0.029
28481.4M279k$15.143.7h
19Qwen3.8-Max-0902
0.0189
±0.042
597221.7M1.1M$63.4119.3h
20GLM-5.3 Flash
0.0157
±0.035
31968.0M279k$3.2517.8h
21DeepSeek V4 Flash Vision Exp
0.0000
±0.000
481138.4M393k$2.5216.6h
22Inkling
0.0000
±0.000
1636.9M56k$6.801.3h

* Content-filtered by the provider. Scores shown are from Claude Opus 5 runs used as fallback.

*Gemini 4 Argon: Cache hit rate is much lower due to a non-production setting.

Background

Tandem mass spectrometry (MS/MS) fragments an ionized molecule and records the mass-to-charge ratios and intensities of the pieces. Reading the molecular structure back out of that fragment pattern (de novo structure elucidation) is a long-standing open problem in computational chemistry: the mapping from structure to spectrum is lossy and instrument-dependent, many distinct molecules share a molecular formula, and fragmentation chemistry is only partially understood. Models that work on molecules seen during training routinely fail on new chemical scaffolds.

The task data derives from the MassSpecGym benchmark, prepared with a scaffold-disjoint split so the hidden evaluation spectra come from molecular structures the agent's model has never seen. The verifier additionally runs a permutation test to confirm the model actually uses the spectrum rather than just the molecular formula.

Task

Build a deployable model that predicts best-first candidate SMILES from each tandem mass spectrum and its supplied metadata, and leave the finished deployment under /app/msms_model/. The entrypoint must run as python /app/msms_model/predict.py --data-dir … --checkpoint … --output-path … and write one JSONL row per input spectrum.

  • Each row carries 1–10 candidate SMILES ordered best-first. Invalid or formula-incompatible candidates score zero at their rank and never promote later candidates.
  • Candidates should be chemically valid and compatible with the supplied molecular formula (heavy-atom counts must match; hydrogen is ignored to avoid adduct artifacts).
  • Output must be deterministic for the same input and checkpoint, and predictions should use the measured mzs and intensities together with the metadata, not the formula alone.
  • The deployment must be self-contained and replayable offline: the checkpoint may occupy at most 2.5 GB and a prediction batch must finish within 30 minutes.

What The Agent Gets

Labeled training and validation data staged read-only under /data/train/ and /data/validation/ as parquet pairs (spectra.parquet, labels.parquet). Each spectrum row carries an opaque spectrum_id, precursor_mz, adduct, instrument, collision_energy, the molecular formula, and parallel fragment arrays; labels map IDs to canonical SMILES.

  • A starter workspace at /app with neutral data readers and contract validators in /app/msms_io and a runnable skeleton deployment in /app/msms_model.
  • A CUDA 12.4 Python stack preinstalled from a hash-pinned lock: PyTorch 2.6, RDKit, matchms, scikit-learn, SciPy, pandas, and pyarrow.
  • The machine is offline (Hugging Face and Transformers caches are forced offline); everything needed is already present.

Verification

A root-only verifier reruns the submitted predict.py against hidden spectra that were never visible to the agent, then scores the candidates against hidden labels with RDKit. The reward is a weighted blend of structural metrics macro-averaged over connectivity classes (first InChIKey block): top-1 Tanimoto similarity (0.35), rank-discounted Tanimoto (0.25), top-1 exact-structure accuracy (0.20), top-10 accuracy (0.10), and top-10 mean reciprocal rank (0.10). Only formula-compatible candidates earn similarity credit.

  • Five prediction runs are captured: the primary hidden set, two determinism runs on a fixed subset that must agree exactly, and noise/swap probes with corrupted or shuffled fragment peaks.
  • A spectral-advantage permutation test (999 permutations) conditionally permutes predictions between structures sharing the same formula; if the model shows no statistically significant association with the actual spectra (p ≤ 0.10 with a minimum effect size), the reward is zero.
  • Zeroed outright: a missing or oversized (>2.5 GB) checkpoint, candidate-run failures or timeouts, fewer than 50% chemically valid candidates, nondeterministic output, missing or duplicate predictions, and submission code that references scoring internals.

Environment

Base imagenvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
Tools availableCUDA, PyTorch, Python 3
Compute8 CPUs · 32 GB RAM
GPU1× L4
Time limit20h

References

  • MassSpecGym (MIT): The spectra and molecular labels (dataset release on Polaris Hub; paper: NeurIPS 2024, arXiv:2410.23326). MassSpecGym aggregates spectra from the MoNA, MassBank, and GNPS community libraries, whose attribution metadata is retained in the prepared data manifest. The scaffold-disjoint split preparation, task scaffolding, and verification code are original to this task.