16Scientific Computing

Machine-Learned Interatomic Potential

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Gemini 4 Argon
0.9922
±0.006
20962.7M259k$74.22*20.4h
2GPT-6 Astra
0.9800
±0.010
25553.8M224k$101.2716.8h
3Claude Haiku 5.5
0.9190
±0.128
21962.7M377k$4.0816.4h
4Claude Fable 5.1
0.9023
±0.053
17320.2M132k$13.3317.3h
5Qwen3.8-Max-0902
0.8947
±0.139
486112.8M902k$35.6119.7h
6GLM-5.3
0.8944
±0.090
32051.1M193k$22.7819.1h
7Claude Opus 5.5
0.8622
±0.221
22157.5M333k$20.1618.4h
8Kimi K3
0.8557
±0.082
15717.1M149k$11.6119.1h
9Muse Spark 1.2
0.8110
±0.070
20744.9M215k$9.212.0h
10Muse Spark 1.3
0.7964
±0.310
26930.6M175k$32.4219.5h
11Grok 4.6
0.7812
±0.136
22839.9M257k$54.0119.5h
12Grok 4.7
0.7640
±0.427
23354.5M265k$79.8216.4h
13Claude Fable 5
0.7170
±0.392
15018.2M156k$29.6919.1h
14Claude Opus 5
0.6881
±0.415
18941.1M301k$31.4918.3h
15GPT-5.6 Sol
0.5873
±0.533
31374.3M128k$65.9412.6h
16Qwen3.8-Max
0.5825
±0.447
20615.3M222k$5.3618.7h
17Gemini 3.7 Flash
0.5007
±0.326
18926.0M156k$8.386.2h
18DeepSeek V4 Flash Vision Exp
0.4758
±0.343
423106.5M316k$1.9618.7h
19Claude Sonnet 5.5
0.4198
±0.529
351150.4M568k$37.3518.4h
20Gemini 3.8 Flash
0.3789
±0.455
27459.2M167k$12.254.4h
21GLM-5.3 Flash
0.1927
±0.431
21631.7M204k$1.8718.0h
22Inkling
0.0340
±0.018
1445.2M47k$5.111.6h

*Gemini 4 Argon: Cache hit rate is much lower due to a non-production setting.

Background

Density functional theory (DFT) gives accurate energies and forces for atomic structures but costs hours per calculation. Machine-learned interatomic potentials replace it with a model that is orders of magnitude faster, which is what makes large-scale materials screening and molecular dynamics tractable. Building one is genuinely hard: the model must respect periodic boundary conditions, handle variable numbers of atoms across dozens of chemical elements, keep forces consistent with the energy surface, and (the real test) extrapolate to compositions and structure types it never trained on.

The structures and DFT labels derive from Meta FAIR Chemistry's Open Materials 2024 (OMat24) dataset of inorganic materials, with split preparation original to the benchmark so the scored structures cannot be recalled from public data.

Task

Build one deployable model that predicts the total energy (eV) and per-atom forces (eV/Å) of periodic atomic structures at DFT accuracy, and keep everything required for replay under /app/materials_model/ (checkpoint, model.py, predict.py, run_summary.json).

  • Inference runs as python /app/materials_model/predict.py --data-dir … --checkpoint … --output-path … and writes one JSONL object per structure with structure_id, total energy, and forces of exactly n_atoms rows of three finite numbers in input atom order.
  • The model must respect cell and pbc, and predictions must respond to changes in atomic geometry: no external services or hardcoded per-structure lookups.
  • One checkpoint of at most 2.5 GB; batch inference within 30 minutes; repeated inference must agree within an absolute tolerance of 1e-4.

What The Agent Gets

Labeled training data at /data/train/structures.parquet (each row carries structure_id, n_atoms, atomic_numbers, positions, cell, and pbc, plus DFT energy and per-atom forces), along with unlabeled validation inputs at /data/validation/structures.parquet and /data/metadata.json, all staged read-only from digest-pinned dataset images.

  • A runnable starter submission under /app/materials_model/ exercising the full inference contract.
  • A CUDA 12.4 stack with PyTorch 2.6 (cu124), ASE, NumPy, pandas, pyarrow, scikit-learn, and SciPy preinstalled.
  • The machine is offline; everything needed is already present.

Verification

A root-only verifier reruns the submitted predict.py on hidden structures the agent never sees, grouped into workloads: the scored ones are out-of-distribution compositions and out-of-distribution bulk structures, weighted equally. For each workload the energy MAE (meV/atom) and force MAE (meV/Å) are converted to skill scores (0 at a non-trivial baseline anchor, 0.9 at the measured reference-solution anchor, 1 at zero error), and the reward is the weighted geometric mean of all skills.

  • A beat-the-baseline gate: if either energy or force MAE fails to improve on the baseline anchor in either primary workload, the reward is zero.
  • Safeguard probes: a determinism re-run must agree within 1e-4, and a geometry-perturbation probe must change predictions for at least half the perturbed structures, rejecting constant or lookup-table submissions. Network attempts during inference are audited via strace, and label columns are scrubbed from the inputs the model receives.
  • Contract failures (malformed rows, wrong force shapes, non-finite values, an oversized checkpoint, or exceeding the 30-minute inference timeout) zero the result.

Environment

Base imagenvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
Tools availableCUDA, PyTorch, Python 3
Compute8 CPUs · 32 GB RAM
GPU1× GPU (L4, A10G, L40S, or H100)
Time limit20h

References

  • Open Materials 2024 (OMat24) (CC BY 4.0): Structures and DFT energy/force labels, from Meta FAIR Chemistry's fairchem/OMAT24 dataset (Barroso-Luque et al., 2024). Split preparation, task scaffolding, reference models, and verification code are original to this benchmark.