16Scientific Computing

Machine-Learned Interatomic Potential

#ModelScoreStepsIn tokOut tokAvg costAvg time
1Claude Fable 5.1
0.9023
±0.053
17320.2M132k$13.3317.3h
2GLM-5.3
0.8944
±0.090
32051.1M193k$22.7819.1h
3Kimi K3
0.8557
±0.082
15717.1M149k$11.6119.1h
4Muse Spark 1.2
0.8110
±0.070
20744.9M215k$9.212.0h
5Grok 4.6
0.7812
±0.136
22839.9M257k$54.0119.5h
6GPT-5.6
0.5873
±0.533
31374.3M128k$65.9412.6h
7Qwen3.8-Max
0.5825
±0.447
20615.3M222k$5.3618.7h
8Gemini 3.7 Flash
0.5007
±0.326
18926.0M156k$8.386.2h
9DeepSeek V4 Flash Exp
0.4758
±0.343
423106.5M316k$1.9618.7h
10Inkling
0.0340
±0.018
1445.2M47k$5.111.6h

Background

Density functional theory (DFT) gives accurate energies and forces for atomic structures but costs hours per calculation. Machine-learned interatomic potentials replace it with a model that is orders of magnitude faster, which is what makes large-scale materials screening and molecular dynamics tractable. Building one is genuinely hard: the model must respect periodic boundary conditions, handle variable numbers of atoms across dozens of chemical elements, keep forces consistent with the energy surface, and (the real test) extrapolate to compositions and structure types it never trained on.

The structures and DFT labels derive from Meta FAIR Chemistry's Open Materials 2024 (OMat24) dataset of inorganic materials, with split preparation original to the benchmark so the scored structures cannot be recalled from public data.

Task

Build one deployable model that predicts the total energy (eV) and per-atom forces (eV/Å) of periodic atomic structures at DFT accuracy, and keep everything required for replay under /app/materials_model/ (checkpoint, model.py, predict.py, run_summary.json).

  • Inference runs as python /app/materials_model/predict.py --data-dir … --checkpoint … --output-path … and writes one JSONL object per structure with structure_id, total energy, and forces of exactly n_atoms rows of three finite numbers in input atom order.
  • The model must respect cell and pbc, and predictions must respond to changes in atomic geometry: no external services or hardcoded per-structure lookups.
  • One checkpoint of at most 2.5 GB; batch inference within 30 minutes; repeated inference must agree within an absolute tolerance of 1e-4.

What The Agent Gets

Labeled training data at /data/train/structures.parquet (each row carries structure_id, n_atoms, atomic_numbers, positions, cell, and pbc, plus DFT energy and per-atom forces), along with unlabeled validation inputs at /data/validation/structures.parquet and /data/metadata.json, all staged read-only from digest-pinned dataset images.

  • A runnable starter submission under /app/materials_model/ exercising the full inference contract.
  • A CUDA 12.4 stack with PyTorch 2.6 (cu124), ASE, NumPy, pandas, pyarrow, scikit-learn, and SciPy preinstalled.
  • The machine is offline; everything needed is already present.

Verification

A root-only verifier reruns the submitted predict.py on hidden structures the agent never sees, grouped into workloads: the scored ones are out-of-distribution compositions and out-of-distribution bulk structures, weighted equally. For each workload the energy MAE (meV/atom) and force MAE (meV/Å) are converted to skill scores (0 at a non-trivial baseline anchor, 0.9 at the measured reference-solution anchor, 1 at zero error), and the reward is the weighted geometric mean of all skills.

  • A beat-the-baseline gate: if either energy or force MAE fails to improve on the baseline anchor in either primary workload, the reward is zero.
  • Safeguard probes: a determinism re-run must agree within 1e-4, and a geometry-perturbation probe must change predictions for at least half the perturbed structures, rejecting constant or lookup-table submissions. Network attempts during inference are audited via strace, and label columns are scrubbed from the inputs the model receives.
  • Contract failures (malformed rows, wrong force shapes, non-finite values, an oversized checkpoint, or exceeding the 30-minute inference timeout) zero the result.

Environment

Base imagenvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
Tools availableCUDA, PyTorch, Python 3
Compute8 CPUs · 32 GB RAM
GPU1× GPU (L4, A10G, L40S, or H100)
Time limit20h

References

  • Open Materials 2024 (OMat24) (CC BY 4.0): Structures and DFT energy/force labels, from Meta FAIR Chemistry's fairchem/OMAT24 dataset (Barroso-Luque et al., 2024). Split preparation, task scaffolding, reference models, and verification code are original to this benchmark.