| # | Model | Score | Steps | In tok | Out tok | Avg cost | Avg time |
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | 0.9023 ±0.053 | 173 | 20.2M | 132k | $13.33 | 17.3h |
| 2 | GLM-5.3 | 0.8944 ±0.090 | 320 | 51.1M | 193k | $22.78 | 19.1h |
| 3 | Kimi K3 | 0.8557 ±0.082 | 157 | 17.1M | 149k | $11.61 | 19.1h |
| 4 | Muse Spark 1.2 | 0.8110 ±0.070 | 207 | 44.9M | 215k | $9.21 | 2.0h |
| 5 | Grok 4.6 | 0.7812 ±0.136 | 228 | 39.9M | 257k | $54.01 | 19.5h |
| 6 | GPT-5.6 | 0.5873 ±0.533 | 313 | 74.3M | 128k | $65.94 | 12.6h |
| 7 | Qwen3.8-Max | 0.5825 ±0.447 | 206 | 15.3M | 222k | $5.36 | 18.7h |
| 8 | Gemini 3.7 Flash | 0.5007 ±0.326 | 189 | 26.0M | 156k | $8.38 | 6.2h |
| 9 | DeepSeek V4 Flash Exp | 0.4758 ±0.343 | 423 | 106.5M | 316k | $1.96 | 18.7h |
| 10 | Inkling | 0.0340 ±0.018 | 144 | 5.2M | 47k | $5.11 | 1.6h |
Density functional theory (DFT) gives accurate energies and forces for atomic structures but costs hours per calculation. Machine-learned interatomic potentials replace it with a model that is orders of magnitude faster, which is what makes large-scale materials screening and molecular dynamics tractable. Building one is genuinely hard: the model must respect periodic boundary conditions, handle variable numbers of atoms across dozens of chemical elements, keep forces consistent with the energy surface, and (the real test) extrapolate to compositions and structure types it never trained on.
The structures and DFT labels derive from Meta FAIR Chemistry's Open Materials 2024 (OMat24) dataset of inorganic materials, with split preparation original to the benchmark so the scored structures cannot be recalled from public data.
Build one deployable model that predicts the total energy (eV) and per-atom forces (eV/Å) of periodic atomic structures at DFT accuracy, and keep everything required for replay under /app/materials_model/ (checkpoint, model.py, predict.py, run_summary.json).
python /app/materials_model/predict.py --data-dir … --checkpoint … --output-path … and writes one JSONL object per structure with structure_id, total energy, and forces of exactly n_atoms rows of three finite numbers in input atom order.cell and pbc, and predictions must respond to changes in atomic geometry: no external services or hardcoded per-structure lookups.Labeled training data at /data/train/structures.parquet (each row carries structure_id, n_atoms, atomic_numbers, positions, cell, and pbc, plus DFT energy and per-atom forces), along with unlabeled validation inputs at /data/validation/structures.parquet and /data/metadata.json, all staged read-only from digest-pinned dataset images.
/app/materials_model/ exercising the full inference contract.A root-only verifier reruns the submitted predict.py on hidden structures the agent never sees, grouped into workloads: the scored ones are out-of-distribution compositions and out-of-distribution bulk structures, weighted equally. For each workload the energy MAE (meV/atom) and force MAE (meV/Å) are converted to skill scores (0 at a non-trivial baseline anchor, 0.9 at the measured reference-solution anchor, 1 at zero error), and the reward is the weighted geometric mean of all skills.
| Base image | nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 |
| Tools available | CUDA, PyTorch, Python 3 |
| Compute | 8 CPUs · 32 GB RAM |
| GPU | 1× GPU (L4, A10G, L40S, or H100) |
| Time limit | 20h |
fairchem/OMAT24 dataset (Barroso-Luque et al., 2024). Split preparation, task scaffolding, reference models, and verification code are original to this benchmark.