Tasks Traces V1 Blog Changelog Contribute GitHub

Traces

Every run: the agent's full trajectory, its submissions over the 20-hour budget, and the final score.

Astronomy Toolkit

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9999	$11.62	30.2M	300k	2h 33m	131	3
2	Gemini 4 Argon	0.8999	$8.32	27.3M	281k	1h 26m	117	2
3	Gemini 4 Argon	0.5135	$11.83	39.0M	318k	1h 48m	154	2
4	Gemini 4 Argon	0.0000	$8.85	35.0M	322k	1h 26m	137	2
5	Gemini 4 Argon	0.0000	$27.00	17.6M	236k	2h 6m	93	2

Cranelift Codegen Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.4043	$256.70	491.0M	1.5M	20h 49m	1116	24
2	Gemini 4 Argon	0.2201	$298.10	572.4M	2.1M	20h 43m	1256	24
3	Gemini 4 Argon	0.1959	$361.37	454.4M	1.3M	17h 22m	1002	27
4	Gemini 4 Argon	0.1238	$407.04	524.2M	1.6M	20h 47m	1255	28
5	Gemini 4 Argon	0.0000	$212.69	506.0M	1.7M	20h 46m	1184	27

Crash-Proof Flash Filesystem

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.7378	$152.44	168.7M	1.1M	7h 34m	429	3
2	Gemini 4 Argon	0.5467	$54.25	127.4M	708k	4h 23m	300	2
3	Gemini 4 Argon	0.2570	$245.29	429.7M	1.7M	13h 47m	960	3
4	Gemini 4 Argon	0.1980	$156.36	254.9M	1.4M	8h 56m	624	2
5	Gemini 4 Argon	0.1446	$77.78	221.0M	1.0M	7h 15m	532	2

Dart Style in Haskell

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9654	$218.42	430.1M	2.0M	15h 3m	1010	21
2	Gemini 4 Argon	0.9568	$288.60	454.6M	2.0M	16h 22m	1093	14
3	Gemini 4 Argon	0.9534	$274.81	581.3M	2.0M	17h 49m	1368	15
4	Gemini 4 Argon	0.9529	$147.73	628.9M	2.2M	16h 46m	1478	23
5	Gemini 4 Argon	0.9457	$337.69	476.2M	2.1M	17h 14m	1055	18

FFmpeg libswscale Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.1572	$160.32	346.6M	1.9M	11h 31m	765	13
2	Gemini 4 Argon	0.0000	$156.84	296.0M	1.6M	9h 55m	668	11
3	Gemini 4 Argon	0.0000	$106.17	273.4M	1.3M	8h 28m	605	17
4	Gemini 4 Argon	0.0000	$58.88	167.7M	1.1M	5h 59m	405	7
5	Gemini 4 Argon	0.0000	$190.93	224.2M	1.2M	7h 43m	531	8

Fitness-Recap Video in Remotion

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.2451	$233.36	596.2M	1.7M	16h 49m	1260	12
2	Gemini 4 Argon	0.2401	$112.07	365.5M	1.1M	10h 13m	860	5
3	Gemini 4 Argon	0.2185	$165.03	335.8M	998k	11h 8m	805	2
4	Gemini 4 Argon	0.2055	$210.50	290.4M	915k	10h 7m	717	2
5	Gemini 4 Argon	0.1980	$159.82	263.8M	756k	8h 13m	636	3

Flight-Sim Renderer in OpenGL

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9334	$206.78	382.0M	1.3M	15h 47m	877	5
2	Gemini 4 Argon	0.9080	$207.90	379.4M	1.5M	15h 34m	854	10
3	Gemini 4 Argon	0.8458	$261.90	442.3M	1.8M	19h 2m	1056	6
4	Gemini 4 Argon	0.7148	$130.35	185.9M	816k	8h 32m	449	3
5	Gemini 4 Argon	0.7072	$170.81	258.5M	1.2M	8h 45m	673	3

FrogsGame Post-Training

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.8920	$46.04	49.7M	235k	14h 34m	172	5
2	Gemini 4 Argon	0.8020	$82.91	62.7M	281k	14h 57m	166	4
3	Gemini 4 Argon	0.6900	$75.10	61.9M	250k	17h 44m	191	4
4	Gemini 4 Argon	0.6040	$30.56	35.0M	214k	7h 42m	148	3
5	Gemini 4 Argon	0.5940	$87.69	65.7M	221k	8h 37m	175	3

Git to Zig

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.5447	$362.36	783.2M	2.2M	20h 48m	1753	26
2	Gemini 4 Argon	0.5225	$190.93	727.6M	2.5M	20h 51m	1657	23
3	Gemini 4 Argon	0.4697	$330.68	663.7M	1.9M	20h 55m	1569	11
4	Gemini 4 Argon	0.4294	$398.78	647.4M	2.0M	20h 52m	1487	26
5	Gemini 4 Argon	0.4167	$333.71	642.4M	2.0M	20h 59m	1402	31

Granite Mamba2 Inference Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9104	$105.47	115.9M	895k	5h 40m	315	6
2	Gemini 4 Argon	0.9046	$83.33	130.1M	792k	5h 37m	359	3
3	Gemini 4 Argon	0.8993	$145.24	118.2M	769k	6h 54m	331	8
4	Gemini 4 Argon	0.8884	$91.03	70.1M	500k	3h 12m	195	3
5	Gemini 4 Argon	0.8832	$36.18	98.9M	556k	3h 28m	236	3

Higgs Uncertainty Inference

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.3435	$96.85	59.0M	525k	6h 34m	155	2
2	Gemini 4 Argon	0.3199	$35.62	42.7M	359k	5h 46m	155	3
3	Gemini 4 Argon	0.2379	$42.74	116.3M	781k	7h 14m	281	4
4	Gemini 4 Argon	0.2135	$100.44	83.2M	842k	7h 14m	239	4
5	Gemini 4 Argon	0.1190	$36.03	53.9M	536k	3h 17m	162	3

Kolmogorov Audio Compression

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.1051	$267.43	253.9M	1.3M	20h 43m	650	14
2	Gemini 4 Argon	0.0786	$189.34	268.5M	1.6M	19h 23m	661	14
3	Gemini 4 Argon	0.0759	$254.33	211.1M	1.1M	19h 34m	466	11
4	Gemini 4 Argon	0.0648	$246.66	202.6M	896k	20h 43m	508	10
5	Gemini 4 Argon	0.0474	$174.59	205.7M	1.1M	13h 22m	472	18

Lean 4 Kernel Type Checker in Pascal

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	1.0000	$45.68	47.8M	303k	2h 17m	135	2
2	Gemini 4 Argon	1.0000	$81.62	85.5M	322k	3h 41m	204	2
3	Gemini 4 Argon	1.0000	$56.56	92.0M	304k	3h 17m	203	2
4	Gemini 4 Argon	0.9807	$13.24	39.9M	250k	1h 35m	117	2
5	Gemini 4 Argon	0.9807	$15.02	49.2M	292k	2h 22m	145	2

libexpat Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.6081	$51.02	180.3M	915k	5h 20m	388	8
2	Gemini 4 Argon	0.5881	$18.38	71.0M	446k	2h 16m	160	9
3	Gemini 4 Argon	0.5870	$86.35	125.1M	819k	4h 40m	340	4
4	Gemini 4 Argon	0.5630	$26.24	74.9M	691k	2h 55m	198	9
5	Gemini 4 Argon	0.4643	$17.19	66.1M	478k	2h 17m	158	4

Lua Native Compiler

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	1.0000	$73.29	90.2M	317k	3h 26m	212	2
2	Gemini 4 Argon	0.9976	$24.66	60.9M	275k	2h 36m	137	2
3	Gemini 4 Argon	0.9976	$34.72	50.2M	259k	2h 22m	136	2
4	Gemini 4 Argon	0.9976	$36.93	59.8M	371k	3h 58m	154	2
5	Gemini 4 Argon	0.9976	$21.71	52.8M	265k	2h 10m	142	2

Machine-Learned Interatomic Potential

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9973	$144.92	105.5M	303k	19h 54m	273	10
2	Gemini 4 Argon	0.9944	$61.24	55.2M	244k	20h 19m	189	14
3	Gemini 4 Argon	0.9938	$60.34	45.2M	281k	20h 31m	167	13
4	Gemini 4 Argon	0.9931	$56.18	57.5M	250k	20h 47m	209	10
5	Gemini 4 Argon	0.9823	$48.43	49.9M	216k	20h 21m	199	20

Medium-Range Weather Forecast

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.5653	$75.42	53.6M	408k	7h 44m	162	9
2	Gemini 4 Argon	0.5347	$151.23	88.8M	463k	15h 22m	239	10
3	Gemini 4 Argon	0.4825	$53.28	50.9M	579k	8h 27m	159	4
4	Gemini 4 Argon	0.4704	$122.30	82.4M	652k	20h 41m	193	10
5	Gemini 4 Argon	0.0000	$44.59	35.1M	340k	6h 53m	144	5

MEG Speech Decoding

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.3815	$127.65	83.3M	493k	10h 58m	219	7
2	Gemini 4 Argon	0.3541	$66.65	65.7M	370k	10h 16m	208	6
3	Gemini 4 Argon	0.3496	$73.61	73.2M	523k	11h 46m	215	9
4	Gemini 4 Argon	0.3253	$89.40	63.7M	382k	14h 54m	195	7
5	Gemini 4 Argon	0.1941	$62.22	52.9M	318k	10h 43m	182	5

MS/MS De Novo Generation

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.1524	$41.91	58.1M	431k	9h 28m	182	5
2	Gemini 4 Argon	0.1303	$43.39	66.9M	512k	10h 28m	195	7
3	Gemini 4 Argon	0.1252	$72.74	47.3M	391k	5h 42m	164	6
4	Gemini 4 Argon	0.0000	$51.82	59.1M	521k	9h 41m	178	5
5	Gemini 4 Argon	0.0000	$75.16	54.4M	420k	6h 54m	185	6

Multi-GPU Efficient Finetuning

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.3095	$59.08	60.8M	237k	16h 56m	198	5
2	Gemini 4 Argon	0.2143	$71.77	65.5M	249k	18h 37m	200	8
3	Gemini 4 Argon	0.1905	$119.59	80.8M	260k	19h 19m	230	3
4	Gemini 4 Argon	0.1429	$59.35	68.3M	219k	18h 18m	207	7
5	Gemini 4 Argon	0.1190	$99.16	76.3M	304k	18h 34m	233	4

Notebook Compression

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.4592	$350.30	290.6M	1.5M	20h 36m	690	25
2	Gemini 4 Argon	0.4467	$232.09	321.0M	1.6M	20h 40m	706	28
3	Gemini 4 Argon	0.4112	$223.42	270.8M	1.5M	21h 0m	634	23
4	Gemini 4 Argon	0.3971	$258.69	226.7M	1.1M	20h 39m	498	16
5	Gemini 4 Argon	0.3901	$194.88	235.0M	1.3M	15h 31m	566	13

Optimizer Design

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.3700	$138.18	88.1M	358k	17h 59m	225	7
2	Gemini 4 Argon	0.3302	$144.66	126.0M	552k	18h 30m	327	6
3	Gemini 4 Argon	0.2842	$147.52	93.7M	408k	17h 44m	246	6
4	Gemini 4 Argon	0.2729	$106.83	62.5M	361k	17h 2m	200	6
5	Gemini 4 Argon	0.2637	$128.39	105.4M	394k	12h 7m	261	6

PostgreSQL 18 on SQLite

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.6923	$41.63	62.6M	389k	2h 48m	192	2
2	Gemini 4 Argon	0.6923	$43.92	65.5M	430k	2h 31m	192	2
3	Gemini 4 Argon	0.6923	$26.32	82.0M	301k	2h 32m	224	2
4	Gemini 4 Argon	0.6923	$25.43	64.1M	365k	2h 39m	186	2
5	Gemini 4 Argon	0.6484	$34.67	93.0M	375k	2h 50m	250	2

Quantum ESPRESSO pw.x in Rust

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9187	$119.37	259.0M	1.1M	10h 5m	645	1
2	Gemini 4 Argon	0.9106	$303.04	318.6M	1.2M	11h 58m	691	2
3	Gemini 4 Argon	0.9024	$154.81	373.7M	1.6M	12h 47m	887	2
4	Gemini 4 Argon	0.9024	$388.42	369.1M	1.6M	13h 39m	867	6
5	Gemini 4 Argon	0.8699	$138.25	365.2M	1.5M	10h 38m	856	2

Qubit Routing

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.8617	$30.08	62.2M	527k	4h 53m	176	4
2	Gemini 4 Argon	0.8488	$43.62	67.5M	441k	6h 4m	194	5
3	Gemini 4 Argon	0.8155	$105.29	73.3M	505k	9h 43m	177	8
4	Gemini 4 Argon	0.7986	$85.40	56.4M	543k	6h 53m	150	4
5	Gemini 4 Argon	0.7982	$74.01	64.4M	466k	6h 51m	181	7

Reconnaissance Blind Chess Recovery

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9286	$102.13	88.3M	440k	7h 8m	224	7
2	Gemini 4 Argon	0.9214	$93.51	158.0M	759k	9h 17m	426	13
3	Gemini 4 Argon	0.8964	$116.54	131.7M	765k	9h 49m	373	8
4	Gemini 4 Argon	0.8893	$85.61	94.9M	445k	9h 1m	199	6
5	Gemini 4 Argon	0.8679	$74.43	68.7M	419k	7h 35m	192	4

SGLang Inference System Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.4359	$139.21	309.9M	955k	8h 57m	746	6
2	Gemini 4 Argon	0.0000	$275.58	621.4M	1.5M	15h 16m	1391	22
3	Gemini 4 Argon	0.0000	$100.53	344.7M	936k	8h 35m	828	8
4	Gemini 4 Argon	0.0000	$197.19	445.3M	1.3M	12h 41m	1022	9
5	Gemini 4 Argon	0.0000	$160.21	500.1M	1.5M	13h 56m	1169	20

Snooker Prediction

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.6242	$58.63	165.8M	890k	6h 8m	418	2
2	Gemini 4 Argon	0.5920	$39.04	128.3M	687k	6h 49m	357	3
3	Gemini 4 Argon	0.5407	$63.97	191.0M	855k	6h 41m	464	3
4	Gemini 4 Argon	0.4552	$134.67	137.5M	744k	5h 5m	372	3
5	Gemini 4 Argon	0.2214	$64.59	161.0M	705k	6h 32m	420	3

SPICE Circuit Simulator in Rust

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9394	$221.62	329.8M	1.9M	11h 7m	731	3
2	Gemini 4 Argon	0.9091	$147.35	234.0M	1.0M	8h 39m	503	6
3	Gemini 4 Argon	0.8889	$270.20	399.0M	2.4M	13h 19m	883	6
4	Gemini 4 Argon	0.8788	$203.70	185.4M	882k	6h 7m	426	3
5	Gemini 4 Argon	0.8283	$51.57	147.9M	741k	4h 35m	403	5

Stepper Music Sequencer GBA

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9654	$544.95	521.5M	2.1M	21h 0m	1162	3
2	Gemini 4 Argon	0.9520	$272.84	638.9M	3.5M	20h 55m	1382	4
3	Gemini 4 Argon	0.9258	$445.36	558.0M	2.3M	20h 59m	1265	1
4	Gemini 4 Argon	0.9237	$409.68	593.6M	2.6M	20h 55m	1445	7
5	Gemini 4 Argon	0.9165	$187.49	667.9M	3.3M	20h 59m	1507	2

Synthetic Music Diarization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.5622	$95.75	86.8M	749k	11h 34m	276	7
2	Gemini 4 Argon	0.5493	$59.76	63.4M	476k	9h 43m	201	5
3	Gemini 4 Argon	0.5456	$46.26	59.1M	483k	7h 58m	169	5
4	Gemini 4 Argon	0.5249	$103.39	100.4M	544k	9h 12m	246	8
5	Gemini 4 Argon	0.4215	$32.54	49.3M	491k	6h 13m	159	5

Verilog Simulator in Swift

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9349	$95.93	83.0M	397k	2h 56m	199	2
2	Gemini 4 Argon	0.9281	$95.89	72.5M	365k	2h 52m	178	3
3	Gemini 4 Argon	0.9281	$83.88	72.7M	366k	2h 46m	166	3
4	Gemini 4 Argon	0.9212	$27.80	68.8M	403k	2h 11m	165	4
5	Gemini 4 Argon	0.9178	$56.50	79.3M	431k	2h 53m	191	2

Vision-only TORCS Racing Bot

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.0000†	$81.71	58.0M	243k	8h 45m	180	3
2	Gemini 4 Argon	0.0000†	$54.62	48.6M	203k	6h 19m	169	2
3	Gemini 4 Argon	0.0000†	$123.80	83.2M	286k	8h 31m	204	3
4	Gemini 4 Argon	0.0000†	$78.71	63.3M	288k	10h 39m	188	5
5	Gemini 4 Argon	0.0000†	$96.39	87.4M	314k	10h 9m	198	3

† score adjusted by manual review; the trace's Judge tab carries the measured score and the rationale.

Wan 2.1 on MAX/Mojo

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Argon	0.9182	$55.03	65.4M	246k	4h 0m	171	2
2	Gemini 4 Argon	0.9115	$102.53	78.3M	305k	5h 26m	181	2
3	Gemini 4 Argon	0.9073	$129.59	105.1M	283k	6h 53m	216	2
4	Gemini 4 Argon	0.9069	$23.60	61.0M	217k	3h 19m	175	2
5	Gemini 4 Argon	0.1758	$81.73	98.5M	410k	6h 12m	259	1