Tasks Traces V1 Blog Changelog Contribute GitHub

Traces

Every run: the agent's full trajectory, its submissions over the 20-hour budget, and the final score.

Astronomy Toolkit

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9259	$8.22	29.7M	260k	1h 31m	125	3
2	Gemini 4 Carbon	0.9046	$7.45	23.1M	248k	1h 15m	98	3
3	Gemini 4 Carbon	0.8999	$9.46	17.9M	258k	1h 25m	80	3
4	Gemini 4 Carbon	0.8058	$7.08	20.9M	281k	1h 23m	93	4
5	Gemini 4 Carbon	0.8058	$15.12	24.4M	286k	2h 5m	106	3

Cranelift Codegen Optimization

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.2874	$158.76	607.0M	2.3M	20h 0m	1316	31
2	Gemini 4 Carbon	0.1962	$162.25	700.7M	2.3M	19h 47m	1576	34
3	Gemini 4 Carbon	0.0000	$163.59	655.6M	2.5M	19h 55m	1467	36
4	Gemini 4 Carbon	0.0000	$141.98	606.5M	2.0M	19h 53m	1298	26
5	Gemini 4 Carbon	0.0000	$162.00	582.9M	2.1M	19h 37m	1351	33

Crash-Proof Flash Filesystem

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.2838	$29.63	127.0M	596k	3h 43m	280	3
2	Gemini 4 Carbon	0.1484	$32.30	139.7M	579k	5h 1m	327	3
3	Gemini 4 Carbon	0.1454	$27.17	107.9M	643k	4h 47m	233	4
4	Gemini 4 Carbon	0.0173	$29.25	113.8M	646k	4h 26m	268	3
5	Gemini 4 Carbon	0.0173	$29.51	97.8M	646k	5h 9m	248	3

Dart Style in Haskell

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9611	$66.56	305.1M	1.5M	9h 30m	743	19
2	Gemini 4 Carbon	0.9539	$71.82	359.3M	1.4M	11h 9m	825	21
3	Gemini 4 Carbon	0.9419	$55.72	253.6M	1.2M	7h 41m	602	18
4	Gemini 4 Carbon	0.9296	$53.84	256.6M	1.2M	7h 35m	618	14
5	Gemini 4 Carbon	0.9267	$34.26	184.5M	928k	5h 56m	437	13

FFmpeg libswscale Optimization

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.6845	$46.91	223.6M	1.3M	6h 54m	495	14
2	Gemini 4 Carbon	0.5793	$33.47	146.0M	1.1M	5h 43m	341	17
3	Gemini 4 Carbon	0.0000	$129.29	568.6M	3.6M	18h 50m	1222	42
4	Gemini 4 Carbon	0.0000	$61.98	257.6M	1.6M	9h 35m	577	20
5	Gemini 4 Carbon	0.0000	$31.20	152.0M	948k	4h 53m	340	14

Fitness-Recap Video in Remotion

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.2407	$125.51	394.8M	1.5M	19h 28m	899	26
2	Gemini 4 Carbon	0.2308	$102.07	384.6M	1.3M	20h 1m	813	34
3	Gemini 4 Carbon	0.2256	$113.52	340.8M	1.6M	19h 47m	747	33
4	Gemini 4 Carbon	0.2247	$126.19	378.0M	1.1M	14h 22m	758	30
5	Gemini 4 Carbon	0.2232	$90.68	338.4M	1.4M	19h 15m	761	38

Flight-Sim Renderer in OpenGL

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9268	$112.64	321.3M	1.4M	19h 55m	763	11
2	Gemini 4 Carbon	0.8907	$78.20	248.2M	1.1M	13h 8m	515	11
3	Gemini 4 Carbon	0.8842	$58.51	166.7M	956k	13h 14m	408	8
4	Gemini 4 Carbon	0.8825	$61.99	196.8M	1.1M	12h 30m	439	7
5	Gemini 4 Carbon	0.8458	$58.27	224.6M	1.0M	11h 25m	507	8

FrogsGame Post-Training

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9580	$32.78	26.8M	208k	19h 58m	122	3
2	Gemini 4 Carbon	0.9500	$31.31	31.7M	172k	19h 52m	134	4
3	Gemini 4 Carbon	0.9100	$38.94	37.9M	210k	19h 55m	154	4
4	Gemini 4 Carbon	0.8980	$36.30	29.8M	192k	19h 58m	131	6
5	Gemini 4 Carbon	0.8000	$40.21	36.7M	231k	20h 0m	156	3

Git to Zig

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.5929	$128.87	637.6M	2.6M	20h 0m	1388	18
2	Gemini 4 Carbon	0.5770	$156.73	612.4M	2.5M	20h 1m	1354	24
3	Gemini 4 Carbon	0.5651	$147.14	549.9M	2.4M	19h 56m	1283	23
4	Gemini 4 Carbon	0.5598	$149.95	565.0M	2.2M	20h 1m	1250	16
5	Gemini 4 Carbon	0.5322	$151.34	576.1M	2.3M	19h 59m	1280	24

Granite Mamba2 Inference Optimization

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9288	$39.64	111.6M	938k	5h 49m	256	5
2	Gemini 4 Carbon	0.9269	$24.61	49.2M	507k	3h 3m	125	5
3	Gemini 4 Carbon	0.9267	$38.69	58.6M	631k	3h 36m	142	4
4	Gemini 4 Carbon	0.9260	$21.81	70.0M	515k	3h 23m	187	4
5	Gemini 4 Carbon	0.8974	$39.68	102.0M	1.2M	6h 41m	270	11

Higgs Uncertainty Inference

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	1.0000	$44.39	42.7M	441k	12h 17m	130	4
2	Gemini 4 Carbon	1.0000	$48.88	61.2M	497k	11h 1m	168	5
3	Gemini 4 Carbon	1.0000	$45.29	54.8M	438k	12h 40m	142	5
4	Gemini 4 Carbon	0.9593	$73.09	82.0M	606k	16h 42m	204	4
5	Gemini 4 Carbon	0.8560	$60.09	65.4M	511k	13h 36m	186	5

Kolmogorov Audio Compression

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.0807	$94.40	89.3M	634k	19h 36m	203	8
2	Gemini 4 Carbon	0.0786	$101.78	79.7M	733k	19h 56m	159	7
3	Gemini 4 Carbon	0.0714	$95.60	93.0M	625k	19h 51m	211	10
4	Gemini 4 Carbon	0.0692	$82.91	81.2M	552k	19h 46m	199	7
5	Gemini 4 Carbon	0.0654	$95.34	111.6M	647k	19h 19m	242	11

Lean 4 Kernel Type Checker in Pascal

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	1.0000	$16.76	53.0M	382k	2h 4m	133	2
2	Gemini 4 Carbon	1.0000	$10.63	38.3M	340k	1h 46m	129	2
3	Gemini 4 Carbon	1.0000	$20.82	62.2M	414k	2h 55m	157	2
4	Gemini 4 Carbon	1.0000	$24.91	59.3M	417k	3h 35m	161	2
5	Gemini 4 Carbon	0.9791	$17.61	37.8M	349k	2h 1m	118	2

libexpat Optimization

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.5702	$17.44	76.3M	522k	2h 14m	186	4
2	Gemini 4 Carbon	0.5613	$13.74	53.5M	502k	2h 33m	141	5
3	Gemini 4 Carbon	0.5291	$14.66	47.9M	485k	2h 10m	129	5
4	Gemini 4 Carbon	0.4501	$11.20	49.3M	410k	2h 38m	146	4
5	Gemini 4 Carbon	0.3847	$14.61	65.7M	517k	3h 0m	162	4

Lua Native Compiler

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	1.0000	$18.24	52.7M	358k	2h 35m	156	3
2	Gemini 4 Carbon	1.0000	$23.92	72.7M	456k	2h 53m	167	3
3	Gemini 4 Carbon	1.0000	$10.86	37.2M	314k	1h 50m	133	2
4	Gemini 4 Carbon	0.9528	$11.49	47.4M	307k	1h 53m	155	2
5	Gemini 4 Carbon	0.9410	$9.59	33.3M	257k	1h 26m	120	3

Machine-Learned Interatomic Potential

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9902	$33.70	26.5M	249k	19h 34m	135	6
2	Gemini 4 Carbon	0.9857	$43.87	38.5M	333k	19h 20m	151	8
3	Gemini 4 Carbon	0.9809	$36.29	29.0M	279k	19h 28m	124	5
4	Gemini 4 Carbon	0.9772	$36.90	34.1M	245k	19h 43m	159	8
5	Gemini 4 Carbon	0.9668	$35.57	29.7M	255k	19h 49m	137	7

Medium-Range Weather Forecast

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.6348	$39.14	38.2M	277k	19h 59m	149	9
2	Gemini 4 Carbon	0.6301	$42.51	40.4M	318k	19h 49m	171	10
3	Gemini 4 Carbon	0.6175	$38.47	42.8M	321k	19h 54m	184	7
4	Gemini 4 Carbon	0.6098	$67.71	69.1M	303k	19h 37m	223	8
5	Gemini 4 Carbon	0.5959	$59.22	80.7M	422k	19h 46m	249	9

MEG Speech Decoding

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.3546	$110.79	105.4M	430k	19h 42m	278	11
2	Gemini 4 Carbon	0.3391	$55.84	65.8M	330k	19h 40m	227	8
3	Gemini 4 Carbon	0.3341	$55.56	60.6M	332k	19h 46m	223	8
4	Gemini 4 Carbon	0.3110	$74.47	77.8M	358k	19h 50m	243	5
5	Gemini 4 Carbon	0.2901	$78.67	101.9M	436k	19h 59m	273	6

MS/MS De Novo Generation

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.1427	$37.17	56.2M	371k	18h 24m	210	6
2	Gemini 4 Carbon	0.1315	$33.30	44.8M	358k	18h 21m	183	6
3	Gemini 4 Carbon	0.1265	$45.86	66.3M	400k	18h 28m	200	5
4	Gemini 4 Carbon	0.1175	$37.13	51.1M	351k	18h 31m	200	6
5	Gemini 4 Carbon	0.1126	$57.88	111.8M	491k	19h 2m	293	9

Multi-GPU Efficient Finetuning

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.3810	$31.53	27.6M	200k	19h 50m	127	5
2	Gemini 4 Carbon	0.3810	$39.00	42.5M	193k	19h 50m	178	4
3	Gemini 4 Carbon	0.3333	$35.15	31.4M	180k	19h 44m	136	5
4	Gemini 4 Carbon	0.1429	$44.54	44.2M	256k	19h 38m	179	4
5	Gemini 4 Carbon	0.1190	$37.42	40.0M	214k	19h 50m	165	5

Notebook Compression

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.5387	$130.03	248.7M	1.8M	19h 48m	562	16
2	Gemini 4 Carbon	0.5092	$108.09	218.0M	1.7M	19h 51m	495	16
3	Gemini 4 Carbon	0.4955	$125.20	215.7M	1.4M	19h 27m	458	15
4	Gemini 4 Carbon	0.4665	$107.43	278.5M	2.3M	20h 2m	655	21
5	Gemini 4 Carbon	0.4640	$139.93	232.8M	1.6M	20h 4m	541	17

Optimizer Design

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.3306	$63.61	105.5M	508k	19h 32m	281	12
2	Gemini 4 Carbon	0.3267	$52.49	116.9M	551k	18h 23m	308	16
3	Gemini 4 Carbon	0.3033	$84.93	125.2M	506k	19h 32m	328	13
4	Gemini 4 Carbon	0.2918	$89.02	127.5M	481k	19h 9m	303	6
5	Gemini 4 Carbon	0.2476	$50.69	46.6M	317k	19h 55m	170	5

PostgreSQL 18 on SQLite

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.5769	$157.79	759.6M	2.3M	20h 1m	1721	19
2	Gemini 4 Carbon	0.5495	$138.96	655.4M	2.3M	19h 59m	1509	21
3	Gemini 4 Carbon	0.5440	$147.39	651.3M	2.1M	19h 52m	1467	18
4	Gemini 4 Carbon	0.5330	$139.16	687.9M	2.3M	19h 46m	1586	14
5	Gemini 4 Carbon	0.5275	$155.78	748.2M	2.5M	20h 1m	1644	15

Quantum ESPRESSO pw.x in Rust

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.8943	$49.66	188.3M	1.1M	8h 17m	441	4
2	Gemini 4 Carbon	0.8943	$78.28	256.0M	1.2M	12h 11m	607	5
3	Gemini 4 Carbon	0.8780	$58.29	215.8M	1.1M	7h 25m	492	5
4	Gemini 4 Carbon	0.8699	$75.46	221.0M	1.2M	9h 56m	534	5
5	Gemini 4 Carbon	0.8537	$65.85	221.2M	1.2M	7h 55m	480	4

Qubit Routing

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.8914	$42.24	55.7M	434k	7h 6m	166	6
2	Gemini 4 Carbon	0.8750	$43.42	50.0M	403k	10h 16m	168	13
3	Gemini 4 Carbon	0.8722	$44.40	61.1M	483k	8h 28m	185	12
4	Gemini 4 Carbon	0.8676	$39.63	47.2M	472k	7h 44m	153	8
5	Gemini 4 Carbon	0.8570	$54.08	72.6M	540k	7h 39m	190	13

Reconnaissance Blind Chess Recovery

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9786	$44.14	57.0M	332k	9h 46m	196	8
2	Gemini 4 Carbon	0.9643	$68.75	85.0M	399k	8h 10m	200	10
3	Gemini 4 Carbon	0.9643	$57.25	70.5M	418k	14h 24m	198	8
4	Gemini 4 Carbon	0.9536	$36.29	40.2M	343k	11h 55m	137	8
5	Gemini 4 Carbon	0.9429	$70.15	81.1M	395k	14h 34m	220	8

SGLang Inference System Optimization

task page →5 runs · indexed 2026-11-12

#	Model	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	$125.13	534.0M	2.2M	18h 47m	1191	25
2	Gemini 4 Carbon	$107.31	433.6M	2.0M	14h 18m	919	19
3	Gemini 4 Carbon	$113.64	517.7M	2.3M	18h 58m	1164	27
4	Gemini 4 Carbon	$146.32	601.4M	2.3M	18h 3m	1270	26
5	Gemini 4 Carbon	$128.67	547.6M	2.2M	18h 33m	1195	26

Snooker Prediction

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.6007	$47.61	98.3M	577k	5h 39m	241	6
2	Gemini 4 Carbon	0.5365	$40.62	123.2M	547k	5h 34m	277	8
3	Gemini 4 Carbon	0.5355	$42.48	69.0M	425k	5h 38m	190	5
4	Gemini 4 Carbon	0.4702	$74.49	187.8M	914k	9h 55m	458	7
5	Gemini 4 Carbon	0.4544	$52.60	116.1M	536k	6h 1m	265	8

SPICE Circuit Simulator in Rust

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9293	$55.45	239.2M	1.7M	7h 8m	523	6
2	Gemini 4 Carbon	0.9192	$73.79	296.9M	2.2M	9h 16m	698	6
3	Gemini 4 Carbon	0.9192	$57.57	230.1M	1.6M	7h 20m	530	6
4	Gemini 4 Carbon	0.9192	$49.77	221.6M	1.5M	6h 44m	497	9
5	Gemini 4 Carbon	0.9192	$90.27	376.7M	3.0M	12h 41m	833	8

Stepper Music Sequencer GBA

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9872	$97.04	446.9M	2.1M	15h 5m	979	12
2	Gemini 4 Carbon	0.9708	$36.92	167.8M	841k	5h 35m	442	6
3	Gemini 4 Carbon	0.9708	$43.61	230.9M	934k	8h 4m	558	9
4	Gemini 4 Carbon	0.9640	$49.83	228.0M	979k	6h 41m	497	8
5	Gemini 4 Carbon	0.9392	$55.85	271.2M	1.0M	7h 41m	575	6

Synthetic Music Diarization

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.5810	$37.66	46.7M	294k	19h 43m	201	6
2	Gemini 4 Carbon	0.5627	$36.13	35.9M	284k	19h 59m	150	4
3	Gemini 4 Carbon	0.5536	$40.60	43.3M	310k	18h 47m	161	6
4	Gemini 4 Carbon	0.5454	$39.14	38.1M	299k	18h 41m	157	5
5	Gemini 4 Carbon	0.5444	$44.84	44.7M	333k	19h 49m	167	4

Verilog Simulator in Swift

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9469	$16.56	68.5M	477k	2h 50m	190	4
2	Gemini 4 Carbon	0.9349	$22.85	73.1M	504k	2h 43m	168	4
3	Gemini 4 Carbon	0.9315	$18.29	54.0M	458k	2h 22m	155	4
4	Gemini 4 Carbon	0.9298	$17.02	69.1M	502k	2h 53m	174	4
5	Gemini 4 Carbon	0.9298	$15.05	47.7M	439k	2h 28m	148	4

Vision-only TORCS Racing Bot

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.5385	$91.84	101.5M	425k	18h 27m	266	13
2	Gemini 4 Carbon	0.3702	$64.09	66.9M	452k	19h 58m	219	4
3	Gemini 4 Carbon	0.2900	$63.15	75.8M	441k	17h 20m	246	8
4	Gemini 4 Carbon	0.2490	$60.03	76.0M	377k	17h 4m	250	5
5	Gemini 4 Carbon	0.1504	$75.41	87.9M	520k	17h 29m	235	7

Wan 2.1 on MAX/Mojo

task page →5 runs · indexed 2026-11-12

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Gemini 4 Carbon	0.9363	$25.01	64.2M	323k	3h 52m	179	3
2	Gemini 4 Carbon	0.9187	$24.34	50.3M	315k	3h 44m	155	2
3	Gemini 4 Carbon	0.9155	$31.96	52.5M	317k	4h 35m	160	3
4	Gemini 4 Carbon	0.9110	$19.74	47.1M	319k	3h 25m	147	2
5	Gemini 4 Carbon	0.9097	$16.95	49.5M	262k	2h 51m	153	2