Tasks Traces V1 Blog Changelog Contribute GitHub

Traces

Every run: the agent's full trajectory, its submissions over the 20-hour budget, and the final score.

Astronomy Toolkit

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.4022	$62.34	193.6M	1.6M	19h 35m	663	3
2	Qwen3.8-Max-0902	0.2897	$62.30	208.2M	1.4M	20h 2m	506	1
3	Qwen3.8-Max-0902	0.0063	$53.54	172.9M	1.4M	19h 54m	408	1
4	Qwen3.8-Max-0902	0.0016	$62.25	179.2M	2.0M	19h 45m	513	2
5	Qwen3.8-Max-0902	0.0003	$33.82	90.4M	1.2M	20h 0m	309	1

Cranelift Codegen Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0080	$64.48	159.6M	3.9M	19h 22m	453	1
2	Qwen3.8-Max-0902	0.0007	$62.21	152.4M	3.8M	19h 36m	511	2
3	Qwen3.8-Max-0902	0.0000	$41.40	90.2M	3.0M	20h 1m	362	2
4	Qwen3.8-Max-0902	0.0000	$33.03	53.9M	3.1M	20h 1m	263	1
5	Qwen3.8-Max-0902	0.0000	$33.18	49.8M	3.3M	20h 0m	260	1

Crash-Proof Flash Filesystem

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0057	$36.03	58.6M	3.4M	19h 58m	271	1
2	Qwen3.8-Max-0902	0.0008	$39.79	67.5M	3.7M	19h 56m	275	6
3	Qwen3.8-Max-0902	0.0008	$36.61	60.3M	3.5M	20h 1m	294	2
4	Qwen3.8-Max-0902	0.0008	$33.78	44.2M	3.7M	20h 1m	226	1
5	Qwen3.8-Max-0902	0.0006	$33.01	43.7M	3.6M	20h 1m	236	1

Dart Style in Haskell

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.2762	$68.64	209.5M	2.5M	19h 58m	650	1
2	Qwen3.8-Max-0902	0.1979	$53.96	142.4M	2.8M	19h 56m	444	1
3	Qwen3.8-Max-0902	0.0514	$56.12	143.5M	3.2M	20h 2m	493	1
4	Qwen3.8-Max-0902	0.0000	$93.37	320.2M	1.9M	20h 1m	755	2
5	Qwen3.8-Max-0902	0.0000	$44.97	92.5M	2.9M	20h 1m	511	1

FFmpeg libswscale Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	$39.61	68.0M	3.6M	20h 1m	275	1
2	Qwen3.8-Max-0902	$44.84	72.8M	3.4M	20h 2m	345	1
3	Qwen3.8-Max-0902	$51.17	115.3M	3.5M	20h 1m	358	1
4	Qwen3.8-Max-0902	$43.43	82.1M	3.6M	20h 1m	306	1
5	Qwen3.8-Max-0902	$72.15	209.7M	3.0M	19h 38m	543	1

Fitness-Recap Video in Remotion

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.1486	$33.48	89.8M	1.0M	20h 8m	410	2
2	Qwen3.8-Max-0902	0.1462	$31.95	68.8M	1.2M	20h 2m	351	1
3	Qwen3.8-Max-0902	0.0314	$43.03	134.8M	748k	20h 9m	405	1
4	Qwen3.8-Max-0902	0.0051	$35.22	96.8M	1.1M	20h 1m	342	1
5	Qwen3.8-Max-0902	0.0029	$28.31	69.9M	1.1M	20h 5m	355	1

Flight-Sim Renderer in OpenGL

task page →5 runs · indexed 2026-10-08

#	Model	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	$58.14	147.3M	957k	20h 6m	434	1
2	Qwen3.8-Max-0902	$33.94	103.0M	952k	20h 7m	367	1
3	Qwen3.8-Max-0902	$41.85	96.9M	886k	20h 4m	414	1
4	Qwen3.8-Max-0902	$50.29	127.1M	914k	20h 7m	479	1
5	Qwen3.8-Max-0902	$56.38	146.6M	1.2M	20h 0m	468	1

FrogsGame Post-Training

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0680	$32.05	102.3M	870k	19h 37m	406	2
2	Qwen3.8-Max-0902	0.0320	$32.58	103.5M	819k	19h 35m	415	2
3	Qwen3.8-Max-0902	0.0220	$35.21	102.2M	1.1M	19h 56m	399	—
4	Qwen3.8-Max-0902	0.0220	$33.10	101.6M	958k	19h 40m	430	3
5	Qwen3.8-Max-0902	0.0000	$4.94	9.5M	357k	5h 14m	93	1

Git to Zig

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.1143	$142.85	503.3M	2.3M	19h 27m	1335	8
2	Qwen3.8-Max-0902	0.0891	$199.99	712.3M	3.1M	19h 47m	1665	9
3	Qwen3.8-Max-0902	0.0536	$90.79	292.3M	2.5M	20h 1m	788	2
4	Qwen3.8-Max-0902	0.0529	$127.39	412.8M	3.6M	20h 1m	1158	1
5	Qwen3.8-Max-0902	0.0086	$101.78	346.1M	2.1M	20h 0m	916	2

Granite Mamba2 Inference Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.9095	$72.15	209.6M	2.3M	19h 47m	563	2
2	Qwen3.8-Max-0902	0.8140	$43.20	91.8M	3.2M	19h 47m	354	1
3	Qwen3.8-Max-0902	0.7969	$68.75	217.9M	2.1M	19h 23m	595	2
4	Qwen3.8-Max-0902	0.6109	$38.46	77.0M	3.0M	18h 44m	298	2
5	Qwen3.8-Max-0902	0.0000	$60.04	188.3M	1.9M	19h 52m	497	3

Higgs Uncertainty Inference

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.9183	$45.24	139.4M	1.5M	19h 44m	428	1
2	Qwen3.8-Max-0902	0.5260	$31.31	94.3M	1.1M	19h 31m	341	1
3	Qwen3.8-Max-0902	0.3378	$32.65	97.4M	1.2M	19h 38m	376	1
4	Qwen3.8-Max-0902	0.1773	$37.51	112.2M	1.0M	19h 48m	461	6
5	Qwen3.8-Max-0902	0.0834	$33.12	98.6M	1.2M	19h 40m	371	1

Kolmogorov Audio Compression

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0172	$73.87	221.2M	2.8M	19h 53m	570	2
2	Qwen3.8-Max-0902	0.0136	$47.03	132.4M	2.1M	19h 23m	432	2
3	Qwen3.8-Max-0902	0.0087	$44.20	104.3M	2.8M	19h 20m	333	2
4	Qwen3.8-Max-0902	0.0080	$44.17	113.8M	2.4M	19h 47m	337	1
5	Qwen3.8-Max-0902	0.0020	$43.94	119.8M	2.1M	19h 50m	371	2

Lean 4 Kernel Type Checker in Pascal

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.1653	$45.56	98.3M	3.3M	20h 1m	294	1
2	Qwen3.8-Max-0902	0.0111	$40.06	77.6M	3.2M	20h 1m	244	1
3	Qwen3.8-Max-0902	0.0000	$35.73	44.7M	3.0M	20h 1m	312	1
4	Qwen3.8-Max-0902	0.0000	$33.43	45.6M	3.5M	20h 1m	184	1
5	Qwen3.8-Max-0902	0.0000	$22.28	25.4M	2.5M	14h 13m	151	1

libexpat Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	$53.45	131.0M	3.2M	20h 1m	378	1
2	Qwen3.8-Max-0902	$42.48	90.3M	3.1M	19h 53m	348	4
3	Qwen3.8-Max-0902	$36.15	60.0M	3.4M	20h 0m	280	3
4	Qwen3.8-Max-0902	$95.18	281.1M	3.9M	19h 0m	632	3
5	Qwen3.8-Max-0902	$38.42	73.2M	3.2M	20h 0m	340	2

Lua Native Compiler

task page →5 runs · indexed 2026-10-08

#	Model	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	$32.39	39.9M	3.6M	20h 0m	206	1
2	Qwen3.8-Max-0902	$40.91	73.7M	3.6M	19h 57m	272	1
3	Qwen3.8-Max-0902	$31.88	33.4M	3.8M	20h 13m	209	1
4	Qwen3.8-Max-0902	$34.76	50.3M	3.5M	20h 2m	251	1
5	Qwen3.8-Max-0902	$91.51	260.0M	4.2M	19h 42m	578	1

Machine-Learned Interatomic Potential

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.9925	$42.93	126.0M	1.3M	20h 0m	617	2
2	Qwen3.8-Max-0902	0.9860	$42.29	135.8M	786k	19h 35m	535	2
3	Qwen3.8-Max-0902	0.9269	$37.45	125.1M	855k	19h 28m	494	1
4	Qwen3.8-Max-0902	0.9150	$18.33	54.0M	689k	20h 5m	323	1
5	Qwen3.8-Max-0902	0.6529	$37.06	123.0M	877k	19h 28m	448	2

Medium-Range Weather Forecast

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.5127	$30.58	103.4M	640k	19h 23m	449	1
2	Qwen3.8-Max-0902	0.4642	$26.04	87.1M	568k	18h 55m	401	2
3	Qwen3.8-Max-0902	0.4423	$42.33	144.4M	856k	19h 32m	548	1
4	Qwen3.8-Max-0902	0.3435	$56.59	197.4M	966k	18h 25m	518	6
5	Qwen3.8-Max-0902	0.0000	$27.25	86.2M	783k	19h 31m	389	3

MEG Speech Decoding

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.1889	$35.75	119.7M	797k	19h 10m	470	2
2	Qwen3.8-Max-0902	0.1871	$50.73	177.9M	853k	19h 26m	592	1
3	Qwen3.8-Max-0902	0.1258	$56.59	193.9M	1.1M	18h 55m	561	4
4	Qwen3.8-Max-0902	0.0904	$20.15	52.8M	623k	19h 46m	279	2
5	Qwen3.8-Max-0902	0.0000	$24.18	67.0M	860k	20h 1m	341	1

MS/MS De Novo Generation

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0947	$60.29	210.7M	1.0M	19h 43m	575	1
2	Qwen3.8-Max-0902	0.0000	$71.57	251.7M	1.2M	18h 34m	626	3
3	Qwen3.8-Max-0902	0.0000	$64.31	226.9M	1.0M	19h 20m	642	1
4	Qwen3.8-Max-0902	0.0000	$63.28	220.8M	1.1M	19h 41m	571	2
5	Qwen3.8-Max-0902	0.0000	$57.59	198.5M	1.1M	19h 22m	561	3

Multi-GPU Efficient Finetuning

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0476	$21.69	51.3M	683k	18h 46m	334	3
2	Qwen3.8-Max-0902	0.0476	$21.74	63.3M	874k	19h 49m	397	3
3	Qwen3.8-Max-0902	0.0000	$6.94	8.9M	369k	5h 21m	120	1
4	Qwen3.8-Max-0902	0.0000	$21.73	60.3M	968k	19h 32m	366	3
5	Qwen3.8-Max-0902	0.0000	$37.59	115.3M	1.0M	19h 35m	541	3

Notebook Compression

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.2600	$30.23	89.6M	1.1M	20h 1m	386	1
2	Qwen3.8-Max-0902	0.2513	$27.92	79.6M	1.2M	19h 13m	407	3
3	Qwen3.8-Max-0902	0.2416	$28.69	81.5M	1.3M	19h 35m	423	1
4	Qwen3.8-Max-0902	0.2219	$31.23	88.3M	1.4M	19h 46m	478	1
5	Qwen3.8-Max-0902	0.2150	$30.55	85.1M	1.4M	19h 31m	403	3

Optimizer Design

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.2342	$15.36	42.2M	714k	19h 22m	336	2
2	Qwen3.8-Max-0902	0.1854	$14.00	33.7M	859k	19h 56m	315	1
3	Qwen3.8-Max-0902	0.1351	$14.61	42.4M	583k	19h 28m	334	1
4	Qwen3.8-Max-0902	0.0958	$20.91	59.3M	909k	19h 56m	375	2
5	Qwen3.8-Max-0902	0.0000	$24.43	66.6M	1.2M	19h 37m	381	2

PostgreSQL 18 on SQLite

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0330	$183.61	654.3M	2.6M	19h 34m	1737	2
2	Qwen3.8-Max-0902	0.0330	$116.07	407.7M	1.9M	20h 1m	1122	2
3	Qwen3.8-Max-0902	0.0220	$150.66	544.1M	1.9M	20h 0m	1434	1
4	Qwen3.8-Max-0902	0.0165	$112.13	383.6M	1.8M	19h 52m	1132	5
5	Qwen3.8-Max-0902	0.0110	$104.91	366.4M	1.7M	20h 1m	1093	1

Quantum ESPRESSO pw.x in Rust

task page →5 runs · indexed 2026-10-08

#	Model	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	$27.89	47.8M	2.0M	20h 3m	236	1
2	Qwen3.8-Max-0902	$30.52	39.7M	2.0M	20h 1m	194	1
3	Qwen3.8-Max-0902	$26.48	47.3M	2.3M	20h 5m	231	1
4	Qwen3.8-Max-0902	$32.89	67.0M	2.0M	20h 5m	285	1
5	Qwen3.8-Max-0902	$26.30	38.4M	2.1M	20h 8m	222	1

Qubit Routing

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.7845	$41.21	136.5M	995k	20h 0m	480	2
2	Qwen3.8-Max-0902	0.7658	$47.17	161.4M	937k	19h 56m	531	1
3	Qwen3.8-Max-0902	0.7203	$52.67	179.1M	1.1M	19h 2m	520	2
4	Qwen3.8-Max-0902	0.6915	$35.54	115.2M	955k	20h 1m	444	1
5	Qwen3.8-Max-0902	0.6374	$51.28	174.8M	1.1M	20h 1m	568	2

Reconnaissance Blind Chess Recovery

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.2321	$22.10	36.7M	1.2M	15h 6m	239	1
2	Qwen3.8-Max-0902	0.0929	$21.34	36.0M	1.2M	15h 8m	263	1
3	Qwen3.8-Max-0902	0.0607	$20.03	44.0M	1.0M	15h 2m	256	1
4	Qwen3.8-Max-0902	0.0429	$15.23	34.3M	1.0M	15h 2m	220	1
5	Qwen3.8-Max-0902	0.0143	$12.30	21.2M	813k	15h 8m	188	1

SGLang Inference System Optimization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0242	$30.19	91.7M	767k	20h 6m	434	1
2	Qwen3.8-Max-0902	0.0009	$28.62	81.5M	1.2M	20h 5m	391	1
3	Qwen3.8-Max-0902	0.0009	$26.00	76.8M	998k	20h 6m	387	2
4	Qwen3.8-Max-0902	0.0000	$29.45	86.5M	1.2M	20h 1m	413	1
5	Qwen3.8-Max-0902	0.0000	$22.37	62.1M	1.0M	20h 3m	330	1

Snooker Prediction

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0341	$31.78	57.0M	1.5M	20h 7m	298	1
2	Qwen3.8-Max-0902	0.0275	$32.69	59.4M	1.3M	20h 2m	303	1
3	Qwen3.8-Max-0902	0.0000	$38.36	77.8M	1.3M	20h 2m	345	1
4	Qwen3.8-Max-0902	0.0000	$30.67	66.2M	1.6M	20h 2m	268	1
5	Qwen3.8-Max-0902	0.0000	$28.07	61.6M	1.3M	20h 0m	287	1

SPICE Circuit Simulator in Rust

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.4242	$87.16	281.5M	2.5M	19h 59m	601	1
2	Qwen3.8-Max-0902	0.4040	$64.29	188.8M	2.6M	20h 1m	516	1
3	Qwen3.8-Max-0902	0.3737	$124.78	385.3M	4.3M	20h 1m	989	2
4	Qwen3.8-Max-0902	0.3030	$73.95	234.0M	2.3M	20h 1m	470	1
5	Qwen3.8-Max-0902	0.2020	$65.44	194.8M	2.5M	19h 35m	444	2

Stepper Music Sequencer GBA

task page →5 runs · indexed 2026-10-08

#	Model	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	$43.55	102.3M	1.5M	20h 4m	428	1
2	Qwen3.8-Max-0902	$38.27	81.2M	1.4M	20h 9m	421	1
3	Qwen3.8-Max-0902	$38.10	91.8M	1.7M	20h 3m	413	1
4	Qwen3.8-Max-0902	$31.88	60.8M	1.6M	20h 2m	390	1
5	Qwen3.8-Max-0902	$41.37	105.1M	1.6M	20h 8m	436	1

Synthetic Music Diarization

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.4844	$67.12	233.9M	1.2M	19h 8m	645	4
2	Qwen3.8-Max-0902	0.4840	$91.60	324.7M	1.4M	19h 50m	793	1
3	Qwen3.8-Max-0902	0.4814	$101.81	367.2M	1.4M	19h 10m	818	2
4	Qwen3.8-Max-0902	0.4015	$71.54	249.6M	1.3M	19h 38m	666	2
5	Qwen3.8-Max-0902	0.3707	$81.42	285.5M	1.4M	19h 28m	710	2

Verilog Simulator in Swift

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.8219	$182.82	643.3M	3.3M	19h 35m	1529	3
2	Qwen3.8-Max-0902	0.8099	$199.79	724.6M	2.6M	19h 4m	1830	12
3	Qwen3.8-Max-0902	0.6473	$162.62	575.2M	2.7M	20h 1m	1239	2
4	Qwen3.8-Max-0902	0.5325	$172.11	623.9M	2.3M	19h 44m	1556	15
5	Qwen3.8-Max-0902	0.4966	$53.74	187.4M	939k	20h 0m	551	1

Vision-only TORCS Racing Bot

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.0000†	$21.92	49.4M	903k	20h 6m	303	2
2	Qwen3.8-Max-0902	0.0000†	$25.56	62.5M	975k	20h 8m	424	1
3	Qwen3.8-Max-0902	0.0000†	$48.14	156.8M	904k	19h 44m	531	1
4	Qwen3.8-Max-0902	0.0000†	$26.82	62.8M	787k	20h 4m	353	1
5	Qwen3.8-Max-0902	0.0000†	$28.34	75.5M	595k	20h 1m	372	1

† score adjusted by manual review; the trace's Judge tab carries the measured score and the rationale.

Wan 2.1 on MAX/Mojo

task page →5 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Qwen3.8-Max-0902	0.9041	$56.78	149.1M	2.5M	19h 20m	497	2
2	Qwen3.8-Max-0902	0.8953	$47.14	117.0M	1.9M	19h 7m	442	3
3	Qwen3.8-Max-0902	0.3268	$45.66	130.9M	1.8M	19h 28m	461	3
4	Qwen3.8-Max-0902	0.3247	$58.55	156.5M	2.2M	19h 45m	512	3
5	Qwen3.8-Max-0902	0.0446	$62.40	166.9M	2.3M	19h 54m	538	1