Tasks Traces V1 Blog Changelog Contribute GitHub

Traces

Every run: the agent's full trajectory, its submissions over the 20-hour budget, and the final score.

Lean 4 Kernel Type Checker in Pascal

task page →110 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Claude Sonnet 5.5	1.0000	$56.70	219.1M	1.0M	5h 58m	501	6
2	Claude Haiku 5.5	1.0000	$12.16	186.9M	1.0M	4h 28m	423	4
3	Claude Sonnet 5.5	1.0000	$55.34	209.5M	1.1M	7h 29m	541	6
4	Claude Sonnet 5.5	1.0000	$53.61	204.3M	990k	7h 10m	485	6
5	Claude Opus 5	1.0000	$174.84	280.2M	919k	15h 16m	714	20
6	Claude Opus 5.5	1.0000	$41.28	133.2M	560k	3h 31m	263	13
7	Claude Fable 5.1	1.0000	$34.17	49.9M	378k	2h 37m	164	6
8	Claude Opus 5	1.0000	$204.51	327.4M	1.1M	13h 23m	746	19
9	Gemini 4 Argon	1.0000	$45.68	47.8M	303k	2h 17m	135	2
10	Claude Opus 5.5	1.0000	$71.71	241.5M	889k	6h 55m	573	21
11	Claude Fable 5.1	1.0000	$45.30	70.2M	418k	4h 0m	200	9
12	Gemini 4 Argon	1.0000	$81.62	85.5M	322k	3h 41m	204	2
13	Claude Fable 5	1.0000	$203.81	160.6M	603k	9h 37m	335	11
14	Claude Opus 5.5	1.0000	$35.94	109.8M	528k	2h 59m	232	8
15	Gemini 4 Argon	1.0000	$56.56	92.0M	304k	3h 17m	203	2
16	Claude Haiku 5.5	1.0000	$12.93	195.5M	1.2M	5h 31m	459	5
17	Claude Sonnet 5.5	1.0000	$55.03	208.1M	1.1M	6h 24m	495	7
18	Claude Fable 5	1.0000	$201.80	156.2M	629k	7h 5m	319	12
19	Claude Opus 5.5	1.0000	$31.11	88.5M	510k	2h 27m	202	4
20	Claude Opus 5.5	1.0000	$37.66	116.3M	544k	3h 20m	243	8
21	Claude Fable 5.1	1.0000	$34.22	50.9M	360k	1h 59m	159	8
22	Claude Fable 5.1	1.0000	$43.08	67.7M	379k	4h 3m	206	11
23	Claude Fable 5.1	1.0000	$50.91	83.1M	385k	3h 9m	232	8
24	Claude Opus 5	0.9984	$103.75	161.1M	640k	5h 43m	315	4
25	Claude Opus 5	0.9841	$206.83	330.5M	1.1M	10h 19m	761	12
26	Claude Haiku 5.5	0.9841	$12.26	189.0M	1.0M	5h 12m	467	2
27	Claude Haiku 5.5	0.9841	$9.10	133.0M	903k	3h 32m	370	2
28	Claude Fable 5	0.9841	$232.48	180.5M	725k	7h 59m	313	8
29	Claude Sonnet 5.5	0.9841	$56.48	214.6M	1.1M	12h 59m	513	8
30	Claude Haiku 5.5	0.9841	$10.51	161.8M	906k	4h 31m	389	2
31	GPT-6 Astra	0.9807	$122.08	64.3M	290k	2h 50m	282	4
32	GPT-5.6 Sol	0.9807	$221.16	222.7M	429k	6h 13m	691	19
33	Gemini 4 Argon	0.9807	$13.24	39.9M	250k	1h 35m	117	2
34	GPT-6 Astra	0.9807	$62.98	37.7M	245k	1h 38m	203	5
35	Gemini 4 Argon	0.9807	$15.02	49.2M	292k	2h 22m	145	2
36	GPT-6 Astra	0.9807	$123.05	61.8M	300k	2h 8m	260	3
37	GPT-6 Astra	0.9791	$34.15	22.2M	179k	1h 14m	133	3
38	GPT-6 Astra	0.9791	$174.62	85.4M	285k	2h 50m	311	4
39	GPT-5.6 Sol	0.9791	$78.01	86.8M	272k	4h 32m	395	2
40	Muse Spark 1.3	0.9776	$411.77	410.4M	713k	19h 52m	799	2
41	GPT-5.6 Sol	0.9757	$207.31	199.1M	381k	7h 23m	620	10
42	Claude Opus 5	0.9635	$169.99	269.9M	940k	17h 12m	655	17
43	Grok 4.7	0.9589	$192.69	192.9M	802k	7h 19m	790	8
44	Grok 4.7	0.9461	$478.73	472.8M	1.7M	19h 19m	1887	5
45	Gemini 3.7 Flash	0.9448	$95.08	678.0M	1.4M	19h 49m	1660	1
46	Gemini 3.7 Flash	0.9361	$54.83	453.2M	1.3M	12h 43m	1224	1
47	GPT-5.6 Sol	0.9339	$340.15	337.0M	382k	8h 35m	789	11
48	Kimi K3	0.9198	$192.68	537.9M	1.4M	20h 0m	1218	1
49	Grok 4.7	0.9103	$351.92	346.3M	1.1M	20h 0m	1425	6
50	Grok 4.7	0.8682	$294.16	294.8M	1.1M	13h 7m	1229	6
51	GLM-5.3	0.8650	$139.71	481.8M	637k	18h 8m	859	6
52	GLM-5.3	0.8372	$111.12	401.3M	867k	19h 42m	904	10
53	Kimi K3	0.8313	$155.97	405.9M	1.5M	19h 47m	935	3
54	GPT-5.6 Sol	0.7874	$397.16	404.0M	396k	8h 54m	1134	7
55	Muse Spark 1.3	0.7421	$154.98	818.9M	1.3M	19h 49m	1805	2
56	Muse Spark 1.3	0.5609	$411.55	416.2M	856k	19h 58m	959	1
57	GLM-5.3 Flash	0.5457	$13.38	394.6M	1.0M	13h 39m	939	5
58	Kimi K3	0.5265	$147.65	387.7M	1.4M	20h 2m	894	1
59	GLM-5.3 Flash	0.5060	$14.37	406.2M	1.1M	16h 35m	1001	3
60	Muse Spark 1.3	0.5029	$384.24	384.5M	940k	19h 50m	937	1
61	Kimi K3	0.4229	$171.33	477.1M	1.3M	19h 55m	1130	12
62	GLM-5.3	0.4065	$133.87	385.6M	660k	19h 35m	677	3
63	Grok 4.6	0.3824	$105.13	105.0M	1.0M	6h 34m	490	2
64	DeepSeek V4 Flash Vision Exp	0.3599	$12.71	725.7M	1.6M	10h 38m	1614	4
65	Muse Spark 1.3	0.3530	$389.61	408.6M	860k	19h 59m	1015	1
66	DeepSeek V4 Flash Vision Exp	0.3164	$9.48	551.4M	1.2M	7h 58m	1228	8
67	GLM-5.3 Flash	0.3164	$17.89	532.1M	1.2M	17h 29m	1198	4
68	Kimi K3	0.2644	$151.43	423.8M	1.1M	20h 1m	928	1
69	DeepSeek V4 Flash Vision Exp	0.2513	$13.44	768.2M	1.8M	13h 43m	1760	7
70	GLM-5.3 Flash	0.2321	$15.97	465.4M	1.2M	13h 20m	1049	31
71	Qwen3.8-Max-0902	0.1653	$45.56	98.3M	3.3M	20h 1m	294	1
72	Gemini 3.7 Flash	0.1248	$74.81	557.5M	1.1M	14h 22m	1217	1
73	GLM-5.3	0.1056	$134.85	474.9M	964k	19h 7m	1105	3
74	DeepSeek V4 Flash Vision Exp	0.0143	$12.56	761.2M	1.2M	12h 33m	1627	35
75	Qwen3.8-Max-0902	0.0111	$40.06	77.6M	3.2M	20h 1m	244	1
76	Gemini 3.7 Flash	0.0087	$48.94	384.1M	939k	13h 20m	1118	1
77	Muse Spark 1.2	0.0000	$19.14	108.5M	529k	2h 29m	348	1
78	Qwen3.8-Max	0.0000	$14.14	28.2M	1.1M	19h 59m	169	1
79	Inkling	0.0000	$5.56	5.7M	37k	1h 16m	115	2
80	Inkling	0.0000	$5.46	5.6M	34k	55m	107	2
81	Muse Spark 1.2	0.0000	$21.09	53.6M	699k	3h 48m	205	2
82	Muse Spark 1.2	0.0000	$29.46	162.8M	830k	6h 1m	404	3
83	Qwen3.8-Max	0.0000	$8.31	10.6M	914k	19h 59m	109	2
84	Inkling	0.0000	$5.70	5.9M	22k	14m	80	2
85	Qwen3.8-Max-0902	0.0000	$35.73	44.7M	3.0M	20h 1m	312	1
86	Grok 4.6	0.0000	$114.14	110.5M	1.1M	7h 47m	492	4
87	Muse Spark 1.2	0.0000	$27.30	88.9M	859k	4h 46m	278	1
88	Grok 4.6	0.0000	$407.54	406.7M	2.0M	19h 47m	1527	137
89	Gemini 3.8 Flash	0.0000	$90.95	821.2M	1.4M	20h 0m	2038	1
90	Grok 4.6	0.0000	$250.10	240.1M	1.8M	13h 13m	970	43
91	Inkling	0.0000	$13.81	14.4M	43k	1h 26m	175	1
92	Qwen3.8-Max-0902	0.0000	$33.43	45.6M	3.5M	20h 1m	184	1
93	DeepSeek V4 Flash Vision Exp	0.0000	$11.64	694.6M	1.3M	11h 22m	1413	33
94	Claude Fable 5	0.0000	$288.50	223.6M	888k	18h 15m	536	13
95	Grok 4.6	0.0000	$233.84	217.8M	2.5M	19h 53m	929	30
96	Muse Spark 1.2	0.0000	$9.51	48.0M	382k	2h 35m	213	2
97	Grok 4.7	0.0000	$418.16	407.2M	1.6M	18h 42m	1640	7
98	Claude Fable 5	0.0000	$144.73	104.4M	563k	6h 16m	235	13
99	GLM-5.3 Flash	0.0000	$17.57	476.0M	1.4M	17h 22m	1023	5
100	Gemini 3.8 Flash	0.0000	$69.03	618.4M	870k	14h 14m	1477	3
101	Gemini 3.8 Flash	0.0000	$77.67	724.4M	1.0M	15h 44m	1676	2
102	Gemini 3.7 Flash	0.0000	$101.50	861.0M	1.8M	19h 58m	2092	1
103	Gemini 3.8 Flash	0.0000	$82.03	739.4M	1.1M	17h 33m	1797	1
104	Qwen3.8-Max	0.0000	$13.92	25.5M	1.2M	19h 56m	167	2
105	Gemini 3.8 Flash	0.0000	$66.46	629.7M	880k	14h 44m	1599	3
106	Qwen3.8-Max	0.0000	$19.68	45.8M	1.3M	20h 0m	211	1
107	GLM-5.3	0.0000	$113.59	400.7M	1.0M	18h 31m	962	5
108	Qwen3.8-Max	0.0000	$10.35	13.6M	1.1M	19h 57m	115	1
109	Qwen3.8-Max-0902	0.0000	$22.28	25.4M	2.5M	14h 13m	151	1
110	Inkling	0.0000	$6.56	6.8M	25k	12m	109	6