Traces

Every run: the agent's full trajectory, its submissions over the 20-hour budget, and the final score.

Lean 4 Kernel Type Checker in Pascal

task page →50 runs · indexed 2026-09-02

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Claude Fable 5.1	1.0000	$34.17	49.9M	378k	2h 37m	164	6
2	Claude Fable 5.1	1.0000	$45.30	70.2M	418k	4h 0m	200	9
3	Claude Fable 5.1	1.0000	$34.22	50.9M	360k	1h 59m	159	8
4	Claude Fable 5.1	1.0000	$43.08	67.7M	379k	4h 3m	206	11
5	Claude Fable 5.1	1.0000	$50.91	83.1M	385k	3h 9m	232	8
6	GPT-5.6	0.9807	$221.16	222.7M	429k	6h 13m	691	19
7	GPT-5.6	0.9791	$78.01	86.8M	272k	4h 32m	395	2
8	GPT-5.6	0.9757	$207.31	199.1M	381k	7h 23m	620	10
9	Gemini 3.7 Flash	0.9448	$95.08	678.0M	1.4M	19h 49m	1660	1
10	Gemini 3.7 Flash	0.9361	$54.83	453.2M	1.3M	12h 43m	1224	1
11	GPT-5.6	0.9339	$340.15	337.0M	382k	8h 35m	789	11
12	Kimi K3	0.9198	$192.68	537.9M	1.4M	20h 0m	1218	1
13	GLM-5.3	0.8650	$139.71	481.8M	637k	18h 8m	859	6
14	GLM-5.3	0.8372	$111.12	401.3M	867k	19h 42m	904	10
15	Kimi K3	0.8313	$155.97	405.9M	1.5M	19h 47m	935	3
16	GPT-5.6	0.7874	$397.16	404.0M	396k	8h 54m	1134	7
17	Kimi K3	0.5265	$147.65	387.7M	1.4M	20h 2m	894	1
18	Kimi K3	0.4229	$171.33	477.1M	1.3M	19h 55m	1130	12
19	GLM-5.3	0.4065	$133.87	385.6M	660k	19h 35m	677	3
20	Grok 4.6	0.3824	$105.13	105.0M	1.0M	6h 34m	490	2
21	DeepSeek V4 Flash Vision Exp	0.3599	$12.71	725.7M	1.6M	10h 38m	1614	4
22	DeepSeek V4 Flash Vision Exp	0.3164	$9.48	551.4M	1.2M	7h 58m	1228	8
23	Kimi K3	0.2644	$151.43	423.8M	1.1M	20h 1m	928	1
24	DeepSeek V4 Flash Vision Exp	0.2513	$13.44	768.2M	1.8M	13h 43m	1760	7
25	Gemini 3.7 Flash	0.1248	$74.81	557.5M	1.1M	14h 22m	1217	1
26	GLM-5.3	0.1056	$134.85	474.9M	964k	19h 7m	1105	3
27	DeepSeek V4 Flash Vision Exp	0.0143	$12.56	761.2M	1.2M	12h 33m	1627	35
28	Gemini 3.7 Flash	0.0087	$48.94	384.1M	939k	13h 20m	1118	1
29	Muse Spark 1.2	0.0000	$19.14	108.5M	529k	2h 29m	348	1
30	Qwen3.8-Max	0.0000	$14.14	28.2M	1.1M	19h 59m	169	1
31	Inkling	0.0000	$5.56	5.7M	37k	1h 16m	115	2
32	Inkling	0.0000	$5.46	5.6M	34k	55m	107	2
33	Muse Spark 1.2	0.0000	$21.09	53.6M	699k	3h 48m	205	2
34	Muse Spark 1.2	0.0000	$29.46	162.8M	830k	6h 1m	404	3
35	Qwen3.8-Max	0.0000	$8.31	10.6M	914k	19h 59m	109	2
36	Inkling	0.0000	$5.70	5.9M	22k	14m	80	2
37	Grok 4.6	0.0000	$114.14	110.5M	1.1M	7h 47m	492	4
38	Muse Spark 1.2	0.0000	$27.30	88.9M	859k	4h 46m	278	1
39	Grok 4.6	0.0000	$407.54	406.7M	2.0M	19h 47m	1527	137
40	Grok 4.6	0.0000	$250.10	240.1M	1.8M	13h 13m	970	43
41	Inkling	0.0000	$13.81	14.4M	43k	1h 26m	175	1
42	DeepSeek V4 Flash Vision Exp	0.0000	$11.64	694.6M	1.3M	11h 22m	1413	33
43	Grok 4.6	0.0000	$233.84	217.8M	2.5M	19h 53m	929	30
44	Muse Spark 1.2	0.0000	$9.51	48.0M	382k	2h 35m	213	2
45	Gemini 3.7 Flash	0.0000	$101.50	861.0M	1.8M	19h 58m	2092	1
46	Qwen3.8-Max	0.0000	$13.92	25.5M	1.2M	19h 56m	167	2
47	Qwen3.8-Max	0.0000	$19.68	45.8M	1.3M	20h 0m	211	1
48	GLM-5.3	0.0000	$113.59	400.7M	1.0M	18h 31m	962	5
49	Qwen3.8-Max	0.0000	$10.35	13.6M	1.1M	19h 57m	115	1
50	Inkling	0.0000	$6.56	6.8M	25k	12m	109	6