Tasks Traces V1 Blog Changelog Contribute GitHub

Traces

Every run: the agent's full trajectory, its submissions over the 20-hour budget, and the final score.

Cranelift Codegen Optimization

task page →109 runs · indexed 2026-10-08

#	Model	Reward	Cost	In tok	Out tok	Time	Steps	Submits
1	Claude Sonnet 5.5	0.5217	$188.61	792.1M	2.3M	19h 20m	1836	19
2	Claude Sonnet 5.5	0.4700	$138.58	570.3M	1.8M	19h 30m	1309	10
3	Gemini 4 Argon	0.4043	$256.70	491.0M	1.5M	20h 49m	1116	24
4	Claude Opus 5.5	0.3651	$135.88	506.9M	1.2M	19h 1m	1217	34
5	Claude Sonnet 5.5	0.3536	$165.14	693.2M	2.0M	18h 53m	1535	3
6	Claude Fable 5.1	0.3484	$166.03	303.2M	604k	16h 29m	825	13
7	Claude Fable 5.1	0.3356	$233.87	425.4M	876k	16h 37m	1021	18
8	GPT-6 Astra	0.3006	$2589.93	1231.7M	1.9M	19h 38m	2587	122
9	Claude Opus 5.5	0.2984	$145.49	543.9M	1.3M	18h 43m	1227	39
10	Claude Opus 5	0.2915	$236.87	396.3M	962k	16h 44m	955	7
11	Claude Fable 5.1	0.2852	$199.91	365.0M	723k	15h 17m	974	15
12	Claude Opus 5.5	0.2828	$138.56	513.8M	1.3M	19h 19m	1211	30
13	Claude Sonnet 5.5	0.2621	$131.90	545.0M	1.7M	19h 41m	1250	18
14	Claude Fable 5	0.2501	$498.77	425.9M	919k	16h 22m	1061	44
15	Claude Opus 5.5	0.2500	$129.78	475.1M	1.3M	18h 48m	1162	27
16	GPT-6 Astra	0.2357	$2732.07	1287.7M	2.4M	19h 34m	2829	97
17	Gemini 4 Argon	0.2201	$298.10	572.4M	2.1M	20h 43m	1256	24
18	GPT-6 Astra	0.2194	$2585.69	1232.6M	2.0M	18h 22m	2643	79
19	Claude Opus 5.5	0.2063	$105.12	402.9M	850k	17h 7m	905	27
20	Claude Fable 5.1	0.2009	$170.71	311.4M	625k	17h 11m	813	17
21	Gemini 4 Argon	0.1959	$361.37	454.4M	1.3M	17h 22m	1002	27
22	Claude Sonnet 5.5	0.1812	$160.59	674.3M	1.9M	17h 6m	1513	4
23	Claude Fable 5	0.1789	$444.68	370.1M	951k	17h 14m	918	22
24	GPT-6 Astra	0.1650	$2812.60	1334.7M	2.3M	19h 56m	2934	92
25	Claude Opus 5	0.1550	$162.82	258.1M	858k	16h 52m	679	10
26	Claude Haiku 5.5	0.1347	$36.63	614.9M	2.1M	17h 29m	1443	8
27	Gemini 4 Argon	0.1238	$407.04	524.2M	1.6M	20h 47m	1255	28
28	Claude Opus 5	0.1213	$221.80	357.8M	1.1M	16h 0m	789	12
29	Claude Opus 5	0.0902	$217.40	355.4M	1.0M	14h 55m	890	16
30	Claude Haiku 5.5	0.0815	$13.27	217.3M	900k	5h 19m	586	1
31	Gemini 3.7 Flash	0.0802	$35.86	231.6M	242k	6h 15m	574	20
32	Kimi K3	0.0603	$148.59	441.7M	786k	16h 22m	1028	15
33	Claude Fable 5	0.0600	$397.22	333.0M	807k	17h 59m	852	28
34	Kimi K3	0.0478	$126.89	362.2M	801k	17h 53m	936	9
35	DeepSeek V4 Flash Vision Exp	0.0237	$8.27	504.6M	723k	9h 18m	1260	11
36	Muse Spark 1.3	0.0231	$411.74	433.6M	529k	18h 32m	875	4
37	Gemini 3.8 Flash	0.0215	$33.75	333.3M	357k	7h 19m	852	10
38	Muse Spark 1.3	0.0192	$411.77	452.6M	517k	18h 56m	944	8
39	Gemini 3.7 Flash	0.0189	$20.57	157.5M	204k	2h 49m	432	17
40	GPT-5.6 Sol	0.0176	$225.59	218.6M	216k	6h 56m	506	6
41	Muse Spark 1.3	0.0168	$495.72	581.2M	737k	19h 59m	1530	3
42	GLM-5.3	0.0162	$93.99	346.3M	474k	11h 28m	766	2
43	Claude Haiku 5.5	0.0152	$18.59	304.3M	1.2M	8h 22m	679	2
44	Kimi K3	0.0132	$145.36	415.9M	1.0M	20h 3m	1079	7
45	GLM-5.3	0.0129	$109.37	311.0M	484k	17h 22m	770	9
46	Qwen3.8-Max	0.0117	$41.08	108.0M	2.2M	18h 38m	484	6
47	GLM-5.3 Flash	0.0115	$9.01	238.7M	447k	10h 39m	679	4
48	Gemini 3.7 Flash	0.0110	$23.00	164.8M	188k	4h 11m	495	19
49	GLM-5.3 Flash	0.0109	$12.92	356.8M	427k	14h 26m	958	4
50	DeepSeek V4 Flash Vision Exp	0.0107	$5.01	303.8M	439k	7h 50m	723	8
51	GLM-5.3	0.0102	$60.68	223.7M	392k	7h 14m	615	4
52	Qwen3.8-Max-0902	0.0080	$64.48	159.6M	3.9M	19h 22m	453	1
53	Gemini 3.8 Flash	0.0078	$39.55	408.0M	435k	8h 49m	1021	22
54	DeepSeek V4 Flash Vision Exp	0.0030	$5.08	311.7M	419k	8h 3m	781	6
55	Qwen3.8-Max-0902	0.0007	$62.21	152.4M	3.8M	19h 36m	511	2
56	Grok 4.6	0.0000	$113.60	103.4M	918k	9h 35m	426	6
57	Inkling	0.0000	$8.38	8.8M	12k	55m	94	1
58	Gemini 3.8 Flash	0.0000	$49.95	454.4M	486k	9h 11m	1075	17
59	Muse Spark 1.3	0.0000	$472.59	577.9M	275k	18h 58m	1276	12
60	Grok 4.7	0.0000	$645.99	652.2M	2.1M	20h 0m	2589	15
61	GPT-5.6 Sol	0.0000	$297.21	276.6M	238k	7h 17m	549	5
62	Muse Spark 1.2	0.0000	$33.79	127.0M	205k	10h 23m	356	5
63	Gemini 3.8 Flash	0.0000	$54.64	599.7M	676k	11h 22m	1488	23
64	Claude Haiku 5.5	0.0000	$17.35	279.6M	1.2M	8h 20m	635	4
65	GLM-5.3 Flash	0.0000	$9.07	226.2M	412k	9h 11m	645	4
66	Qwen3.8-Max-0902	0.0000	$41.40	90.2M	3.0M	20h 1m	362	2
67	Muse Spark 1.2	0.0000	$18.72	115.3M	194k	3h 23m	350	4
68	Qwen3.8-Max	0.0000	$34.88	81.5M	2.3M	20h 0m	413	1
69	Kimi K3	0.0000	$136.80	389.4M	1.1M	20h 0m	1006	19
70	GLM-5.3 Flash	0.0000	$9.52	253.8M	421k	12h 41m	720	2
71	Claude Opus 5	0.0000	$223.36	369.6M	980k	18h 26m	878	18
72	Qwen3.8-Max-0902	0.0000	$33.03	53.9M	3.1M	20h 1m	263	1
73	Gemini 4 Argon	0.0000	$212.69	506.0M	1.7M	20h 46m	1184	27
74	Gemini 3.7 Flash	0.0000	$30.61	210.1M	211k	5h 15m	593	21
75	GPT-5.6 Sol	0.0000	$234.61	222.5M	240k	7h 38m	461	5
76	Qwen3.8-Max-0902	0.0000	$33.18	49.8M	3.3M	20h 0m	260	1
77	Grok 4.6	0.0000	$240.25	234.7M	1.3M	16h 47m	895	24
78	Inkling	0.0000	$13.75	14.4M	22k	1h 40m	135	2
79	Muse Spark 1.3	0.0000	$394.96	454.8M	322k	15h 45m	1021	12
80	Claude Fable 5	0.0000	$391.33	323.6M	852k	17h 7m	849	24
81	Kimi K3	0.0000	$136.72	403.0M	752k	19h 8m	1147	11
82	Grok 4.7	0.0000	$228.14	230.9M	694k	6h 44m	948	7
83	Inkling	0.0000	$11.79	12.4M	15k	1h 2m	121	1
84	Muse Spark 1.2	0.0000	$87.58	303.3M	711k	10h 13m	562	5
85	GPT-5.6 Sol	0.0000	$212.19	205.4M	208k	5h 52m	461	5
86	Inkling	0.0000	$8.24	8.6M	21k	30m	104	2
87	DeepSeek V4 Flash Vision Exp	0.0000	$3.68	220.8M	344k	8h 44m	670	8
88	Inkling	0.0000	$13.82	14.5M	21k	59m	137	2
89	Grok 4.6	0.0000	$138.74	131.9M	1.2M	8h 47m	587	6
90	Gemini 3.8 Flash	0.0000	$43.81	479.7M	493k	7h 38m	1152	19
91	Claude Fable 5.1	0.0000	$232.11	427.5M	765k	16h 0m	1023	35
92	Grok 4.7	0.0000	$477.48	483.7M	1.5M	16h 55m	1975	20
93	Grok 4.6	0.0000	$138.58	127.2M	927k	7h 29m	488	9
94	Muse Spark 1.2	0.0000	$76.77	223.4M	655k	10h 10m	450	7
95	Muse Spark 1.2	0.0000	$49.18	242.6M	532k	8h 28m	495	9
96	Claude Haiku 5.5	0.0000	$32.91	562.1M	1.7M	19h 9m	1233	14
97	Qwen3.8-Max	0.0000	$43.78	119.5M	2.2M	18h 46m	496	2
98	Qwen3.8-Max	0.0000	$50.74	159.1M	1.7M	16h 16m	624	4
99	GLM-5.3	0.0000	$110.08	353.7M	515k	18h 43m	761	18
100	Grok 4.7	0.0000	$635.61	634.0M	1.8M	19h 42m	2545	20
101	Gemini 3.7 Flash	0.0000	$47.46	341.2M	378k	8h 44m	828	22
102	GLM-5.3 Flash	0.0000	$13.40	389.9M	465k	13h 31m	921	2
103	Grok 4.6	0.0000	$108.30	103.9M	740k	6h 37m	447	6
104	DeepSeek V4 Flash Vision Exp	0.0000	$4.87	296.6M	422k	6h 43m	730	3
105	Claude Fable 5	0.0000	$346.79	290.6M	710k	18h 28m	737	21
106	GLM-5.3	0.0000	$69.79	258.7M	392k	7h 49m	703	7
107	Grok 4.7	0.0000	$640.87	669.7M	1.8M	19h 25m	2877	27
108	GPT-5.6 Sol	0.0000	$355.12	344.3M	254k	10h 32m	627	3
109	Qwen3.8-Max	0.0000	$42.54	115.4M	2.2M	20h 0m	491	1