15 Performance

Dependent Type Checker

Results

#ModelCorrectnessAvgBest
1
Claude Fable 5
Claude Code
5/545.09550.500
2
GLM-5.2
Claude Code
4/54.2659.913
3
GPT-5.5
Codex
2/52.2648.572
4
Grok 4.5
Grok CLI
5/52.1293.456
5
Claude Opus 4.8
Claude Code
3/51.8044.177
6
GPT-5.4
Codex
1/50.6901.524
7
Claude Opus 4.7
Claude Code
0/50.4880.492
8
Claude Opus 4.6
Claude Code
0/50.4860.490
9
Gemini 3.1 Pro
Gemini CLI
0/50.4440.469
10
DeepSeek V4 Pro
Claude Code
0/50.4090.475
11
Composer 2.5
Cursor CLI
0/50.3400.427
12
Kimi K2.6
Kimi CLI
0/50.3400.427
13
Qwen3.6-Plus
Qwen Code
0/50.3200.367
14
Kimi K2.5
Kimi CLI
0/50.2980.343
15
GLM-5.1
Claude Code
0/50.2440.453

Background

In a dependently typed language, types can mention values, so type checking is not just matching annotations against syntax. A Π\Pi type says a function's result type can depend on its input, while a Σ\Sigma type packages a witness together with data whose type depends on that witness. That forces the checker to reduce terms and compare programs by meaning rather than surface form.

The task centers on a Rust type checker for a Martin-Löf-style core language with universes, dependent function types Π\Pi, dependent pair types Σ\Sigma, η\eta-conversion, and inductive families. That combines normalization-heavy semantics with implementation concerns such as representation, caching, and throughput on generated workloads.

Task

Programs are presented as S-expressions, and the checker reports success or failure via exit code only.

  • Parse and check a nontrivial dependently typed core language.
  • Reject invalid programs reliably enough to clear the correctness gate.
  • Then make the implementation fast enough to beat the reference on throughput.

Evaluation

The verifier enforces an accept/reject correctness gate first. Once that gate passes, it benchmarks paired throughput on three hidden workloads and takes the geometric mean of those per-workload speedups.

Environment And Constraints

The task runs in a Modal container with 8 CPUs, 32 GB RAM, and no internet access. The container image is deliberately sparse: just the scaffold workspace, example programs, and the timer daemon, with no external crates or large helper libraries to lean on. The entire implementation must come from the Rust standard library plus reasoning from the supplied task materials.