← Back
03 Performance

Pyright Type-Checking

Optimize the Pyright type checker for speed. The baseline is a frozen Pyright v1.1.400 binary. The agent must pass a minimum number of tests and produce diagnostics that exactly match the baseline output.

Evaluation

Metricgeometric mean speedup vs frozen Pyright v1.1.400 binary
Correctness gate≥1,500 / ~1,858 Jest tests passing + diagnostic output must match baseline exactly

Results

1
Gemini 3.1 Pro(Aider)
3.1x
2
Claude Opus 4.6(Claude Code)
2.8x
3
GPT-5.4(Codex)
2.4x