SWE-Bench Pro

Long-horizon software engineering tasks in public repositories. Scale AI.

Published by

Scale AI SEAL

Category

Coding

Index weight

1.0

Models

23

Data as of

9 Sept 2026

Scale AI SEAL Leaderboards.

1. 1Muse Spark 1.161.5%
2. 2GPT-5.459.1%
3. 3Muse Spark55.0%
4. 4Claude Opus 4.651.9%
5. 5Gemini 3.1 Pro46.1%
6. 6Claude Opus 4.545.9%
7. 7Claude Sonnet 4.543.6%
8. 8Gemini 3 Pro43.3%
9. 9Claude Sonnet 442.7%
10. 10GPT-541.8%
11. 11GPT-5.2 Codex41.0%
12. 12Claude Haiku 4.539.5%
13. 13Qwen3 Coder 480B38.7%
14. 14Minimax 2.136.8%
15. 15Gemini 3 Flash34.6%

SearchAll providersAlibabaAnthropicGoogleMetaMiniMaxMoonshot AIOpenAIZhipu AIOpen weights23 of 23

#	Model	Score ↓	Run ↕	Index ↕	Published ↕
1	Muse Spark 1.1Meta	61.5%	Muse Spark 1.1*	64.9	–
2	GPT-5.4OpenAI	59.1%	–	61.3	–
3	Muse SparkMeta	55.0%	–	65.4	–
4	Claude Opus 4.6Anthropic	51.9%	–	62.3	–
5	Gemini 3.1 ProGoogle	46.1%	–	–	–
6	Claude Opus 4.5Anthropic	45.9%	–	58.0	–
7	Claude Sonnet 4.5Anthropic	43.6%	–	52.3	–
8	Gemini 3 ProGoogle	43.3%	–	61.6	–
9	Claude Sonnet 4Anthropic	42.7%	–	49.5	–
10	GPT-5OpenAI	41.8%	–	54.5	–
11	GPT-5.2 CodexOpenAI	41.0%	–	60.7	–
12	Claude Haiku 4.5Anthropic	39.5%	–	47.9	–
13	Qwen3 Coder 480BAlibabaopen	38.7%	–	48.5	–
14	Minimax 2.1MiniMax	36.8%	minimax-2.1	–	–
15	Gemini 3 FlashGoogle	34.6%	–	56.6	–
16	GPT-5.2OpenAI	29.9%	gpt-5.2	61.5	–
17	Kimi K2Moonshot AIopen	27.7%	–	48.8	–
18	Qwen3 235B A22BAlibabaopen	21.4%	–	48.7	–
19	gpt-oss-120bOpenAIopen	16.2%	–	47.6	–
20	Gemma 3 27BGoogleopen	11.4%	–	39.7	–
21	Llama 3.1 405BMetaopen	11.2%	–	42.6	–
22	GLM-4.6Zhipu AIopen	9.7%	glm-4.6	47.6	–
23	Llama 4 Maverick 17BMeta	5.2%	–	–	–