Open beta Zabio is in open beta: anyone can sign up and use it in production. Models, prices and features can still change while we learn from the first customers. Tell us what breaks at hello@zabio.ai.
Benchmarks / Models / DeepSeek / deepseek-v4-pro Model from DeepSeek
deepseek-v4-pro benchmarks On Math (hardest), thinking high gets 98.2% right (range 92.3 to 99.6) against 44.2% with thinking off (range 33.8 to 55.3), for 7.5 times the cost.
On SQL and data queries (hardest), thinking high gets 98.2% right (range 92.3 to 99.6) against 84.8% with thinking off (range 75.3 to 91.1), for 13.7 times the cost.
Model id deepseek-v4-pro
Price per 1M tokens $1.32 in, $3.96 out
Context window 1,000,000
Processing location DeepSeek, China
Zabio default Thinking off
Tested 25 September 2026 Math Right answers by difficulty and thinking setting Each dot is one thinking setting, and its bar is the 90% range.
Thinking off Thinking low Thinking high0 25 50 75 100 Easy Medium Hard Hardest Easy, Thinking off: 96.0% (83.9 to 99.1) Easy, Thinking low: 100.0% (90.2 to 100.0) Easy, Thinking high: 100.0% (90.2 to 100.0) Medium, Thinking off: 94.5% (87.1 to 97.8) Medium, Thinking low: 98.2% (92.3 to 99.6) Medium, Thinking high: 98.2% (92.3 to 99.6) Hard, Thinking off: 86.4% (77.0 to 92.4) Hard, Thinking low: 98.1% (92.0 to 99.6) Hard, Thinking high: 98.2% (92.3 to 99.6) Hardest, Thinking off: 44.2% (33.8 to 55.3) Hardest, Thinking low: 89.1% (80.3 to 94.3) Hardest, Thinking high: 98.2% (92.3 to 99.6) Cost per 1,000 tasks At list price, by difficulty. Thinking costs more because the model writes more tokens before it answers.
$1 $10 Easy Easy, Off: $1.04 Off $1.04 Easy, Low: $0.81 Low $0.81 Easy, High: $1.28 High $1.28 Medium Medium, Off: $1.60 Off $1.60 Medium, Low: $1.57 Low $1.57 Medium, High: $3.44 High $3.44 Hard Hard, Off: $3.94 Off $3.94 Hard, Low: $5.98 Low $5.98 Hard, High: $15.63 High $15.63 Hardest Hardest, Off: $6.52 Off $6.52 Hardest, Low: $25.10 Low $25.10 Hardest, High: $48.68 High $48.68 The numbers behind this chart are in the table below it.
SQL and data queries Right answers by difficulty and thinking setting Each dot is one thinking setting, and its bar is the 90% range.
Thinking off Thinking low Thinking high50 60 70 80 90 100 Easy Medium Hard Hardest Easy, Thinking off: 100.0% (90.2 to 100.0) Easy, Thinking low: 100.0% (90.2 to 100.0) Easy, Thinking high: 100.0% (90.2 to 100.0) Medium, Thinking off: 90.9% (82.5 to 95.5) Medium, Thinking low: 98.2% (92.3 to 99.6) Medium, Thinking high: 98.2% (92.3 to 99.6) Hard, Thinking off: 84.8% (75.3 to 91.1) Hard, Thinking low: 97.6% (91.3 to 99.4) Hard, Thinking high: 100.0% (95.3 to 100.0) Hardest, Thinking off: 84.8% (75.3 to 91.1) Hardest, Thinking low: 94.5% (87.1 to 97.8) Hardest, Thinking high: 98.2% (92.3 to 99.6) Cost per 1,000 tasks At list price, by difficulty. Thinking costs more because the model writes more tokens before it answers.
$1 $5 $20 Easy Easy, Off: $1.03 Off $1.03 Easy, Low: $1.76 Low $1.76 Easy, High: $2.35 High $2.35 Medium Medium, Off: $1.76 Off $1.76 Medium, Low: $8.92 Low $8.92 Medium, High: $12.51 High $12.51 Hard Hard, Off: $2.24 Off $2.24 Hard, Low: $13.45 Low $13.45 Hard, High: $22.80 High $22.80 Hardest Hardest, Off: $3.06 Off $3.06 Hardest, Low: $26.46 Low $26.46 Hardest, High: $41.95 High $41.95 The numbers behind this chart are in the table below it.
Where it is a top choice Jobs where deepseek-v4-pro is the best, balanced or cheap choice among the models tested so far.
Not the best, balanced or cheap choice for any job tested so far.
Stability 12.7%
Different result on a second run. Share of sampled questions that got a different result when asked again.
Compare with Public results for this model One bar per kind of job, from the sources named below. Hover a bar to see the benchmarks behind it, with the score each one reports and whether it was run at settings like ours.
Aggregated from public results Read on 16 November 2026
Zabio has tested this kind of job too. Those results are further up the page and are counted separately.
deepseek-v4-pro Writing, Medium: 67.2 +/- 3.9
Evidence, in standard errors: 0.6
LiveBench Language: 78.1 (Same settings as ours)
LiveBench paraphrase: 56.2 (Same settings as ours)
LMArena text style control, instruction following: 1,453.2 (Same settings as ours) Writing, Medium 67.2 Creative writing, Medium: 67.8 +/- 6.6
Evidence, in standard errors: 1.2
LiveBench story generation: 67.8 (Same settings as ours)
LMArena text style control, creative writing: 1,446.4 (Same settings as ours) Creative writing, Medium 67.8 Summarizing, Medium: 69.5 +/- 6.5
Evidence, in standard errors: -0.0
LiveBench summarize: 69.5 (Same settings as ours) Summarizing, Medium 69.5 Pulling out data, Medium: Rating 1,453.2
Evidence, in standard errors: 0.7
LMArena text style control, instruction following: 1,453.2 (Same settings as ours) Pulling out data, Medium Rating 1,453.2 Sorting into categories, Medium: 62.4 +/- 3.4
Evidence, in standard errors: -0.9
LiveBench IF: 62.4 (Same settings as ours) Sorting into categories, Medium 62.4 Conversation, Medium: 47.0 +/- 1.6
Evidence, in standard errors: 1.5
SimpleQA Verified: 47.0 (Different settings)
LMArena text style control, multi turn: 1,475.1 (Same settings as ours) Conversation, Medium 47.0 Analysis and reasoning, Hard: 73.2 +/- 3.2
Evidence, in standard errors: -0.4
GPQA diamond: 73.2 (Same settings as ours)
LMArena text style control, hard prompts: 1,480.2 (Same settings as ours) Analysis and reasoning, Hard 73.2 Analysis and reasoning, Hardest: Rating 1,479.5
Evidence, in standard errors: 0.2
LMArena text style control, expert: 1,479.5 (Same settings as ours) Analysis and reasoning, Hardest Rating 1,479.5 Math, Medium: Rating 1,443.1
Evidence, in standard errors: 0.1
LMArena text style control, math: 1,443.1 (Same settings as ours) Math, Medium Rating 1,443.1 Math, Hard: 90.7 +/- 1.5
Evidence, in standard errors: 1.6
LiveBench Mathematics: 90.7 (Same settings as ours) Math, Hard 90.7 Math, Hardest: 46.1 +/- 4.6
Evidence, in standard errors: -1.5
FrontierMath-Tiers-1-3-v2-Private: 45.3 (Different settings)
OTIS Mock AIME 2024-2025: 46.7 (Same settings as ours) Math, Hardest 46.1 Writing code, Hard: Rating 1,500.5
Evidence, in standard errors: -0.3
LMArena text style control, coding: 1,500.5 (Same settings as ours)
LMArena webdev, overall: 1,448.5 (Same settings as ours) Writing code, Hard Rating 1,500.5 Fixing code, Hard: 77.6 +/- 1.9
Evidence, in standard errors: 2.5
SWE-Bench verified: 77.6 (Different settings) Fixing code, Hard 77.6 Questions about a document, Medium: Rating 1,471.8
Evidence, in standard errors: 0.8
LMArena text style control, longer query: 1,471.8 (Same settings as ours) Questions about a document, Medium Rating 1,471.8 The benchmarks behind each bar Price against score in each kind of job (14) Price per 1,000 tasks, USD. Score, %. Greyed out: a cheaper model already scores as well. Ringed: the best, balanced and cheap choice.
Writing, Medium 50 60 70 80 90 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 70.4 +/- 3.8, $0.12
deepseek-v4-flash@deepinfra: 66.3 +/- 3.9, $0.14
deepseek-v4-flash@parasail: 66.3 +/- 3.9, $0.22
deepseek-v4-flash-0731: 70.4 +/- 3.8, $0.22
deepseek-v4-flash-0731@parasail: 70.4 +/- 3.8, $0.22
deepseek-flash@deepinfra: 76.0 +/- 3.6, $0.28
gpt-6-luna: 61.8 +/- 4.0, $0.28
claude-haiku-5-5: 63.0 +/- 4.0, $0.28
glm-5.3-flash: 62.7 +/- 3.9, $0.32
glm-5.3-flash@deepinfra: 62.7 +/- 3.9, $0.32
deepseek-v4-flash: 66.3 +/- 3.9, $0.32
deepseek-v4-flash-vision-exp: 76.6 +/- 3.5, $0.43
gpt-5.6-luna: 60.2 +/- 4.0, $0.64
gpt-5.4-nano: 64.6 +/- 3.9, $0.66
minimax-m3@deepinfra: 66.4 +/- 3.9, $0.66
deepseek-flash: 76.0 +/- 3.6, $0.72
deepseek-flash@alibaba-intl: 76.0 +/- 3.6, $0.72
deepseek-flash@parasail: 76.0 +/- 3.6, $0.72
deepseek-flash@together: 76.0 +/- 3.6, $0.72
minimax-m3: 66.4 +/- 3.9, $0.72
minimax-m3@parasail: 66.4 +/- 3.9, $0.72
qwen3.8-27b@deepinfra: 74.6 +/- 3.6, $0.87
deepseek-v4-flash-0731@alibaba-intl: 70.4 +/- 3.8, $0.88
qwen3.8-27b@parasail: 74.6 +/- 3.6, $1.07
deepseek-flash-fast@parasail: 76.0 +/- 3.6, $1.08
minimax-m3-fast@parasail: 66.4 +/- 3.9, $1.08
glm-5.2@deepinfra: 69.3 +/- 3.9, $1.17
gemini-3.5-flash-lite: 68.4 +/- 3.8, $1.24
glm-5.3@deepinfra: 72.3 +/- 3.8, $1.45
qwen3.6-27b@deepinfra: 56.4 +/- 4.1, $1.54
qwen3.8-27b: 74.6 +/- 3.6, $1.60
qwen3.6-plus: 67.8 +/- 3.9, $1.60
qwen3.6-27b: 56.4 +/- 4.1, $1.92
kimi-k2.6: 68.4 +/- 3.9, $2.00
deepseek-v4-pro-0813@deepinfra: 75.4 +/- 3.6, $2.08
deepseek-v4-pro@deepinfra: 67.2 +/- 3.9, $2.08
gemini-3.8-flash: 83.9 +/- 3.1, $2.10
gemini-3.7-flash: 81.8 +/- 3.3, $2.10
gemini-3.6-flash: 79.1 +/- 3.4, $2.10
gemini-3.5-flash: 80.1 +/- 3.4, $2.10
kimi-k2.7-code: 63.2 +/- 3.9, $2.36
gpt-5.4-mini: 64.9 +/- 3.9, $2.40
inkling: 72.2 +/- 3.7, $2.42
deepseek-v4-pro-0813: 75.4 +/- 3.6, $2.64
deepseek-v4-pro: 67.2 +/- 3.9, $2.64
deepseek-v4-pro@parasail: 67.2 +/- 3.9, $2.64
deepseek-v4-pro@together: 67.2 +/- 3.9, $2.64
glm-5.3: 72.3 +/- 3.8, $2.88
glm-5.3@alibaba-intl: 72.3 +/- 3.8, $2.88
glm-5.3@parasail: 72.3 +/- 3.8, $2.88
glm-5.2: 69.3 +/- 3.9, $2.88
glm-5.2@alibaba-intl: 69.3 +/- 3.9, $2.88
glm-5.2@alibaba-us: 69.3 +/- 3.9, $2.88
deepseek-v4-pro-fast@parasail: 67.2 +/- 3.9, $3.96
qwen3.8-max-0902: 76.5 +/- 3.5, $4.00
glm-5.3-fast@parasail: 72.3 +/- 3.8, $4.32
grok-4.6: 77.7 +/- 3.5, $4.40
grok-4.7: 79.1 +/- 3.4, $4.40
qwen3.7-max: 75.9 +/- 3.6, $5.00
qwen3.7-max@alibaba-us: 75.9 +/- 3.6, $5.00
gpt-6.1-sol: 81.4 +/- 3.4, $5.60
gpt-6-sol: 77.7 +/- 3.6, $5.60
claude-sonnet-5-5: 63.1 +/- 3.9, $5.60
claude-sonnet-5: 68.1 +/- 3.9, $5.60
gemini-3.1-pro-preview: 82.6 +/- 3.2, $6.40
gpt-5.6-terra: 70.0 +/- 3.8, $6.40
gpt-5.2: 72.0 +/- 3.8, $7.00
kimi-k3@deepinfra: 80.0 +/- 3.4, $7.98
gpt-5.4: 73.0 +/- 3.7, $8.00
kimi-k3: 80.0 +/- 3.4, $8.40
kimi-k3@moonshot: 80.0 +/- 3.4, $8.40
claude-sonnet-4-6: 67.8 +/- 3.9, $8.40
gpt-5.6-sol: 79.6 +/- 3.5, $11.20
claude-opus-5-5: 75.3 +/- 3.7, $11.20
claude-opus-5: 77.2 +/- 3.6, $14.00
claude-opus-4-6: 73.1 +/- 3.7, $14.00
claude-opus-4-7: 71.8 +/- 3.8, $14.00
claude-opus-4-8: 73.8 +/- 3.7, $14.00
claude-opus-4-5: 73.5 +/- 3.7, $14.00
gpt-5.5: 79.6 +/- 3.5, $16.00
claude-fable-5: 84.2 +/- 3.2, $28.00
claude-fable-5-1: 80.5 +/- 3.4, $28.00
gpt-6-astra: 83.8 +/- 3.2, $28.00
gemini-3.8-flash Cheap deepseek-v4-pro claude-fable-5 Best, Balanced Creative writing, Medium 0 25 50 75 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 70.5 +/- 6.5, $0.12
deepseek-v4-flash@deepinfra: 67.3 +/- 6.6, $0.14
deepseek-v4-flash@parasail: 67.3 +/- 6.6, $0.22
deepseek-v4-flash-0731: 70.5 +/- 6.5, $0.22
deepseek-v4-flash-0731@parasail: 70.5 +/- 6.5, $0.22
deepseek-flash@deepinfra: 72.7 +/- 6.3, $0.28
gpt-6-luna: 63.3 +/- 6.8, $0.28
claude-haiku-5-5: 73.6 +/- 6.2, $0.28
glm-5.3-flash: 59.9 +/- 6.9, $0.32
glm-5.3-flash@deepinfra: 59.9 +/- 6.9, $0.32
deepseek-v4-flash: 67.3 +/- 6.6, $0.32
deepseek-v4-flash-vision-exp: 73.8 +/- 6.2, $0.43
gpt-5.6-luna: 69.1 +/- 6.5, $0.64
gpt-5.4-nano: 67.7 +/- 6.6, $0.66
minimax-m3@deepinfra: 63.6 +/- 6.8, $0.66
deepseek-flash: 72.7 +/- 6.3, $0.72
deepseek-flash@alibaba-intl: 72.7 +/- 6.3, $0.72
deepseek-flash@parasail: 72.7 +/- 6.3, $0.72
deepseek-flash@together: 72.7 +/- 6.3, $0.72
minimax-m3: 63.6 +/- 6.8, $0.72
minimax-m3@parasail: 63.6 +/- 6.8, $0.72
qwen3.8-27b@deepinfra: 73.1 +/- 6.3, $0.87
deepseek-v4-flash-0731@alibaba-intl: 70.5 +/- 6.5, $0.88
qwen3.8-27b@parasail: 73.1 +/- 6.3, $1.07
deepseek-flash-fast@parasail: 72.7 +/- 6.3, $1.08
minimax-m3-fast@parasail: 63.6 +/- 6.8, $1.08
glm-5.2@deepinfra: 62.9 +/- 6.8, $1.17
gemini-3.5-flash-lite: 67.5 +/- 6.6, $1.24
glm-5.3@deepinfra: 72.1 +/- 6.3, $1.45
qwen3.6-27b@deepinfra: 60.1 +/- 6.9, $1.54
qwen3.6-plus: 56.9 +/- 7.0, $1.60
qwen3.8-27b: 73.1 +/- 6.3, $1.60
qwen3.6-27b: 60.1 +/- 6.9, $1.92
kimi-k2.6: 66.9 +/- 6.7, $2.00
deepseek-v4-pro-0813@deepinfra: 74.9 +/- 6.1, $2.08
deepseek-v4-pro@deepinfra: 67.8 +/- 6.6, $2.08
gemini-3.8-flash: 82.8 +/- 5.3, $2.10
gemini-3.7-flash: 82.3 +/- 5.4, $2.10
gemini-3.6-flash: 75.6 +/- 6.1, $2.10
gemini-3.5-flash: 75.2 +/- 6.1, $2.10
kimi-k2.7-code: 68.6 +/- 6.6, $2.36
gpt-5.4-mini: 66.6 +/- 6.7, $2.40
inkling: 74.2 +/- 6.2, $2.42
deepseek-v4-pro-0813: 74.9 +/- 6.1, $2.64
deepseek-v4-pro: 67.8 +/- 6.6, $2.64
deepseek-v4-pro@parasail: 67.8 +/- 6.6, $2.64
deepseek-v4-pro@together: 67.8 +/- 6.6, $2.64
glm-5.3: 72.1 +/- 6.3, $2.88
glm-5.3@alibaba-intl: 72.1 +/- 6.3, $2.88
glm-5.3@parasail: 72.1 +/- 6.3, $2.88
glm-5.2: 62.9 +/- 6.8, $2.88
glm-5.2@alibaba-intl: 62.9 +/- 6.8, $2.88
glm-5.2@alibaba-us: 62.9 +/- 6.8, $2.88
deepseek-v4-pro-fast@parasail: 67.8 +/- 6.6, $3.96
qwen3.8-max-0902: 76.5 +/- 6.0, $4.00
glm-5.3-fast@parasail: 72.1 +/- 6.3, $4.32
grok-4.6: 78.3 +/- 5.8, $4.40
grok-4.7: 74.9 +/- 6.1, $4.40
qwen3.7-max: 76.3 +/- 6.0, $5.00
qwen3.7-max@alibaba-us: 76.3 +/- 6.0, $5.00
claude-sonnet-5-5: 69.9 +/- 6.5, $5.60
gpt-6.1-sol: 70.8 +/- 6.4, $5.60
gpt-6-sol: 66.9 +/- 6.7, $5.60
claude-sonnet-5: 62.3 +/- 6.9, $5.60
gemini-3.1-pro-preview: 77.3 +/- 5.9, $6.40
gpt-5.6-terra: 66.8 +/- 6.7, $6.40
gpt-5.2: 65.7 +/- 6.7, $7.00
kimi-k3@deepinfra: 75.4 +/- 6.1, $7.98
gpt-5.4: 79.0 +/- 5.8, $8.00
kimi-k3: 75.4 +/- 6.1, $8.40
kimi-k3@moonshot: 75.4 +/- 6.1, $8.40
claude-sonnet-4-6: 68.9 +/- 6.6, $8.40
claude-opus-5-5: 70.9 +/- 6.4, $11.20
gpt-5.6-sol: 75.9 +/- 6.1, $11.20
claude-opus-4-8: 80.6 +/- 5.6, $14.00
claude-opus-4-7: 69.4 +/- 6.5, $14.00
claude-opus-4-6: 66.1 +/- 6.7, $14.00
claude-opus-4-5: 65.9 +/- 6.7, $14.00
claude-opus-5: 61.3 +/- 6.9, $14.00
gpt-5.5: 72.6 +/- 6.3, $16.00
claude-fable-5-1: 78.1 +/- 5.9, $28.00
claude-fable-5: 71.7 +/- 6.4, $28.00
gpt-6-astra: 74.0 +/- 6.2, $28.00
deepseek-v4-flash-0731@deepinfra Cheap gemini-3.8-flash Best, Balanced deepseek-v4-pro Summarizing, Medium 0 25 50 75 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 71.5 +/- 6.4, $0.18
deepseek-v4-flash@deepinfra: 68.9 +/- 6.6, $0.25
claude-haiku-5-5: 67.0 +/- 6.7, $0.34
gpt-6-luna: 55.3 +/- 7.0, $0.34
deepseek-v4-flash-0731: 71.5 +/- 6.4, $0.39
deepseek-v4-flash-0731@parasail: 71.5 +/- 6.4, $0.39
deepseek-v4-flash@parasail: 68.9 +/- 6.6, $0.39
deepseek-flash@deepinfra: 69.7 +/- 6.5, $0.42
glm-5.3-flash: 56.5 +/- 7.0, $0.46
glm-5.3-flash@deepinfra: 56.5 +/- 7.0, $0.46
deepseek-v4-flash: 68.9 +/- 6.6, $0.56
deepseek-v4-flash-vision-exp: 72.1 +/- 6.3, $0.65
gpt-5.6-luna: 64.9 +/- 6.8, $0.72
gpt-5.4-nano: 71.8 +/- 6.4, $0.73
qwen3.8-27b@deepinfra: 75.9 +/- 6.1, $0.74
minimax-m3@deepinfra: 53.9 +/- 7.1, $0.89
deepseek-flash: 69.7 +/- 6.5, $0.96
deepseek-flash@alibaba-intl: 69.7 +/- 6.5, $0.96
deepseek-flash@parasail: 69.7 +/- 6.5, $0.96
deepseek-flash@together: 69.7 +/- 6.5, $0.96
minimax-m3: 53.9 +/- 7.1, $0.96
minimax-m3@parasail: 53.9 +/- 7.1, $0.96
qwen3.8-27b@parasail: 75.9 +/- 6.1, $1.02
gemini-3.5-flash-lite: 68.5 +/- 6.6, $1.22
deepseek-v4-flash-0731@alibaba-intl: 71.5 +/- 6.4, $1.32
qwen3.6-27b@deepinfra: 52.6 +/- 7.1, $1.41
deepseek-flash-fast@parasail: 69.7 +/- 6.5, $1.44
minimax-m3-fast@parasail: 53.9 +/- 7.1, $1.44
glm-5.2@deepinfra: 68.4 +/- 6.6, $1.71
qwen3.8-27b: 75.9 +/- 6.1, $1.80
qwen3.6-plus: 64.3 +/- 6.8, $1.80
glm-5.3@deepinfra: 74.6 +/- 6.2, $1.85
qwen3.6-27b: 52.6 +/- 7.1, $2.16
kimi-k2.6: 67.5 +/- 6.6, $2.50
gemini-3.8-flash: 87.5 +/- 4.7, $2.55
gemini-3.7-flash: 84.8 +/- 5.1, $2.55
gemini-3.5-flash: 82.6 +/- 5.4, $2.55
gemini-3.6-flash: 75.9 +/- 6.1, $2.55
gpt-5.4-mini: 59.6 +/- 6.9, $2.70
kimi-k2.7-code: 58.3 +/- 7.0, $3.08
inkling: 74.9 +/- 6.1, $3.21
deepseek-v4-pro@deepinfra: 69.5 +/- 6.5, $3.64
deepseek-v4-pro-0813@deepinfra: 65.7 +/- 6.7, $3.64
deepseek-v4-pro: 69.5 +/- 6.5, $3.96
deepseek-v4-pro@parasail: 69.5 +/- 6.5, $3.96
deepseek-v4-pro@together: 69.5 +/- 6.5, $3.96
deepseek-v4-pro-0813: 65.7 +/- 6.7, $3.96
glm-5.3: 74.6 +/- 6.2, $4.24
glm-5.3@alibaba-intl: 74.6 +/- 6.2, $4.24
glm-5.3@parasail: 74.6 +/- 6.2, $4.24
glm-5.2: 68.4 +/- 6.6, $4.24
glm-5.2@alibaba-intl: 68.4 +/- 6.6, $4.24
glm-5.2@alibaba-us: 68.4 +/- 6.6, $4.24
deepseek-v4-pro-fast@parasail: 69.5 +/- 6.5, $5.94
qwen3.8-max-0902: 79.2 +/- 5.8, $6.00
glm-5.3-fast@parasail: 74.6 +/- 6.2, $6.36
grok-4.7: 78.2 +/- 5.8, $6.60
grok-4.6: 68.1 +/- 6.6, $6.60
gpt-6.1-sol: 81.3 +/- 5.5, $6.80
claude-sonnet-5: 71.2 +/- 6.4, $6.80
gpt-6-sol: 68.0 +/- 6.6, $6.80
claude-sonnet-5-5: 58.2 +/- 7.0, $6.80
gpt-5.2: 63.2 +/- 6.8, $7.00
gemini-3.1-pro-preview: 88.2 +/- 4.6, $7.20
gpt-5.6-terra: 73.2 +/- 6.3, $7.20
qwen3.7-max: 83.8 +/- 5.2, $7.50
qwen3.7-max@alibaba-us: 83.8 +/- 5.2, $7.50
gpt-5.4: 68.4 +/- 6.6, $9.00
kimi-k3@deepinfra: 69.4 +/- 6.5, $9.69
kimi-k3: 69.4 +/- 6.5, $10.20
kimi-k3@moonshot: 69.4 +/- 6.5, $10.20
claude-sonnet-4-6: 61.7 +/- 6.9, $10.20
gpt-5.6-sol: 69.4 +/- 6.5, $13.60
claude-opus-5-5: 69.1 +/- 6.5, $13.60
claude-opus-4-8: 77.7 +/- 5.9, $17.00
claude-opus-5: 66.4 +/- 6.7, $17.00
claude-opus-4-7: 66.3 +/- 6.7, $17.00
claude-opus-4-6: 65.2 +/- 6.7, $17.00
claude-opus-4-5: 63.7 +/- 6.8, $17.00
gpt-5.5: 72.3 +/- 6.3, $18.00
claude-fable-5: 81.7 +/- 5.5, $34.00
gpt-6-astra: 79.6 +/- 5.7, $34.00
claude-fable-5-1: 72.2 +/- 6.3, $34.00
deepseek-v4-flash-vision-exp Cheap qwen3.8-27b@deepinfra Balanced gemini-3.8-flash Best deepseek-v4-pro Pulling out data, Medium 50 60 70 80 90 100 $0.1 $1 $10
deepseek-v4.1-flash-1017@deepinfra: 89.8 +/- 1.7, $0.14
gpt-6-luna: 86.7 +/- 1.9, $0.18
claude-haiku-5-5: 75.6 +/- 2.3, $0.18
glm-5.3-flash: 86.3 +/- 2.2, $0.22
glm-5.3-flash@deepinfra: 86.3 +/- 2.2, $0.22
gemma-4-31b-it: 76.7 +/- 2.1, $0.24
minimax-m3.1-flash: 84.7 +/- 2.1, $0.24
minimax-m3.1-flash@parasail: 84.7 +/- 2.1, $0.24
mimo-v2.6-pro: 85.2 +/- 2.1, $0.47
deepseek-v4.1-flash-1017: 89.8 +/- 1.7, $0.48
deepseek-v4.1-flash-1017@parasail: 89.8 +/- 1.7, $0.48
qwen3.8-27b@deepinfra: 86.4 +/- 2.2, $0.50
qwen3.8-27b@parasail: 86.4 +/- 2.2, $0.63
mistral-large-4-0: 88.3 +/- 1.8, $0.98
qwen3.8-27b: 86.4 +/- 2.2, $1.00
gemini-3.8-flash: 75.1 +/- 2.4, $1.35
inkling: 86.4 +/- 1.9, $1.61
gpt-6.1-terra: 91.2 +/- 1.6, $1.80
gemini-4-hydrogen: 90.6 +/- 1.8, $2.70
qwen3.8-max-0902: 88.2 +/- 1.9, $2.80
grok-4.6: 86.7 +/- 2.0, $3.08
grok-4.7: 88.5 +/- 1.8, $3.08
kimi-k3.1-pro@deepinfra: 88.6 +/- 1.9, $3.42
gemini-4-carbon: 90.3 +/- 2.2, $3.60
gemini-4-argon: 89.7 +/- 2.2, $3.60
gpt-6.1-sol: 91.4 +/- 1.6, $3.60
kimi-k3.1-pro: 88.6 +/- 1.9, $3.60
kimi-k3.1-pro@together: 88.6 +/- 1.9, $3.60
claude-sonnet-5-5: 78.8 +/- 2.2, $3.60
gpt-5.6-terra: 89.7 +/- 1.7, $4.00
kimi-k3@deepinfra: 88.4 +/- 1.9, $5.13
kimi-k3: 88.4 +/- 1.9, $5.40
kimi-k3@moonshot: 88.4 +/- 1.9, $5.40
claude-opus-5-5: 89.2 +/- 1.9, $7.20
claude-fable-5-2: 89.8 +/- 1.9, $14.40
claude-fable-5-1: 87.2 +/- 2.3, $18.00
gpt-6-astra: 91.5 +/- 1.6, $18.00
deepseek-v4.1-flash-1017@deepinfra Cheap glm-5.3-flash Balanced gemini-4-hydrogen Best Sorting into categories, Medium 0 25 50 75 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 65.5 +/- 3.4, $0.08
deepseek-v4-flash@deepinfra: 63.1 +/- 3.4, $0.11
deepseek-v4-flash-0731: 65.5 +/- 3.4, $0.17
deepseek-v4-flash-0731@parasail: 65.5 +/- 3.4, $0.17
deepseek-v4-flash@parasail: 63.1 +/- 3.4, $0.17
claude-haiku-5-5: 66.5 +/- 3.3, $0.18
gpt-6-luna: 55.9 +/- 3.5, $0.18
deepseek-flash@deepinfra: 70.0 +/- 3.2, $0.20
glm-5.3-flash: 52.8 +/- 3.5, $0.22
glm-5.3-flash@deepinfra: 52.8 +/- 3.5, $0.22
deepseek-v4-flash: 63.1 +/- 3.4, $0.24
deepseek-v4-flash-vision-exp: 71.0 +/- 3.2, $0.30
gpt-5.6-luna: 60.1 +/- 3.5, $0.40
gpt-5.4-nano: 67.2 +/- 3.3, $0.41
minimax-m3@deepinfra: 57.5 +/- 3.5, $0.44
deepseek-flash: 70.0 +/- 3.2, $0.48
deepseek-flash@alibaba-intl: 70.0 +/- 3.2, $0.48
deepseek-flash@parasail: 70.0 +/- 3.2, $0.48
deepseek-flash@together: 70.0 +/- 3.2, $0.48
minimax-m3: 57.5 +/- 3.5, $0.48
minimax-m3@parasail: 57.5 +/- 3.5, $0.48
qwen3.8-27b@deepinfra: 72.7 +/- 3.2, $0.50
deepseek-v4-flash-0731@alibaba-intl: 65.5 +/- 3.4, $0.62
qwen3.8-27b@parasail: 72.7 +/- 3.2, $0.63
deepseek-flash-fast@parasail: 70.0 +/- 3.2, $0.72
minimax-m3-fast@parasail: 57.5 +/- 3.5, $0.72
gemini-3.5-flash-lite: 67.2 +/- 3.3, $0.74
glm-5.2@deepinfra: 62.3 +/- 3.4, $0.81
qwen3.6-27b@deepinfra: 53.2 +/- 3.5, $0.90
glm-5.3@deepinfra: 69.3 +/- 3.3, $0.95
qwen3.8-27b: 72.7 +/- 3.2, $1.00
qwen3.6-plus: 58.3 +/- 3.5, $1.00
qwen3.6-27b: 53.2 +/- 3.5, $1.20
kimi-k2.6: 64.4 +/- 3.4, $1.30
gemini-3.8-flash: 81.4 +/- 2.8, $1.35
gemini-3.7-flash: 79.9 +/- 2.8, $1.35
gemini-3.5-flash: 75.6 +/- 3.0, $1.35
gemini-3.6-flash: 75.4 +/- 3.1, $1.35
gpt-5.4-mini: 59.8 +/- 3.5, $1.50
deepseek-v4-pro-0813@deepinfra: 67.7 +/- 3.3, $1.56
deepseek-v4-pro@deepinfra: 62.4 +/- 3.4, $1.56
kimi-k2.7-code: 56.3 +/- 3.5, $1.56
inkling: 70.1 +/- 3.2, $1.61
deepseek-v4-pro-0813: 67.7 +/- 3.3, $1.85
deepseek-v4-pro: 62.4 +/- 3.4, $1.85
deepseek-v4-pro@parasail: 62.4 +/- 3.4, $1.85
deepseek-v4-pro@together: 62.4 +/- 3.4, $1.85
glm-5.3: 69.3 +/- 3.3, $2.00
glm-5.3@alibaba-intl: 69.3 +/- 3.3, $2.00
glm-5.3@parasail: 69.3 +/- 3.3, $2.00
glm-5.2: 62.3 +/- 3.4, $2.00
glm-5.2@alibaba-intl: 62.3 +/- 3.4, $2.00
glm-5.2@alibaba-us: 62.3 +/- 3.4, $2.00
deepseek-v4-pro-fast@parasail: 62.4 +/- 3.4, $2.77
qwen3.8-max-0902: 74.1 +/- 3.1, $2.80
glm-5.3-fast@parasail: 69.3 +/- 3.3, $3.00
grok-4.7: 75.3 +/- 3.1, $3.08
grok-4.6: 71.9 +/- 3.2, $3.08
qwen3.7-max: 74.0 +/- 3.1, $3.50
qwen3.7-max@alibaba-us: 74.0 +/- 3.1, $3.50
gpt-6.1-sol: 74.2 +/- 3.1, $3.60
gpt-6-sol: 68.6 +/- 3.3, $3.60
claude-sonnet-5: 63.9 +/- 3.4, $3.60
claude-sonnet-5-5: 56.8 +/- 3.5, $3.60
gemini-3.1-pro-preview: 79.1 +/- 2.9, $4.00
gpt-5.6-terra: 64.6 +/- 3.4, $4.00
gpt-5.2: 61.8 +/- 3.4, $4.20
gpt-5.4: 70.2 +/- 3.2, $5.00
kimi-k3@deepinfra: 71.4 +/- 3.2, $5.13
kimi-k3: 71.4 +/- 3.2, $5.40
kimi-k3@moonshot: 71.4 +/- 3.2, $5.40
claude-sonnet-4-6: 63.2 +/- 3.4, $5.40
gpt-5.6-sol: 71.9 +/- 3.2, $7.20
claude-opus-5-5: 65.7 +/- 3.4, $7.20
claude-opus-4-8: 72.0 +/- 3.2, $9.00
claude-opus-4-7: 66.7 +/- 3.3, $9.00
claude-opus-5: 63.8 +/- 3.4, $9.00
claude-opus-4-6: 63.3 +/- 3.4, $9.00
claude-opus-4-5: 62.6 +/- 3.4, $9.00
gpt-5.5: 70.7 +/- 3.2, $10.00
claude-fable-5: 75.8 +/- 3.0, $18.00
gpt-6-astra: 75.6 +/- 3.0, $18.00
claude-fable-5-1: 73.0 +/- 3.1, $18.00
gemini-3.8-flash Best, Balanced, Cheap deepseek-v4-pro Conversation, Medium 0 25 50 75 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 33.6 +/- 1.5, $0.17
gpt-5-nano: 11.7 +/- 1.0, $0.24
deepseek-v4.1-flash-1017@deepinfra: 42.5 +/- 1.6, $0.27
qwen3.5-flash-2026-02-23: 20.3 +/- 1.3, $0.32
gpt-4.1-nano: 6.0 +/- 0.8, $0.32
deepseek-v4-flash-0731: 33.6 +/- 1.5, $0.34
deepseek-v4-flash-0731@parasail: 33.6 +/- 1.5, $0.34
gpt-6-luna: 41.4 +/- 1.6, $0.36
claude-haiku-5-5: 23.8 +/- 1.4, $0.36
deepseek-flash@deepinfra: 42.9 +/- 1.6, $0.39
gemma-4-31b-it: 10.4 +/- 1.0, $0.48
gpt-4o-mini: 8.3 +/- 0.9, $0.48
gpt-5.6-luna: 41.0 +/- 1.6, $0.80
gpt-5.4-nano: 11.7 +/- 1.0, $0.82
deepseek-flash: 42.9 +/- 1.6, $0.96
deepseek-flash@alibaba-intl: 42.9 +/- 1.6, $0.96
deepseek-flash@parasail: 42.9 +/- 1.6, $0.96
deepseek-flash@together: 42.9 +/- 1.6, $0.96
deepseek-v4.1-flash-1017: 42.5 +/- 1.6, $0.96
deepseek-v4.1-flash-1017@parasail: 42.5 +/- 1.6, $0.96
qwen3.6-flash: 15.9 +/- 1.2, $1.00
qwen3.6-flash@alibaba-us: 15.9 +/- 1.2, $1.00
inkling-small: 19.1 +/- 1.2, $1.20
gpt-5-mini: 21.6 +/- 1.3, $1.20
deepseek-v4-flash-0731@alibaba-intl: 33.6 +/- 1.5, $1.23
gpt-4.1-mini: 12.7 +/- 1.1, $1.28
qwen3-235b-a22b-thinking-2507: 40.4 +/- 1.6, $1.29
glm-4.7: 32.2 +/- 1.5, $1.34
deepseek-flash-fast@parasail: 42.9 +/- 1.6, $1.44
glm-5.2@deepinfra: 34.2 +/- 1.5, $1.62
glm-5.3@deepinfra: 41.0 +/- 1.6, $1.90
mistral-large-4-0: 20.0 +/- 1.3, $1.96
gemini-3-flash-preview: 66.8 +/- 1.5, $2.00
qwen3.6-plus: 44.1 +/- 1.6, $2.00
kimi-k2.6: 34.9 +/- 1.5, $2.60
gemini-3.8-flash: 69.7 +/- 1.5, $2.70
gemini-3.7-flash: 69.2 +/- 1.5, $2.70
gemini-3.6-flash: 66.2 +/- 1.5, $2.70
gemini-3.5-flash: 66.2 +/- 1.5, $2.70
gpt-5.4-mini: 29.4 +/- 1.4, $3.00
deepseek-v4-pro-0813@deepinfra: 52.9 +/- 1.6, $3.12
deepseek-v4-pro@deepinfra: 47.0 +/- 1.6, $3.12
kimi-k2.7-code: 36.5 +/- 1.5, $3.12
inkling: 40.3 +/- 1.6, $3.22
o4-mini: 19.6 +/- 1.3, $3.52
o3-mini: 15.3 +/- 1.1, $3.52
gpt-6.1-terra: 57.3 +/- 1.6, $3.60
claude-haiku-4-5: 13.2 +/- 1.1, $3.60
deepseek-v4-pro-0813: 52.9 +/- 1.6, $3.70
deepseek-v4-pro: 47.0 +/- 1.6, $3.70
deepseek-v4-pro@parasail: 47.0 +/- 1.6, $3.70
deepseek-v4-pro@together: 47.0 +/- 1.6, $3.70
glm-5.3: 41.0 +/- 1.6, $4.00
glm-5.3@alibaba-intl: 41.0 +/- 1.6, $4.00
glm-5.3@parasail: 41.0 +/- 1.6, $4.00
glm-5.1: 34.0 +/- 1.5, $4.00
glm-5.2: 34.2 +/- 1.5, $4.00
glm-5.2@alibaba-intl: 34.2 +/- 1.5, $4.00
glm-5.2@alibaba-us: 34.2 +/- 1.5, $4.00
qwen3.6-max-preview: 52.0 +/- 1.6, $5.20
deepseek-v4-pro-fast@parasail: 47.0 +/- 1.6, $5.54
qwen3.8-max-0902: 47.3 +/- 1.6, $5.60
glm-5.3-fast@parasail: 41.0 +/- 1.6, $6.00
gpt-5.1: 48.0 +/- 1.6, $6.00
gpt-5: 50.1 +/- 1.6, $6.00
grok-4.6: 49.3 +/- 1.6, $6.16
grok-4.7: 56.0 +/- 1.6, $6.16
o3: 49.4 +/- 1.6, $6.40
gpt-4.1: 31.1 +/- 1.5, $6.40
qwen3.7-max: 55.8 +/- 1.6, $7.00
qwen3.7-max@alibaba-us: 55.8 +/- 1.6, $7.00
gemini-4-carbon: 77.7 +/- 1.3, $7.20
gemini-4-argon: 76.0 +/- 1.4, $7.20
gpt-6.1-sol: 73.9 +/- 1.4, $7.20
gpt-6-sol: 60.7 +/- 1.6, $7.20
claude-sonnet-5-5: 46.5 +/- 1.6, $7.20
claude-sonnet-5: 33.7 +/- 1.5, $7.20
gemini-3.1-pro-preview: 73.5 +/- 1.4, $8.00
gpt-5.6-terra: 43.2 +/- 1.6, $8.00
gpt-4o: 26.0 +/- 1.4, $8.00
gpt-5.2: 32.8 +/- 1.5, $8.40
gpt-5.4: 45.1 +/- 1.6, $10.00
kimi-k3@deepinfra: 50.6 +/- 1.6, $10.26
kimi-k3: 50.6 +/- 1.6, $10.80
kimi-k3@moonshot: 50.6 +/- 1.6, $10.80
claude-sonnet-4-6: 32.8 +/- 1.5, $10.80
claude-sonnet-4-5: 23.7 +/- 1.4, $10.80
claude-opus-5-5: 72.2 +/- 1.4, $14.40
gpt-5.6-sol: 69.7 +/- 1.5, $14.40
claude-opus-4-7: 51.7 +/- 1.6, $18.00
claude-opus-4-8: 53.0 +/- 1.6, $18.00
claude-opus-5: 59.9 +/- 1.6, $18.00
claude-opus-4-6: 47.0 +/- 1.6, $18.00
claude-opus-4-5: 45.7 +/- 1.6, $18.00
gpt-5.5: 63.0 +/- 1.5, $20.00
claude-fable-5-2: 94.0 +/- 0.8, $28.80
claude-fable-5: 70.7 +/- 1.4, $36.00
gpt-6-astra: 75.6 +/- 1.4, $36.00
claude-fable-5-1: 70.8 +/- 1.4, $36.00
o1: 41.1 +/- 1.6, $48.00
deepseek-v4-pro gemini-4-carbon Best, Balanced, Cheap Analysis and reasoning, Hard 0 25 50 75 100 $0.1 $1 $10 $100
gpt-oss-20b: 53.2 +/- 2.5, $0.19
gemma-3-4b-it: 23.2 +/- 1.9, $0.19
gpt-oss-120b@deepinfra: 75.8 +/- 2.7, $0.23
gemma-3-12b-it: 39.5 +/- 2.5, $0.24
phi-4: 56.1 +/- 2.6, $0.27
deepseek-v4-flash-0731@deepinfra: 91.0 +/- 1.8, $0.28
gemma-3-27b-it: 47.7 +/- 2.7, $0.30
qwen3.5-9b@deepinfra: 79.0 +/- 2.4, $0.34
qwen3-32b: 65.7 +/- 2.8, $0.41
qwen3.5-9b@parasail: 79.0 +/- 2.4, $0.43
gemma-4-26b-a4b-it: 73.2 +/- 3.2, $0.45
qwen3-14b: 63.8 +/- 2.9, $0.46
gpt-5-nano: 57.6 +/- 3.5, $0.46
deepseek-v4.1-flash-1017@deepinfra: 87.0 +/- 3.1, $0.46
llama-4-scout-17b-16e-instruct: 51.8 +/- 3.2, $0.47
llama-3.3-70b-instruct-turbo@deepinfra: 47.4 +/- 2.9, $0.49
deepseek-v4-flash-0731: 91.0 +/- 1.8, $0.53
deepseek-v4-flash-0731@parasail: 91.0 +/- 1.8, $0.53
qwen3.5-flash-2026-02-23: 82.3 +/- 2.7, $0.56
gpt-4.1-nano: 48.9 +/- 2.5, $0.56
qwen3.5-9b: 79.0 +/- 2.4, $0.57
gpt-6-luna: 73.6 +/- 3.7, $0.65
claude-haiku-5-5: 85.5 +/- 2.8, $0.65
deepseek-flash@deepinfra: 89.8 +/- 1.7, $0.66
qwen3-30b-a3b: 61.7 +/- 3.0, $0.69
glm-5.3-flash: 80.6 +/- 3.3, $0.75
glm-5.3-flash@deepinfra: 80.6 +/- 3.3, $0.75
gemma-4-31b-it: 61.7 +/- 3.6, $0.76
minimax-m3.1-flash: 79.8 +/- 4.7, $0.84
minimax-m3.1-flash@parasail: 79.8 +/- 4.7, $0.84
gpt-oss-120b: 75.8 +/- 2.7, $0.84
gpt-4o-mini: 37.7 +/- 2.4, $0.84
llama-3.3-70b-instruct-turbo@parasail: 47.4 +/- 2.9, $0.89
qwen3.6-35b-a3b@parasail: 83.8 +/- 2.6, $1.01
qwen3.6-35b-a3b@deepinfra: 83.8 +/- 2.6, $1.06
qwen3-30b-a3b-instruct-2507: 55.6 +/- 2.9, $1.12
qwen3.5-35b-a3b@deepinfra: 83.5 +/- 2.2, $1.18
qwen3.5-35b-a3b@parasail: 83.5 +/- 2.2, $1.20
mimo-v2.6-pro: 82.2 +/- 4.1, $1.48
gpt-5.6-luna: 63.6 +/- 3.4, $1.48
gpt-5.4-nano: 55.6 +/- 3.5, $1.53
minimax-m3@deepinfra: 90.9 +/- 2.1, $1.55
deepseek-v4.1-flash-1017: 87.0 +/- 3.1, $1.68
deepseek-v4.1-flash-1017@parasail: 87.0 +/- 3.1, $1.68
deepseek-flash: 89.8 +/- 1.7, $1.68
deepseek-flash@alibaba-intl: 89.8 +/- 1.7, $1.68
deepseek-flash@parasail: 89.8 +/- 1.7, $1.68
deepseek-flash@together: 89.8 +/- 1.7, $1.68
minimax-m3: 90.9 +/- 2.1, $1.68
minimax-m3@parasail: 90.9 +/- 2.1, $1.68
qwen3.7-plus: 87.9 +/- 2.3, $1.79
qwen3.7-plus@alibaba-us: 87.9 +/- 2.3, $1.79
qwen3.6-flash: 83.3 +/- 2.7, $1.85
qwen3.6-flash@alibaba-us: 83.3 +/- 2.7, $1.85
gemini-3.1-flash-lite: 74.2 +/- 3.1, $1.85
qwen-plus-2025-01-25: 48.1 +/- 2.7, $1.88
inkling-small: 88.5 +/- 1.9, $1.98
qwen3.8-27b@deepinfra: 70.7 +/- 5.4, $1.99
deepseek-v4-flash-0731@alibaba-intl: 91.0 +/- 1.8, $2.07
gpt-4.1-mini: 65.9 +/- 2.7, $2.24
qwen3.5-35b-a3b: 83.5 +/- 2.2, $2.30
gpt-5-mini: 71.7 +/- 3.2, $2.30
gpt-3.5-turbo: 27.2 +/- 2.0, $2.35
glm-4.7: 83.3 +/- 2.4, $2.38
qwen3.8-27b@parasail: 70.7 +/- 5.4, $2.46
deepseek-flash-fast@parasail: 89.8 +/- 1.7, $2.52
minimax-m3-fast@parasail: 90.9 +/- 2.1, $2.52
qwen3-235b-a22b-thinking-2507: 80.1 +/- 2.6, $2.53
qwen3-30b-a3b-thinking-2507: 70.1 +/- 2.8, $2.56
glm-5.2@deepinfra: 71.2 +/- 3.2, $2.75
qwen3.6-35b-a3b: 83.8 +/- 2.6, $2.78
gemini-3.5-flash-lite: 75.8 +/- 3.1, $2.85
deepseek-r1-0528: 76.3 +/- 2.4, $2.94
llama-3.3-70b-instruct-turbo: 47.4 +/- 2.9, $3.02
mistral-large-4-0: 84.5 +/- 3.0, $3.29
glm-5.3@deepinfra: 90.9 +/- 1.6, $3.38
qwen3.6-27b@deepinfra: 85.9 +/- 2.5, $3.52
qwen3.5-397b-a17b@deepinfra: 85.9 +/- 2.5, $3.60
qwen3.6-plus: 88.4 +/- 2.3, $3.70
gemini-3-flash-preview: 83.2 +/- 1.9, $3.70
qwen3.8-27b: 70.7 +/- 5.4, $3.70
qwen3.5-397b-a17b@parasail: 85.9 +/- 2.5, $4.24
qwen3.6-27b: 85.9 +/- 2.5, $4.44
qwen3.5-397b-a17b: 85.9 +/- 2.5, $4.44
kimi-k2.6: 90.8 +/- 1.7, $4.65
gemini-3.8-flash: 88.0 +/- 3.0, $4.88
gemini-3.7-flash: 94.8 +/- 1.4, $4.88
gemini-3.5-flash: 88.9 +/- 2.2, $4.88
gemini-3.6-flash: 85.9 +/- 2.5, $4.88
deepseek-v4-pro-0813@deepinfra: 91.7 +/- 1.5, $4.94
deepseek-v4-pro@deepinfra: 73.2 +/- 3.2, $4.94
kimi-k2.7-code: 87.9 +/- 2.3, $5.50
gpt-5.4-mini: 64.1 +/- 3.4, $5.55
inkling: 75.6 +/- 3.7, $5.65
o4-mini: 75.3 +/- 3.1, $6.16
o3-mini: 68.2 +/- 3.3, $6.16
deepseek-v4-pro-0813: 91.7 +/- 1.5, $6.20
deepseek-v4-pro: 73.2 +/- 3.2, $6.20
deepseek-v4-pro@parasail: 73.2 +/- 3.2, $6.20
deepseek-v4-pro@together: 73.2 +/- 3.2, $6.20
gpt-6.1-terra: 87.4 +/- 3.0, $6.50
claude-haiku-4-5: 60.5 +/- 2.8, $6.50
glm-5.3: 90.9 +/- 1.6, $6.76
glm-5.3@alibaba-intl: 90.9 +/- 1.6, $6.76
glm-5.3@parasail: 90.9 +/- 1.6, $6.76
glm-5.1: 89.9 +/- 2.2, $6.76
glm-5.2: 71.2 +/- 3.2, $6.76
glm-5.2@alibaba-intl: 71.2 +/- 3.2, $6.76
glm-5.2@alibaba-us: 71.2 +/- 3.2, $6.76
deepseek-v4-pro-fast@parasail: 73.2 +/- 3.2, $9.31
qwen3.8-max-0902: 84.4 +/- 3.3, $9.40
qwen3.6-max-preview: 87.4 +/- 2.4, $9.62
gemini-4-hydrogen: 87.7 +/- 3.8, $9.75
glm-5.3-fast@parasail: 90.9 +/- 1.6, $10.14
grok-4.6: 91.1 +/- 2.4, $10.34
grok-4.7: 88.3 +/- 2.4, $10.34
o3: 79.8 +/- 2.9, $11.20
gpt-4.1: 66.9 +/- 2.8, $11.20
gpt-5: 71.7 +/- 3.2, $11.50
gpt-5.1: 66.7 +/- 3.4, $11.50
qwen3.7-max: 90.9 +/- 2.1, $11.75
qwen3.7-max@alibaba-us: 90.9 +/- 2.1, $11.75
kimi-k3.1-pro@deepinfra: 86.6 +/- 4.2, $12.35
gemini-4-argon: 92.8 +/- 2.5, $13.00
gemini-4-carbon: 93.2 +/- 2.0, $13.00
gpt-6.1-sol: 92.7 +/- 2.3, $13.00
claude-sonnet-5-5: 91.7 +/- 2.4, $13.00
gpt-6-sol: 94.3 +/- 1.5, $13.00
kimi-k3.1-pro: 86.6 +/- 4.2, $13.00
kimi-k3.1-pro@together: 86.6 +/- 4.2, $13.00
claude-sonnet-5: 80.3 +/- 2.8, $13.00
gpt-4o: 47.9 +/- 2.6, $14.00
gemini-3.1-pro-preview: 94.1 +/- 1.7, $14.80
gpt-5.6-terra: 81.9 +/- 2.9, $14.80
gpt-5.2: 73.2 +/- 3.2, $16.10
gpt-5.4: 74.8 +/- 3.1, $18.50
kimi-k3@deepinfra: 85.1 +/- 2.9, $18.53
kimi-k3: 85.1 +/- 2.9, $19.50
kimi-k3@moonshot: 85.1 +/- 2.9, $19.50
claude-sonnet-4-6: 78.8 +/- 2.9, $19.50
claude-sonnet-4-5: 73.7 +/- 3.1, $19.50
claude-opus-5-5: 90.6 +/- 2.4, $26.00
gpt-5.6-sol: 82.8 +/- 2.7, $26.00
claude-opus-5: 92.9 +/- 1.8, $32.50
claude-opus-4-6: 88.4 +/- 2.3, $32.50
claude-opus-4-7: 86.4 +/- 2.5, $32.50
claude-opus-4-8: 85.4 +/- 2.5, $32.50
claude-opus-4-5: 80.7 +/- 2.4, $32.50
gpt-5.5: 77.3 +/- 3.0, $37.00
gpt-4-turbo: 46.6 +/- 2.7, $47.00
claude-fable-5-2: 92.4 +/- 1.9, $52.00
gpt-6-astra: 93.0 +/- 2.2, $65.00
claude-fable-5-1: 90.1 +/- 3.5, $65.00
claude-fable-5: 78.8 +/- 2.9, $65.00
o1: 50.3 +/- 2.8, $84.00
gpt-4: 35.7 +/- 2.4, $114
gemini-3.8-flash Cheap deepseek-v4-pro gemini-4-argon Best, Balanced Analysis and reasoning, Hardest 0 25 50 75 100 $1 $10 $100
deepseek-v4-flash-0731@deepinfra: 71.5 +/- 2.4, $0.45
deepseek-v4.1-flash-1017@deepinfra: 81.6 +/- 2.2, $0.74
gpt-5-nano: 9.6 +/- 1.2, $0.75
llama-4-scout-17b-16e-instruct: 0.3 +/- 0.3, $0.75
deepseek-v4-flash-0731: 71.5 +/- 2.4, $0.84
deepseek-v4-flash-0731@parasail: 71.5 +/- 2.4, $0.84
gpt-4.1-nano: 0.0 +/- 0.2, $0.90
deepseek-flash@deepinfra: 78.0 +/- 2.3, $1.05
gpt-6-luna: 50.9 +/- 2.4, $1.05
claude-haiku-5-5: 45.3 +/- 1.6, $1.05
qwen3-235b-a22b-instruct-2507@deepinfra: 6.1 +/- 0.9, $1.10
glm-5.3-flash: 61.0 +/- 2.5, $1.20
glm-5.3-flash@deepinfra: 61.0 +/- 2.5, $1.20
deepseek-v3.2: 30.5 +/- 1.5, $1.35
gpt-4o-mini: 0.0 +/- 0.4, $1.35
qwen3-235b-a22b-instruct-2507@parasail: 6.1 +/- 0.9, $1.62
qwen3-235b-a22b-instruct-2507: 6.1 +/- 0.9, $2.07
inkling-2-small@deepinfra: 89.1 +/- 1.8, $2.28
inkling-2-small: 89.1 +/- 1.8, $2.40
gpt-5.6-luna: 52.9 +/- 2.3, $2.40
gpt-5.4-nano: 20.9 +/- 1.5, $2.48
deepseek-v4.1-flash-1017: 81.6 +/- 2.2, $2.70
deepseek-v4.1-flash-1017@parasail: 81.6 +/- 2.2, $2.70
deepseek-flash: 78.0 +/- 2.3, $2.70
deepseek-flash@alibaba-intl: 78.0 +/- 2.3, $2.70
deepseek-flash@parasail: 78.0 +/- 2.3, $2.70
deepseek-flash@together: 78.0 +/- 2.3, $2.70
gemini-3.1-flash-lite: 8.6 +/- 1.1, $3.00
inkling-small: 55.5 +/- 2.4, $3.15
qwen3.8-27b@deepinfra: 17.8 +/- 1.3, $3.26
deepseek-v4-flash-0731@alibaba-intl: 71.5 +/- 2.4, $3.30
gpt-4.1-mini: 1.8 +/- 0.5, $3.60
gpt-5-mini: 25.2 +/- 1.1, $3.75
qwen3.8-27b@parasail: 17.8 +/- 1.3, $4.02
deepseek-flash-fast@parasail: 78.0 +/- 2.3, $4.05
glm-5.2@deepinfra: 49.9 +/- 2.2, $4.39
gemini-3.5-flash-lite: 31.9 +/- 1.9, $4.65
deepseek-r1-0528: 11.2 +/- 1.1, $4.73
gemini-3-flash-preview: 11.4 +/- 1.1, $6.00
qwen3.8-27b: 17.8 +/- 1.3, $6.00
deepseek-v4-pro-0813@deepinfra: 73.4 +/- 2.4, $7.80
gemini-3.8-flash: 73.3 +/- 1.2, $7.88
gemini-3.7-flash: 90.0 +/- 1.7, $7.88
gemini-3.5-flash: 82.3 +/- 2.2, $7.88
gemini-3.6-flash: 75.8 +/- 2.3, $7.88
gpt-5.4-mini: 35.6 +/- 2.0, $9.00
inkling: 58.0 +/- 2.4, $9.08
deepseek-v4-pro-0813: 73.4 +/- 2.4, $9.90
o4-mini: 27.6 +/- 1.2, $9.90
o3-mini: 18.8 +/- 1.4, $9.90
claude-haiku-4-5: 7.8 +/- 1.0, $10.50
glm-5.2: 49.9 +/- 2.2, $10.80
glm-5.2@alibaba-intl: 49.9 +/- 2.2, $10.80
glm-5.2@alibaba-us: 49.9 +/- 2.2, $10.80
grok-4.6: 76.1 +/- 2.3, $16.50
grok-4.7: 74.3 +/- 2.4, $16.50
o3: 29.2 +/- 1.2, $18.00
gpt-4.1: 3.8 +/- 0.5, $18.00
gpt-5: 32.6 +/- 1.3, $18.75
gpt-5.1: 36.3 +/- 1.4, $18.75
gemini-4-argon: 94.9 +/- 1.3, $21.00
gemini-4-carbon: 94.8 +/- 1.3, $21.00
gpt-6.1-sol: 95.1 +/- 1.3, $21.00
claude-sonnet-5-5: 68.5 +/- 1.6, $21.00
gpt-6-sol: 80.0 +/- 2.2, $21.00
gpt-4o: 2.4 +/- 0.4, $22.50
gemini-3.1-pro-preview: 73.8 +/- 1.5, $24.00
gpt-5.6-terra: 79.5 +/- 2.3, $24.00
gpt-5.2: 47.2 +/- 1.6, $26.25
kimi-k3@deepinfra: 70.8 +/- 2.4, $29.93
gpt-5.4: 65.5 +/- 1.6, $30.00
kimi-k3: 70.8 +/- 2.4, $31.50
kimi-k3@moonshot: 70.8 +/- 2.4, $31.50
claude-sonnet-4-6: 73.5 +/- 2.4, $31.50
claude-sonnet-4-5: 12.3 +/- 1.0, $31.50
claude-opus-5-5: 95.9 +/- 1.2, $42.00
gpt-5.6-sol: 91.2 +/- 1.7, $42.00
claude-opus-5: 92.9 +/- 1.5, $52.50
claude-opus-4-7: 62.3 +/- 1.5, $52.50
claude-opus-4-8: 82.0 +/- 2.2, $52.50
claude-opus-4-6: 51.8 +/- 1.5, $52.50
claude-opus-4-5: 20.7 +/- 1.2, $52.50
gpt-5.5: 88.9 +/- 1.8, $60.00
claude-fable-5-2: 94.0 +/- 1.4, $84.00
claude-fable-5: 91.5 +/- 1.6, $105
claude-fable-5-1: 77.1 +/- 1.2, $105
gpt-6-astra: 78.4 +/- 1.1, $105
o1: 13.0 +/- 1.1, $135
gemini-4-argon Best, Balanced Math, Medium: Nobody publishes a scored result for this one yet. The models below have a preference rating only.
Math, Hard 0 25 50 75 100 $1 $10 $100
phi-4: 64.9 +/- 1.1, $0.27
deepseek-v4-flash-0731@deepinfra: 86.8 +/- 1.7, $0.28
gemma-3-27b-it: 74.0 +/- 1.1, $0.30
deepseek-v4-flash@deepinfra: 79.7 +/- 2.1, $0.34
gpt-5-nano: 94.9 +/- 0.5, $0.46
llama-4-scout-17b-16e-instruct: 62.3 +/- 1.2, $0.47
llama-3.3-70b-instruct-turbo@deepinfra: 41.6 +/- 1.2, $0.49
deepseek-v4-flash-0731: 86.8 +/- 1.7, $0.53
deepseek-v4-flash-0731@parasail: 86.8 +/- 1.7, $0.53
deepseek-v4-flash@parasail: 79.7 +/- 2.1, $0.53
gpt-4.1-nano: 70.0 +/- 1.0, $0.56
claude-haiku-5-5: 92.8 +/- 1.3, $0.65
gpt-6-luna: 89.1 +/- 1.6, $0.65
deepseek-flash@deepinfra: 93.3 +/- 1.3, $0.66
glm-5.3-flash: 81.2 +/- 2.0, $0.75
glm-5.3-flash@deepinfra: 81.2 +/- 2.0, $0.75
deepseek-v4-flash: 79.7 +/- 2.1, $0.76
gpt-4o-mini: 52.6 +/- 1.1, $0.84
llama-3.3-70b-instruct-turbo@parasail: 41.6 +/- 1.2, $0.89
deepseek-v4-flash-vision-exp: 87.8 +/- 1.7, $1.01
gpt-5.6-luna: 87.2 +/- 1.7, $1.48
gpt-5.4-nano: 91.0 +/- 1.5, $1.53
minimax-m3@deepinfra: 77.0 +/- 2.2, $1.55
deepseek-flash: 93.3 +/- 1.3, $1.68
deepseek-flash@alibaba-intl: 93.3 +/- 1.3, $1.68
deepseek-flash@parasail: 93.3 +/- 1.3, $1.68
deepseek-flash@together: 93.3 +/- 1.3, $1.68
minimax-m3: 77.0 +/- 2.2, $1.68
minimax-m3@parasail: 77.0 +/- 2.2, $1.68
qwen-plus-2025-01-25: 65.3 +/- 1.1, $1.88
qwen3.8-27b@deepinfra: 86.2 +/- 1.8, $1.99
deepseek-v4-flash-0731@alibaba-intl: 86.8 +/- 1.7, $2.07
gpt-4.1-mini: 87.3 +/- 0.7, $2.24
gpt-5-mini: 97.9 +/- 0.3, $2.30
gpt-3.5-turbo: 11.6 +/- 0.6, $2.35
qwen3.8-27b@parasail: 86.2 +/- 1.8, $2.46
deepseek-flash-fast@parasail: 93.3 +/- 1.3, $2.52
minimax-m3-fast@parasail: 77.0 +/- 2.2, $2.52
glm-5.2@deepinfra: 89.8 +/- 1.6, $2.75
gemini-3.5-flash-lite: 73.7 +/- 2.3, $2.85
deepseek-r1-0528: 96.6 +/- 0.4, $2.94
llama-3.3-70b-instruct-turbo: 41.6 +/- 1.2, $3.02
glm-5.3@deepinfra: 87.9 +/- 1.7, $3.38
qwen3.6-27b@deepinfra: 79.9 +/- 2.1, $3.52
qwen3.8-27b: 86.2 +/- 1.8, $3.70
qwen3.6-plus: 83.7 +/- 1.9, $3.70
qwen3.6-27b: 79.9 +/- 2.1, $4.44
kimi-k2.6: 84.3 +/- 1.9, $4.65
gemini-3.7-flash: 93.5 +/- 1.3, $4.88
gemini-3.8-flash: 91.6 +/- 1.4, $4.88
gemini-3.5-flash: 88.2 +/- 1.7, $4.88
gemini-3.6-flash: 86.4 +/- 1.8, $4.88
deepseek-v4-pro-0813@deepinfra: 95.1 +/- 1.1, $4.94
deepseek-v4-pro@deepinfra: 90.7 +/- 1.5, $4.94
kimi-k2.7-code: 79.6 +/- 2.1, $5.50
gpt-5.4-mini: 78.5 +/- 2.1, $5.55
inkling: 88.4 +/- 1.7, $5.65
o4-mini: 97.8 +/- 0.3, $6.16
o3-mini: 96.5 +/- 0.4, $6.16
deepseek-v4-pro-0813: 95.1 +/- 1.1, $6.20
deepseek-v4-pro: 90.7 +/- 1.5, $6.20
deepseek-v4-pro@parasail: 90.7 +/- 1.5, $6.20
deepseek-v4-pro@together: 90.7 +/- 1.5, $6.20
claude-haiku-4-5: 86.9 +/- 0.8, $6.50
glm-5.2: 89.8 +/- 1.6, $6.76
glm-5.2@alibaba-intl: 89.8 +/- 1.6, $6.76
glm-5.2@alibaba-us: 89.8 +/- 1.6, $6.76
glm-5.3: 87.9 +/- 1.7, $6.76
glm-5.3@alibaba-intl: 87.9 +/- 1.7, $6.76
glm-5.3@parasail: 87.9 +/- 1.7, $6.76
deepseek-v4-pro-fast@parasail: 90.7 +/- 1.5, $9.31
qwen3.8-max-0902: 91.3 +/- 1.5, $9.40
glm-5.3-fast@parasail: 87.9 +/- 1.7, $10.14
grok-4.7: 95.7 +/- 1.0, $10.34
grok-4.6: 92.6 +/- 1.4, $10.34
o3: 97.8 +/- 0.3, $11.20
gpt-4.1: 83.0 +/- 0.9, $11.20
gpt-5: 98.1 +/- 0.3, $11.50
qwen3.7-max: 85.3 +/- 1.8, $11.75
qwen3.7-max@alibaba-us: 85.3 +/- 1.8, $11.75
gpt-6.1-sol: 96.8 +/- 0.9, $13.00
gpt-6-sol: 96.4 +/- 1.0, $13.00
claude-sonnet-5-5: 96.1 +/- 1.0, $13.00
claude-sonnet-5: 92.9 +/- 1.3, $13.00
gpt-4o: 49.8 +/- 1.1, $14.00
gpt-5.6-terra: 94.9 +/- 1.1, $14.80
gemini-3.1-pro-preview: 91.0 +/- 1.5, $14.80
gpt-5.2: 93.2 +/- 1.3, $16.10
gpt-5.4: 94.2 +/- 1.2, $18.50
kimi-k3@deepinfra: 84.4 +/- 1.9, $18.53
claude-sonnet-4-5: 97.7 +/- 0.4, $19.50
claude-sonnet-4-6: 87.0 +/- 1.7, $19.50
kimi-k3: 84.4 +/- 1.9, $19.50
kimi-k3@moonshot: 84.4 +/- 1.9, $19.50
claude-opus-5-5: 97.1 +/- 0.9, $26.00
gpt-5.6-sol: 96.2 +/- 1.0, $26.00
claude-opus-5: 95.7 +/- 1.0, $32.50
claude-opus-4-8: 94.3 +/- 1.2, $32.50
claude-opus-4-7: 92.9 +/- 1.3, $32.50
claude-opus-4-5: 90.4 +/- 1.5, $32.50
claude-opus-4-6: 89.3 +/- 1.6, $32.50
gpt-5.5: 95.9 +/- 1.0, $37.00
gpt-4-turbo: 46.7 +/- 1.1, $47.00
claude-fable-5-1: 97.0 +/- 0.9, $65.00
gpt-6-astra: 96.8 +/- 0.9, $65.00
claude-fable-5: 96.0 +/- 1.0, $65.00
o1: 81.7 +/- 1.1, $84.00
gpt-4: 23.0 +/- 0.9, $114
gpt-5-mini Best, Balanced, Cheap deepseek-v4-pro Math, Hardest 0 25 50 75 100 $1 $10 $100
gpt-oss-20b: 40.3 +/- 6.3, $0.30
gemma-3-4b-it: 7.5 +/- 2.8, $0.30
gpt-oss-120b@deepinfra: 88.9 +/- 4.5, $0.37
gemma-3-12b-it: 16.7 +/- 4.2, $0.38
phi-4: 13.8 +/- 3.7, $0.42
deepseek-v4-flash-0731@deepinfra: 76.0 +/- 1.8, $0.45
gemma-3-27b-it: 22.5 +/- 5.3, $0.48
qwen3.5-9b@deepinfra: 42.8 +/- 6.5, $0.53
qwen3-32b: 66.9 +/- 6.0, $0.66
qwen3.5-9b@parasail: 42.8 +/- 6.5, $0.68
gemma-4-26b-a4b-it: 82.2 +/- 5.8, $0.72
qwen3-14b: 66.4 +/- 6.2, $0.72
deepseek-v4.1-flash-1017@deepinfra: 85.3 +/- 1.4, $0.74
gpt-5-nano: 21.1 +/- 2.8, $0.75
llama-4-scout-17b-16e-instruct: 3.9 +/- 2.9, $0.75
llama-3.3-70b-instruct-turbo@deepinfra: 5.1 +/- 2.4, $0.78
deepseek-v4-flash-0731: 76.0 +/- 1.8, $0.84
deepseek-v4-flash-0731@parasail: 76.0 +/- 1.8, $0.84
qwen3.5-9b: 42.8 +/- 6.5, $0.89
qwen3.5-flash-2026-02-23: 33.4 +/- 2.0, $0.90
gpt-4.1-nano: 15.0 +/- 3.0, $0.90
gpt-6-luna: 88.9 +/- 1.3, $1.05
claude-haiku-5-5: 86.2 +/- 1.4, $1.05
deepseek-flash@deepinfra: 82.9 +/- 1.4, $1.05
qwen3-30b-a3b: 62.8 +/- 6.2, $1.11
glm-5.3-flash: 74.8 +/- 1.8, $1.20
glm-5.3-flash@deepinfra: 74.8 +/- 1.8, $1.20
gemma-4-31b-it: 73.3 +/- 6.7, $1.20
gpt-oss-120b: 88.9 +/- 4.5, $1.35
gpt-4o-mini: 3.8 +/- 1.4, $1.35
llama-3.3-70b-instruct-turbo@parasail: 5.1 +/- 2.4, $1.41
qwen3.6-35b-a3b@parasail: 53.5 +/- 2.8, $1.65
qwen3.6-35b-a3b@deepinfra: 53.5 +/- 2.8, $1.73
qwen3-30b-a3b-instruct-2507: 62.2 +/- 6.5, $1.80
qwen3.5-35b-a3b@deepinfra: 54.4 +/- 6.2, $1.92
qwen3.5-35b-a3b@parasail: 54.4 +/- 6.2, $1.95
gpt-5.6-luna: 57.3 +/- 4.4, $2.40
gpt-5.4-nano: 25.7 +/- 2.9, $2.48
minimax-m3@deepinfra: 71.1 +/- 6.8, $2.49
deepseek-v4.1-flash-1017: 85.3 +/- 1.4, $2.70
deepseek-v4.1-flash-1017@parasail: 85.3 +/- 1.4, $2.70
deepseek-flash: 82.9 +/- 1.4, $2.70
deepseek-flash@alibaba-intl: 82.9 +/- 1.4, $2.70
deepseek-flash@parasail: 82.9 +/- 1.4, $2.70
deepseek-flash@together: 82.9 +/- 1.4, $2.70
minimax-m3: 71.1 +/- 6.8, $2.70
minimax-m3@parasail: 71.1 +/- 6.8, $2.70
qwen3.7-plus: 63.9 +/- 2.4, $2.88
qwen3.7-plus@alibaba-us: 63.9 +/- 2.4, $2.88
qwen3.6-flash: 37.3 +/- 2.1, $3.00
qwen3.6-flash@alibaba-us: 37.3 +/- 2.1, $3.00
gemini-3.1-flash-lite: 33.5 +/- 3.9, $3.00
qwen-plus-2025-01-25: 17.8 +/- 4.4, $3.00
qwen-plus: 1.7 +/- 0.8, $3.00
inkling-small: 68.2 +/- 2.5, $3.15
deepseek-v4-flash-0731@alibaba-intl: 76.0 +/- 1.8, $3.30
gpt-4.1-mini: 18.6 +/- 2.2, $3.60
qwen3.5-35b-a3b: 54.4 +/- 6.2, $3.75
gpt-5-mini: 34.1 +/- 2.9, $3.75
gpt-3.5-turbo: 1.1 +/- 2.7, $3.75
glm-4.7: 42.9 +/- 2.4, $3.83
deepseek-flash-fast@parasail: 82.9 +/- 1.4, $4.05
minimax-m3-fast@parasail: 71.1 +/- 6.8, $4.05
qwen3-235b-a22b-thinking-2507: 47.6 +/- 2.7, $4.14
qwen3-30b-a3b-thinking-2507: 70.3 +/- 6.0, $4.20
glm-5.2@deepinfra: 41.4 +/- 4.2, $4.39
qwen3.6-35b-a3b: 53.5 +/- 2.8, $4.50
glm-4.6: 3.8 +/- 1.1, $4.50
gemini-3.5-flash-lite: 44.6 +/- 4.2, $4.65
llama-3.3-70b-instruct-turbo: 5.1 +/- 2.4, $4.68
deepseek-r1-0528: 66.4 +/- 4.9, $4.73
mistral-large-4-0: 78.5 +/- 1.5, $5.25
glm-5.3@deepinfra: 79.9 +/- 2.2, $5.44
qwen3.6-27b@deepinfra: 62.6 +/- 2.6, $5.76
qwen3.5-397b-a17b@deepinfra: 60.1 +/- 2.7, $5.85
gemini-3-flash-preview: 59.9 +/- 1.8, $6.00
qwen3.6-plus: 52.6 +/- 1.8, $6.00
qwen3.5-397b-a17b@parasail: 60.1 +/- 2.7, $6.90
qwen3.6-27b: 62.6 +/- 2.6, $7.20
qwen3.5-397b-a17b: 60.1 +/- 2.7, $7.20
kimi-k2.6: 64.1 +/- 1.6, $7.50
deepseek-v4-pro-0813@deepinfra: 81.6 +/- 1.5, $7.80
deepseek-v4-pro@deepinfra: 46.1 +/- 4.6, $7.80
gemini-3.8-flash: 83.7 +/- 1.5, $7.88
gemini-3.7-flash: 84.4 +/- 1.6, $7.88
gemini-3.5-flash: 65.3 +/- 2.2, $7.88
gemini-3.6-flash: 71.7 +/- 3.4, $7.88
kimi-k2.7-code: 74.8 +/- 2.1, $8.85
gpt-5.4-mini: 23.6 +/- 2.7, $9.00
inkling: 61.1 +/- 2.4, $9.08
deepseek-v4-pro-0813: 81.6 +/- 1.5, $9.90
deepseek-v4-pro: 46.1 +/- 4.6, $9.90
deepseek-v4-pro@parasail: 46.1 +/- 4.6, $9.90
deepseek-v4-pro@together: 46.1 +/- 4.6, $9.90
o4-mini: 28.2 +/- 2.7, $9.90
o3-mini: 20.7 +/- 2.7, $9.90
gpt-6.1-terra: 95.4 +/- 2.6, $10.50
claude-haiku-4-5: 20.0 +/- 3.2, $10.50
glm-5.3: 79.9 +/- 2.2, $10.80
glm-5.3@alibaba-intl: 79.9 +/- 2.2, $10.80
glm-5.3@parasail: 79.9 +/- 2.2, $10.80
glm-5.1: 54.5 +/- 1.8, $10.80
glm-5.2: 41.4 +/- 4.2, $10.80
glm-5.2@alibaba-intl: 41.4 +/- 4.2, $10.80
glm-5.2@alibaba-us: 41.4 +/- 4.2, $10.80
deepseek-v4-pro-fast@parasail: 46.1 +/- 4.6, $14.85
qwen3.8-max-0902: 82.8 +/- 2.9, $15.00
qwen3.6-max-preview: 57.1 +/- 2.5, $15.60
glm-5.3-fast@parasail: 79.9 +/- 2.2, $16.20
grok-4.6: 82.6 +/- 1.4, $16.50
grok-4.7: 75.5 +/- 1.6, $16.50
o3: 29.7 +/- 2.7, $18.00
gpt-4.1: 16.6 +/- 2.2, $18.00
qwen3.7-max: 80.1 +/- 2.1, $18.75
qwen3.7-max@alibaba-us: 80.1 +/- 2.1, $18.75
gpt-5: 39.2 +/- 3.0, $18.75
gpt-5.1: 19.9 +/- 3.7, $18.75
gpt-6.1-sol: 96.8 +/- 2.6, $21.00
gemini-4-carbon: 95.3 +/- 2.7, $21.00
gpt-6-sol: 94.9 +/- 2.7, $21.00
claude-sonnet-5-5: 94.4 +/- 2.7, $21.00
gemini-4-argon: 93.0 +/- 2.7, $21.00
claude-sonnet-5: 72.8 +/- 3.3, $21.00
gpt-4o: 2.3 +/- 0.8, $22.50
gemini-3.1-pro-preview: 64.1 +/- 1.7, $24.00
gpt-5.6-terra: 66.8 +/- 4.5, $24.00
gpt-5.2: 51.8 +/- 3.1, $26.25
kimi-k3@deepinfra: 70.4 +/- 4.0, $29.93
gpt-5.4: 60.9 +/- 3.3, $30.00
kimi-k3: 70.4 +/- 4.0, $31.50
kimi-k3@moonshot: 70.4 +/- 4.0, $31.50
claude-sonnet-4-6: 51.8 +/- 3.7, $31.50
claude-sonnet-4-5: 22.8 +/- 2.8, $31.50
claude-opus-5-5: 95.6 +/- 2.6, $42.00
gpt-5.6-sol: 77.2 +/- 4.2, $42.00
claude-opus-5: 92.8 +/- 1.6, $52.50
claude-opus-4-8: 72.3 +/- 2.5, $52.50
claude-opus-4-7: 66.9 +/- 2.2, $52.50
claude-opus-4-6: 61.8 +/- 1.9, $52.50
claude-opus-4-5: 34.4 +/- 2.6, $52.50
gpt-5.5: 69.1 +/- 4.5, $60.00
gpt-4-turbo: 3.7 +/- 1.2, $75.00
claude-fable-5-2: 97.4 +/- 0.7, $84.00
gpt-6-astra: 96.8 +/- 2.6, $105
claude-fable-5-1: 95.1 +/- 2.7, $105
claude-fable-5: 92.9 +/- 1.5, $105
o1: 17.6 +/- 2.8, $135
gpt-4: 1.1 +/- 0.5, $180
gpt-oss-120b@deepinfra Cheap deepseek-v4.1-flash-1017@deepinfra Balanced deepseek-v4-pro gpt-6.1-sol Best Writing code, Hard 0 25 50 75 100 $1 $10 $100
deepseek-v4.1-flash-1017@deepinfra: 38.8 +/- 11.5, $0.59
deepseek-flash@deepinfra: 64.4 +/- 11.4, $0.85
claude-haiku-5-5: 63.4 +/- 7.1, $0.88
gpt-6-luna: 25.7 +/- 6.8, $0.88
gemma-4-31b-it: 8.5 +/- 6.6, $0.94
glm-5.3-flash: 20.7 +/- 9.6, $0.98
glm-5.3-flash@deepinfra: 20.7 +/- 9.6, $0.98
minimax-m3.1-flash: 25.7 +/- 10.3, $1.11
minimax-m3.1-flash@parasail: 25.7 +/- 10.3, $1.11
mimo-v2.6-pro: 36.4 +/- 11.3, $1.83
gpt-5.6-luna: 21.1 +/- 9.8, $2.02
deepseek-flash: 64.4 +/- 11.4, $2.22
deepseek-flash@alibaba-intl: 64.4 +/- 11.4, $2.22
deepseek-flash@parasail: 64.4 +/- 11.4, $2.22
deepseek-flash@together: 64.4 +/- 11.4, $2.22
deepseek-v4.1-flash-1017: 38.8 +/- 11.5, $2.22
deepseek-v4.1-flash-1017@parasail: 38.8 +/- 11.5, $2.22
qwen3.8-27b@deepinfra: 15.9 +/- 8.6, $2.83
deepseek-flash-fast@parasail: 64.4 +/- 11.4, $3.33
qwen3.8-27b@parasail: 15.9 +/- 8.6, $3.45
glm-5.2@deepinfra: 36.7 +/- 11.5, $3.56
mistral-large-4-0: 32.4 +/- 7.4, $4.24
qwen3.8-27b: 15.9 +/- 8.6, $5.05
deepseek-v4-pro-0813@deepinfra: 33.3 +/- 10.0, $6.11
gemini-3.8-flash: 27.7 +/- 7.4, $6.56
gemini-3.7-flash: 41.1 +/- 12.2, $6.56
gemini-3.6-flash: 23.3 +/- 9.3, $6.56
inkling: 9.8 +/- 7.0, $7.47
deepseek-v4-pro-0813: 33.3 +/- 10.0, $7.99
glm-5.2: 36.7 +/- 11.5, $8.74
glm-5.2@alibaba-intl: 36.7 +/- 11.5, $8.74
glm-5.2@alibaba-us: 36.7 +/- 11.5, $8.74
gpt-6.1-terra: 66.4 +/- 7.5, $8.75
qwen3.8-max-0902: 37.2 +/- 7.3, $12.10
gemini-4-hydrogen: 61.8 +/- 11.5, $13.13
grok-4.7: 49.9 +/- 8.2, $13.31
grok-4.6: 29.9 +/- 7.2, $13.31
gpt-5: 4.4 +/- 3.4, $16.00
kimi-k3.1-pro@deepinfra: 38.7 +/- 11.5, $16.63
gemini-4-carbon: 72.7 +/- 10.5, $17.50
gpt-6.1-sol: 63.4 +/- 8.0, $17.50
gemini-4-argon: 58.7 +/- 8.3, $17.50
claude-sonnet-5-5: 80.6 +/- 5.7, $17.50
kimi-k3.1-pro: 38.7 +/- 11.5, $17.50
kimi-k3.1-pro@together: 38.7 +/- 11.5, $17.50
gpt-6-sol: 48.9 +/- 11.2, $17.50
claude-sonnet-5: 63.3 +/- 10.7, $17.50
gpt-5.6-terra: 33.5 +/- 8.1, $20.20
gemini-3.1-pro-preview: 8.9 +/- 4.6, $20.20
gpt-5.2: 32.2 +/- 10.0, $22.40
kimi-k3@deepinfra: 45.2 +/- 8.0, $24.94
gpt-5.4: 23.3 +/- 8.1, $25.25
kimi-k3: 45.2 +/- 8.0, $26.25
kimi-k3@moonshot: 45.2 +/- 8.0, $26.25
claude-opus-5-5: 84.2 +/- 5.4, $35.00
gpt-5.6-sol: 20.0 +/- 9.2, $35.00
claude-opus-5: 87.8 +/- 7.2, $43.75
claude-opus-4-7: 43.3 +/- 9.6, $43.75
claude-opus-4-8: 65.6 +/- 10.9, $43.75
claude-opus-4-6: 33.3 +/- 10.0, $43.75
gpt-5.5: 23.3 +/- 10.1, $50.50
claude-fable-5-2: 82.9 +/- 8.9, $70.00
claude-fable-5-1: 83.8 +/- 5.5, $87.50
gpt-6-astra: 67.8 +/- 7.8, $87.50
claude-fable-5: 70.0 +/- 10.4, $87.50
deepseek-flash@deepinfra Cheap gemini-4-carbon Best, Balanced Fixing code, Hard 0 25 50 75 100 $1 $10 $100
gpt-oss-20b: 3.1 +/- 1.5, $0.25
gpt-oss-120b@deepinfra: 25.6 +/- 1.9, $0.30
gemma-3-27b-it: 4.9 +/- 1.4, $0.38
qwen3.5-9b@deepinfra: 9.2 +/- 2.4, $0.40
qwen3.5-9b@parasail: 9.2 +/- 2.4, $0.54
qwen3-32b: 40.0 +/- 3.3, $0.54
deepseek-v4.1-flash-1017@deepinfra: 80.4 +/- 8.1, $0.59
gpt-5-nano: 7.0 +/- 1.9, $0.64
qwen3.5-9b: 9.2 +/- 2.4, $0.68
gpt-4.1-nano: 8.9 +/- 1.9, $0.74
claude-haiku-5-5: 77.6 +/- 8.5, $0.88
gpt-6-luna: 70.8 +/- 9.3, $0.88
qwen3-235b-a22b-instruct-2507@deepinfra: 59.6 +/- 3.3, $0.92
gemma-4-31b-it: 20.3 +/- 8.2, $0.94
glm-5.3-flash: 70.9 +/- 9.3, $0.98
glm-5.3-flash@deepinfra: 70.9 +/- 9.3, $0.98
deepseek-v3.2: 39.6 +/- 2.8, $1.03
minimax-m3.1-flash: 68.8 +/- 9.5, $1.11
minimax-m3.1-flash@parasail: 68.8 +/- 9.5, $1.11
gpt-oss-120b: 25.6 +/- 1.9, $1.11
gpt-4o-mini: 3.6 +/- 1.2, $1.11
qwen3-235b-a22b-instruct-2507@parasail: 59.6 +/- 3.3, $1.36
qwen3.6-35b-a3b@parasail: 24.6 +/- 3.2, $1.42
qwen3.6-35b-a3b@deepinfra: 24.6 +/- 3.2, $1.48
qwen3-235b-a22b-instruct-2507: 59.6 +/- 3.3, $1.70
mimo-v2.6-pro: 64.5 +/- 9.8, $1.83
deepseek-v4.1-flash-1017: 80.4 +/- 8.1, $2.22
deepseek-v4.1-flash-1017@parasail: 80.4 +/- 8.1, $2.22
qwen3.8-27b@deepinfra: 69.3 +/- 9.4, $2.83
gpt-4.1-mini: 32.4 +/- 3.1, $2.96
glm-4.7: 33.3 +/- 2.5, $3.16
gpt-5-mini: 47.0 +/- 1.7, $3.20
qwen3.8-27b@parasail: 69.3 +/- 9.4, $3.45
glm-5.2@deepinfra: 78.7 +/- 1.9, $3.56
glm-4.6: 24.5 +/- 2.4, $3.70
qwen3.6-35b-a3b: 24.6 +/- 3.2, $3.79
deepseek-r1-0528: 71.4 +/- 3.0, $3.90
mistral-large-4-0: 79.4 +/- 8.3, $4.24
gemini-3-flash-preview: 66.1 +/- 1.7, $5.05
qwen3.8-27b: 69.3 +/- 9.4, $5.05
qwen3.6-plus: 57.9 +/- 2.3, $5.05
deepseek-v4-pro@deepinfra: 77.6 +/- 1.9, $6.11
kimi-k2.6: 76.7 +/- 1.9, $6.23
gemini-3.5-flash: 79.3 +/- 1.8, $6.56
gemini-3.8-flash: 73.3 +/- 9.0, $6.56
inkling: 48.7 +/- 10.2, $7.47
deepseek-v4-pro: 77.6 +/- 1.9, $7.99
deepseek-v4-pro@parasail: 77.6 +/- 1.9, $7.99
deepseek-v4-pro@together: 77.6 +/- 1.9, $7.99
o4-mini: 72.0 +/- 3.0, $8.14
o3-mini: 60.4 +/- 3.3, $8.14
glm-5.2: 78.7 +/- 1.9, $8.74
glm-5.2@alibaba-intl: 78.7 +/- 1.9, $8.74
glm-5.2@alibaba-us: 78.7 +/- 1.9, $8.74
glm-5.1: 74.2 +/- 2.0, $8.74
gpt-6.1-terra: 89.2 +/- 6.3, $8.75
claude-haiku-4-5: 13.9 +/- 2.7, $8.75
deepseek-v4-pro-fast@parasail: 77.6 +/- 1.9, $11.98
qwen3.8-max-0902: 60.9 +/- 10.0, $12.10
gemini-4-hydrogen: 78.5 +/- 8.4, $13.13
qwen3.6-max-preview: 76.7 +/- 1.9, $13.13
grok-4.6: 76.7 +/- 8.6, $13.31
grok-4.7: 73.1 +/- 9.1, $13.31
o3: 68.4 +/- 1.7, $14.80
gpt-4.1: 49.8 +/- 1.9, $14.80
qwen3.7-max: 77.3 +/- 1.9, $15.13
qwen3.7-max@alibaba-us: 77.3 +/- 1.9, $15.13
gpt-5: 63.4 +/- 1.4, $16.00
gpt-5.1: 59.5 +/- 1.7, $16.00
kimi-k3.1-pro@deepinfra: 67.8 +/- 9.5, $16.63
gemini-4-carbon: 91.0 +/- 5.8, $17.50
gemini-4-argon: 85.7 +/- 7.2, $17.50
gpt-6.1-sol: 82.1 +/- 7.8, $17.50
claude-sonnet-5-5: 77.2 +/- 8.6, $17.50
kimi-k3.1-pro: 67.8 +/- 9.5, $17.50
kimi-k3.1-pro@together: 67.8 +/- 9.5, $17.50
gpt-4o: 28.4 +/- 1.7, $18.50
gpt-5.6-terra: 84.2 +/- 7.5, $20.20
gemini-3.1-pro-preview: 61.4 +/- 4.1, $20.20
gpt-5.2: 65.5 +/- 1.7, $22.40
kimi-k3@deepinfra: 53.1 +/- 10.2, $24.94
gpt-5.4: 78.9 +/- 1.4, $25.25
claude-sonnet-4-6: 67.9 +/- 1.6, $26.25
claude-sonnet-4-5: 60.9 +/- 1.7, $26.25
kimi-k3: 53.1 +/- 10.2, $26.25
kimi-k3@moonshot: 53.1 +/- 10.2, $26.25
claude-opus-5-5: 87.1 +/- 6.8, $35.00
claude-opus-4-7: 82.1 +/- 1.3, $43.75
claude-opus-4-6: 75.8 +/- 1.5, $43.75
claude-opus-4-5: 72.1 +/- 1.6, $43.75
gpt-5.5: 66.1 +/- 2.5, $50.50
claude-fable-5-2: 79.2 +/- 8.3, $70.00
claude-fable-5-1: 89.7 +/- 6.2, $87.50
gpt-6-astra: 87.5 +/- 6.8, $87.50
o1: 61.7 +/- 3.2, $111
deepseek-v4.1-flash-1017@deepinfra Cheap deepseek-v4-pro gpt-6.1-terra Best, Balanced Questions about a document, Medium: Nobody publishes a scored result for this one yet. The models below have a preference rating only.
How we tested Called through Zabio's own API at each thinking setting, question set pilot-2026-09-25, methodology 1.0. Without a thinking setting in the request, Zabio uses thinking off.
Model id to use through Zabio: deepseek-v4-pro
Changes 25 September 2026 First results: math and SQL at three thinking settings.