Open beta Zabio is in open beta: anyone can sign up and use it in production. Models, prices and features can still change while we learn from the first customers. Tell us what breaks at hello@zabio.ai.

Model

o1

o1 from OpenAI, Frontier class. List prices per million tokens, and what the public benchmarks say about it.

Class
Frontier
Input, per million
$15.00
Output, per million
$60.00
Cached input
$7.50
Context window
200,000
Longest answer
100,000
Processed by
OpenAI, United States
Processed in
global
Prices from
5 October 2026

o1 was retired on 14 November 2026. Calls that name it answer model_retired. All model retirements

Calling it

Name it in the model field, or let auto choose it for you.

"model": "o1"

Supports: images, tools, JSON schema

What it costs

  • Input $15.00 and output $60.00 per million tokens, at the provider list price.
  • Input read from cache: $7.50 per million tokens.

Public results for this model

Zabio has not run its own tests on this model. Everything below is aggregated from public benchmarks, with the sources named.

Aggregated from public resultsRead on 16 November 2026

o1
Writing, Medium: Rating 1,407.0 Evidence, in standard errors: -1.1 LMArena text style control, instruction following: 1,407.0 (Same settings as ours)Writing, MediumRating 1,407.0Creative writing, Medium: Rating 1,381.3 Evidence, in standard errors: -0.8 LMArena text style control, creative writing: 1,381.3 (Same settings as ours)Creative writing, MediumRating 1,381.3Pulling out data, Medium: Rating 1,407.0 Evidence, in standard errors: -1.1 LMArena text style control, instruction following: 1,407.0 (Same settings as ours)Pulling out data, MediumRating 1,407.0Conversation, Medium: 41.1 +/- 1.6 Evidence, in standard errors: -1.3 SimpleQA Verified: 41.1 (Different settings) LMArena text style control, multi turn: 1,389.8 (Same settings as ours)Conversation, Medium41.1Analysis and reasoning, Medium: 41.7 +/- 3.5 Evidence, in standard errors: -1.7 SimpleBench: 41.7 (Same settings as ours)Analysis and reasoning, Medium41.7Analysis and reasoning, Hard: 50.3 +/- 2.8 Evidence, in standard errors: -2.7 GPQA diamond: 50.3 (Same settings as ours) LMArena text style control, hard prompts: 1,418.0 (Same settings as ours)Analysis and reasoning, Hard50.3Analysis and reasoning, Hardest: 13.0 +/- 1.1 Evidence, in standard errors: -2.4 ARC-AGI: 18.0 (Same settings as ours) HLE: 8.0 (Same settings as ours) LMArena text style control, expert: 1,401.9 (Same settings as ours)Analysis and reasoning, Hardest13.0Math, Medium: Rating 1,408.9 Evidence, in standard errors: -1.4 LMArena text style control, math: 1,408.9 (Same settings as ours)Math, MediumRating 1,408.9Math, Hard: 81.7 +/- 1.1 Evidence, in standard errors: -1.6 MATH level 5: 81.7 (Same settings as ours)Math, Hard81.7Math, Hardest: 17.6 +/- 2.8 Evidence, in standard errors: -2.8 FrontierMath-2025-02-28-Private: 9.3 (Different settings) FrontierMath-Tiers-1-3-v2-Private: 8.4 (Close to ours) OTIS Mock AIME 2024-2025: 31.1 (Same settings as ours)Math, Hardest17.6Writing code, Hard: Rating 1,432.7 Evidence, in standard errors: -1.9 LMArena text style control, coding: 1,432.7 (Same settings as ours)Writing code, HardRating 1,432.7Fixing code, Hard: 61.7 +/- 3.2 Evidence, in standard errors: 0.2 Aider polyglot: 61.7 (Different settings)Fixing code, Hard61.7Reading images, Medium: Rating 1,168.3 Evidence, in standard errors: -2.5 LMArena vision, overall: 1,168.3 (Same settings as ours)Reading images, MediumRating 1,168.3Questions about a document, Medium: Rating 1,412.5 Evidence, in standard errors: -1.3 LMArena text style control, longer query: 1,412.5 (Same settings as ours)Questions about a document, MediumRating 1,412.5Questions about a document, Hard: 83.3 +/- 5.0 Evidence, in standard errors: 2.3 Fiction.LiveBench: 83.3 (Different settings)Questions about a document, Hard83.3
The benchmarks behind each bar
Kind of jobBenchmarkScoreSettings
Writing, MediumLMArena text style control, instruction followingRating 1,407.0Same settings as ours
Creative writing, MediumLMArena text style control, creative writingRating 1,381.3Same settings as ours
Pulling out data, MediumLMArena text style control, instruction followingRating 1,407.0Same settings as ours
Conversation, MediumSimpleQA Verified41.1Different settings
Conversation, MediumLMArena text style control, multi turnRating 1,389.8Same settings as ours
Analysis and reasoning, MediumSimpleBench41.7Same settings as ours
Analysis and reasoning, HardGPQA diamond50.3Same settings as ours
Analysis and reasoning, HardLMArena text style control, hard promptsRating 1,418.0Same settings as ours
Analysis and reasoning, HardestARC-AGI18.0Same settings as ours
Analysis and reasoning, HardestHLE8.0Same settings as ours
Analysis and reasoning, HardestLMArena text style control, expertRating 1,401.9Same settings as ours
Math, MediumLMArena text style control, mathRating 1,408.9Same settings as ours
Math, HardMATH level 581.7Same settings as ours
Math, HardestFrontierMath-2025-02-28-Private9.3Different settings
Math, HardestFrontierMath-Tiers-1-3-v2-Private8.4Close to ours
Math, HardestOTIS Mock AIME 2024-202531.1Same settings as ours
Writing code, HardLMArena text style control, codingRating 1,432.7Same settings as ours
Fixing code, HardAider polyglot61.7Different settings
Reading images, MediumLMArena vision, overallRating 1,168.3Same settings as ours
Questions about a document, MediumLMArena text style control, longer queryRating 1,412.5Same settings as ours
Questions about a document, HardFiction.LiveBench83.3Different settings
Price against score in each kind of job (15)

Price per 1,000 tasks, USD. Score, %. Greyed out: a cheaper model already scores as well. Ringed: the best, balanced and cheap choice.

Writing, Medium
5060708090100$0.1$1$10 deepseek-v4-flash-0731@deepinfra: 70.4 +/- 3.8, $0.12 deepseek-v4-flash@deepinfra: 66.3 +/- 3.9, $0.14 deepseek-v4-flash@parasail: 66.3 +/- 3.9, $0.22 deepseek-v4-flash-0731: 70.4 +/- 3.8, $0.22 deepseek-v4-flash-0731@parasail: 70.4 +/- 3.8, $0.22 deepseek-flash@deepinfra: 76.0 +/- 3.6, $0.28 gpt-6-luna: 61.8 +/- 4.0, $0.28 claude-haiku-5-5: 63.0 +/- 4.0, $0.28 glm-5.3-flash: 62.7 +/- 3.9, $0.32 glm-5.3-flash@deepinfra: 62.7 +/- 3.9, $0.32 deepseek-v4-flash: 66.3 +/- 3.9, $0.32 deepseek-v4-flash-vision-exp: 76.6 +/- 3.5, $0.43 gpt-5.6-luna: 60.2 +/- 4.0, $0.64 gpt-5.4-nano: 64.6 +/- 3.9, $0.66 minimax-m3@deepinfra: 66.4 +/- 3.9, $0.66 deepseek-flash: 76.0 +/- 3.6, $0.72 deepseek-flash@alibaba-intl: 76.0 +/- 3.6, $0.72 deepseek-flash@parasail: 76.0 +/- 3.6, $0.72 deepseek-flash@together: 76.0 +/- 3.6, $0.72 minimax-m3: 66.4 +/- 3.9, $0.72 minimax-m3@parasail: 66.4 +/- 3.9, $0.72 qwen3.8-27b@deepinfra: 74.6 +/- 3.6, $0.87 deepseek-v4-flash-0731@alibaba-intl: 70.4 +/- 3.8, $0.88 qwen3.8-27b@parasail: 74.6 +/- 3.6, $1.07 deepseek-flash-fast@parasail: 76.0 +/- 3.6, $1.08 minimax-m3-fast@parasail: 66.4 +/- 3.9, $1.08 glm-5.2@deepinfra: 69.3 +/- 3.9, $1.17 gemini-3.5-flash-lite: 68.4 +/- 3.8, $1.24 glm-5.3@deepinfra: 72.3 +/- 3.8, $1.45 qwen3.6-27b@deepinfra: 56.4 +/- 4.1, $1.54 qwen3.8-27b: 74.6 +/- 3.6, $1.60 qwen3.6-plus: 67.8 +/- 3.9, $1.60 qwen3.6-27b: 56.4 +/- 4.1, $1.92 kimi-k2.6: 68.4 +/- 3.9, $2.00 deepseek-v4-pro-0813@deepinfra: 75.4 +/- 3.6, $2.08 deepseek-v4-pro@deepinfra: 67.2 +/- 3.9, $2.08 gemini-3.8-flash: 83.9 +/- 3.1, $2.10 gemini-3.7-flash: 81.8 +/- 3.3, $2.10 gemini-3.6-flash: 79.1 +/- 3.4, $2.10 gemini-3.5-flash: 80.1 +/- 3.4, $2.10 kimi-k2.7-code: 63.2 +/- 3.9, $2.36 gpt-5.4-mini: 64.9 +/- 3.9, $2.40 inkling: 72.2 +/- 3.7, $2.42 deepseek-v4-pro-0813: 75.4 +/- 3.6, $2.64 deepseek-v4-pro: 67.2 +/- 3.9, $2.64 deepseek-v4-pro@parasail: 67.2 +/- 3.9, $2.64 deepseek-v4-pro@together: 67.2 +/- 3.9, $2.64 glm-5.3: 72.3 +/- 3.8, $2.88 glm-5.3@alibaba-intl: 72.3 +/- 3.8, $2.88 glm-5.3@parasail: 72.3 +/- 3.8, $2.88 glm-5.2: 69.3 +/- 3.9, $2.88 glm-5.2@alibaba-intl: 69.3 +/- 3.9, $2.88 glm-5.2@alibaba-us: 69.3 +/- 3.9, $2.88 deepseek-v4-pro-fast@parasail: 67.2 +/- 3.9, $3.96 qwen3.8-max-0902: 76.5 +/- 3.5, $4.00 glm-5.3-fast@parasail: 72.3 +/- 3.8, $4.32 grok-4.6: 77.7 +/- 3.5, $4.40 grok-4.7: 79.1 +/- 3.4, $4.40 qwen3.7-max: 75.9 +/- 3.6, $5.00 qwen3.7-max@alibaba-us: 75.9 +/- 3.6, $5.00 gpt-6.1-sol: 81.4 +/- 3.4, $5.60 gpt-6-sol: 77.7 +/- 3.6, $5.60 claude-sonnet-5-5: 63.1 +/- 3.9, $5.60 claude-sonnet-5: 68.1 +/- 3.9, $5.60 gemini-3.1-pro-preview: 82.6 +/- 3.2, $6.40 gpt-5.6-terra: 70.0 +/- 3.8, $6.40 gpt-5.2: 72.0 +/- 3.8, $7.00 kimi-k3@deepinfra: 80.0 +/- 3.4, $7.98 gpt-5.4: 73.0 +/- 3.7, $8.00 kimi-k3: 80.0 +/- 3.4, $8.40 kimi-k3@moonshot: 80.0 +/- 3.4, $8.40 claude-sonnet-4-6: 67.8 +/- 3.9, $8.40 gpt-5.6-sol: 79.6 +/- 3.5, $11.20 claude-opus-5-5: 75.3 +/- 3.7, $11.20 claude-opus-5: 77.2 +/- 3.6, $14.00 claude-opus-4-6: 73.1 +/- 3.7, $14.00 claude-opus-4-7: 71.8 +/- 3.8, $14.00 claude-opus-4-8: 73.8 +/- 3.7, $14.00 claude-opus-4-5: 73.5 +/- 3.7, $14.00 gpt-5.5: 79.6 +/- 3.5, $16.00 claude-fable-5: 84.2 +/- 3.2, $28.00 claude-fable-5-1: 80.5 +/- 3.4, $28.00 gpt-6-astra: 83.8 +/- 3.2, $28.00 gemini-3.8-flash Cheapclaude-fable-5 Best, Balanced
Creative writing, Medium
0255075100$0.1$1$10 deepseek-v4-flash-0731@deepinfra: 70.5 +/- 6.5, $0.12 deepseek-v4-flash@deepinfra: 67.3 +/- 6.6, $0.14 deepseek-v4-flash@parasail: 67.3 +/- 6.6, $0.22 deepseek-v4-flash-0731: 70.5 +/- 6.5, $0.22 deepseek-v4-flash-0731@parasail: 70.5 +/- 6.5, $0.22 deepseek-flash@deepinfra: 72.7 +/- 6.3, $0.28 gpt-6-luna: 63.3 +/- 6.8, $0.28 claude-haiku-5-5: 73.6 +/- 6.2, $0.28 glm-5.3-flash: 59.9 +/- 6.9, $0.32 glm-5.3-flash@deepinfra: 59.9 +/- 6.9, $0.32 deepseek-v4-flash: 67.3 +/- 6.6, $0.32 deepseek-v4-flash-vision-exp: 73.8 +/- 6.2, $0.43 gpt-5.6-luna: 69.1 +/- 6.5, $0.64 gpt-5.4-nano: 67.7 +/- 6.6, $0.66 minimax-m3@deepinfra: 63.6 +/- 6.8, $0.66 deepseek-flash: 72.7 +/- 6.3, $0.72 deepseek-flash@alibaba-intl: 72.7 +/- 6.3, $0.72 deepseek-flash@parasail: 72.7 +/- 6.3, $0.72 deepseek-flash@together: 72.7 +/- 6.3, $0.72 minimax-m3: 63.6 +/- 6.8, $0.72 minimax-m3@parasail: 63.6 +/- 6.8, $0.72 qwen3.8-27b@deepinfra: 73.1 +/- 6.3, $0.87 deepseek-v4-flash-0731@alibaba-intl: 70.5 +/- 6.5, $0.88 qwen3.8-27b@parasail: 73.1 +/- 6.3, $1.07 deepseek-flash-fast@parasail: 72.7 +/- 6.3, $1.08 minimax-m3-fast@parasail: 63.6 +/- 6.8, $1.08 glm-5.2@deepinfra: 62.9 +/- 6.8, $1.17 gemini-3.5-flash-lite: 67.5 +/- 6.6, $1.24 glm-5.3@deepinfra: 72.1 +/- 6.3, $1.45 qwen3.6-27b@deepinfra: 60.1 +/- 6.9, $1.54 qwen3.6-plus: 56.9 +/- 7.0, $1.60 qwen3.8-27b: 73.1 +/- 6.3, $1.60 qwen3.6-27b: 60.1 +/- 6.9, $1.92 kimi-k2.6: 66.9 +/- 6.7, $2.00 deepseek-v4-pro-0813@deepinfra: 74.9 +/- 6.1, $2.08 deepseek-v4-pro@deepinfra: 67.8 +/- 6.6, $2.08 gemini-3.8-flash: 82.8 +/- 5.3, $2.10 gemini-3.7-flash: 82.3 +/- 5.4, $2.10 gemini-3.6-flash: 75.6 +/- 6.1, $2.10 gemini-3.5-flash: 75.2 +/- 6.1, $2.10 kimi-k2.7-code: 68.6 +/- 6.6, $2.36 gpt-5.4-mini: 66.6 +/- 6.7, $2.40 inkling: 74.2 +/- 6.2, $2.42 deepseek-v4-pro-0813: 74.9 +/- 6.1, $2.64 deepseek-v4-pro: 67.8 +/- 6.6, $2.64 deepseek-v4-pro@parasail: 67.8 +/- 6.6, $2.64 deepseek-v4-pro@together: 67.8 +/- 6.6, $2.64 glm-5.3: 72.1 +/- 6.3, $2.88 glm-5.3@alibaba-intl: 72.1 +/- 6.3, $2.88 glm-5.3@parasail: 72.1 +/- 6.3, $2.88 glm-5.2: 62.9 +/- 6.8, $2.88 glm-5.2@alibaba-intl: 62.9 +/- 6.8, $2.88 glm-5.2@alibaba-us: 62.9 +/- 6.8, $2.88 deepseek-v4-pro-fast@parasail: 67.8 +/- 6.6, $3.96 qwen3.8-max-0902: 76.5 +/- 6.0, $4.00 glm-5.3-fast@parasail: 72.1 +/- 6.3, $4.32 grok-4.6: 78.3 +/- 5.8, $4.40 grok-4.7: 74.9 +/- 6.1, $4.40 qwen3.7-max: 76.3 +/- 6.0, $5.00 qwen3.7-max@alibaba-us: 76.3 +/- 6.0, $5.00 claude-sonnet-5-5: 69.9 +/- 6.5, $5.60 gpt-6.1-sol: 70.8 +/- 6.4, $5.60 gpt-6-sol: 66.9 +/- 6.7, $5.60 claude-sonnet-5: 62.3 +/- 6.9, $5.60 gemini-3.1-pro-preview: 77.3 +/- 5.9, $6.40 gpt-5.6-terra: 66.8 +/- 6.7, $6.40 gpt-5.2: 65.7 +/- 6.7, $7.00 kimi-k3@deepinfra: 75.4 +/- 6.1, $7.98 gpt-5.4: 79.0 +/- 5.8, $8.00 kimi-k3: 75.4 +/- 6.1, $8.40 kimi-k3@moonshot: 75.4 +/- 6.1, $8.40 claude-sonnet-4-6: 68.9 +/- 6.6, $8.40 claude-opus-5-5: 70.9 +/- 6.4, $11.20 gpt-5.6-sol: 75.9 +/- 6.1, $11.20 claude-opus-4-8: 80.6 +/- 5.6, $14.00 claude-opus-4-7: 69.4 +/- 6.5, $14.00 claude-opus-4-6: 66.1 +/- 6.7, $14.00 claude-opus-4-5: 65.9 +/- 6.7, $14.00 claude-opus-5: 61.3 +/- 6.9, $14.00 gpt-5.5: 72.6 +/- 6.3, $16.00 claude-fable-5-1: 78.1 +/- 5.9, $28.00 claude-fable-5: 71.7 +/- 6.4, $28.00 gpt-6-astra: 74.0 +/- 6.2, $28.00 deepseek-v4-flash-0731@deepinfra Cheapgemini-3.8-flash Best, Balanced
Pulling out data, Medium
5060708090100$0.1$1$10 deepseek-v4.1-flash-1017@deepinfra: 89.8 +/- 1.7, $0.14 gpt-6-luna: 86.7 +/- 1.9, $0.18 claude-haiku-5-5: 75.6 +/- 2.3, $0.18 glm-5.3-flash: 86.3 +/- 2.2, $0.22 glm-5.3-flash@deepinfra: 86.3 +/- 2.2, $0.22 gemma-4-31b-it: 76.7 +/- 2.1, $0.24 minimax-m3.1-flash: 84.7 +/- 2.1, $0.24 minimax-m3.1-flash@parasail: 84.7 +/- 2.1, $0.24 mimo-v2.6-pro: 85.2 +/- 2.1, $0.47 deepseek-v4.1-flash-1017: 89.8 +/- 1.7, $0.48 deepseek-v4.1-flash-1017@parasail: 89.8 +/- 1.7, $0.48 qwen3.8-27b@deepinfra: 86.4 +/- 2.2, $0.50 qwen3.8-27b@parasail: 86.4 +/- 2.2, $0.63 mistral-large-4-0: 88.3 +/- 1.8, $0.98 qwen3.8-27b: 86.4 +/- 2.2, $1.00 gemini-3.8-flash: 75.1 +/- 2.4, $1.35 inkling: 86.4 +/- 1.9, $1.61 gpt-6.1-terra: 91.2 +/- 1.6, $1.80 gemini-4-hydrogen: 90.6 +/- 1.8, $2.70 qwen3.8-max-0902: 88.2 +/- 1.9, $2.80 grok-4.6: 86.7 +/- 2.0, $3.08 grok-4.7: 88.5 +/- 1.8, $3.08 kimi-k3.1-pro@deepinfra: 88.6 +/- 1.9, $3.42 gemini-4-carbon: 90.3 +/- 2.2, $3.60 gemini-4-argon: 89.7 +/- 2.2, $3.60 gpt-6.1-sol: 91.4 +/- 1.6, $3.60 kimi-k3.1-pro: 88.6 +/- 1.9, $3.60 kimi-k3.1-pro@together: 88.6 +/- 1.9, $3.60 claude-sonnet-5-5: 78.8 +/- 2.2, $3.60 gpt-5.6-terra: 89.7 +/- 1.7, $4.00 kimi-k3@deepinfra: 88.4 +/- 1.9, $5.13 kimi-k3: 88.4 +/- 1.9, $5.40 kimi-k3@moonshot: 88.4 +/- 1.9, $5.40 claude-opus-5-5: 89.2 +/- 1.9, $7.20 claude-fable-5-2: 89.8 +/- 1.9, $14.40 claude-fable-5-1: 87.2 +/- 2.3, $18.00 gpt-6-astra: 91.5 +/- 1.6, $18.00 deepseek-v4.1-flash-1017@deepinfra Cheapglm-5.3-flash Balancedgemini-4-hydrogen Best
Conversation, Medium
0255075100$0.1$1$10 deepseek-v4-flash-0731@deepinfra: 33.6 +/- 1.5, $0.17 gpt-5-nano: 11.7 +/- 1.0, $0.24 deepseek-v4.1-flash-1017@deepinfra: 42.5 +/- 1.6, $0.27 qwen3.5-flash-2026-02-23: 20.3 +/- 1.3, $0.32 gpt-4.1-nano: 6.0 +/- 0.8, $0.32 deepseek-v4-flash-0731: 33.6 +/- 1.5, $0.34 deepseek-v4-flash-0731@parasail: 33.6 +/- 1.5, $0.34 gpt-6-luna: 41.4 +/- 1.6, $0.36 claude-haiku-5-5: 23.8 +/- 1.4, $0.36 deepseek-flash@deepinfra: 42.9 +/- 1.6, $0.39 gemma-4-31b-it: 10.4 +/- 1.0, $0.48 gpt-4o-mini: 8.3 +/- 0.9, $0.48 gpt-5.6-luna: 41.0 +/- 1.6, $0.80 gpt-5.4-nano: 11.7 +/- 1.0, $0.82 deepseek-flash: 42.9 +/- 1.6, $0.96 deepseek-flash@alibaba-intl: 42.9 +/- 1.6, $0.96 deepseek-flash@parasail: 42.9 +/- 1.6, $0.96 deepseek-flash@together: 42.9 +/- 1.6, $0.96 deepseek-v4.1-flash-1017: 42.5 +/- 1.6, $0.96 deepseek-v4.1-flash-1017@parasail: 42.5 +/- 1.6, $0.96 qwen3.6-flash: 15.9 +/- 1.2, $1.00 qwen3.6-flash@alibaba-us: 15.9 +/- 1.2, $1.00 inkling-small: 19.1 +/- 1.2, $1.20 gpt-5-mini: 21.6 +/- 1.3, $1.20 deepseek-v4-flash-0731@alibaba-intl: 33.6 +/- 1.5, $1.23 gpt-4.1-mini: 12.7 +/- 1.1, $1.28 qwen3-235b-a22b-thinking-2507: 40.4 +/- 1.6, $1.29 glm-4.7: 32.2 +/- 1.5, $1.34 deepseek-flash-fast@parasail: 42.9 +/- 1.6, $1.44 glm-5.2@deepinfra: 34.2 +/- 1.5, $1.62 glm-5.3@deepinfra: 41.0 +/- 1.6, $1.90 mistral-large-4-0: 20.0 +/- 1.3, $1.96 gemini-3-flash-preview: 66.8 +/- 1.5, $2.00 qwen3.6-plus: 44.1 +/- 1.6, $2.00 kimi-k2.6: 34.9 +/- 1.5, $2.60 gemini-3.8-flash: 69.7 +/- 1.5, $2.70 gemini-3.7-flash: 69.2 +/- 1.5, $2.70 gemini-3.6-flash: 66.2 +/- 1.5, $2.70 gemini-3.5-flash: 66.2 +/- 1.5, $2.70 gpt-5.4-mini: 29.4 +/- 1.4, $3.00 deepseek-v4-pro-0813@deepinfra: 52.9 +/- 1.6, $3.12 deepseek-v4-pro@deepinfra: 47.0 +/- 1.6, $3.12 kimi-k2.7-code: 36.5 +/- 1.5, $3.12 inkling: 40.3 +/- 1.6, $3.22 o4-mini: 19.6 +/- 1.3, $3.52 o3-mini: 15.3 +/- 1.1, $3.52 gpt-6.1-terra: 57.3 +/- 1.6, $3.60 claude-haiku-4-5: 13.2 +/- 1.1, $3.60 deepseek-v4-pro-0813: 52.9 +/- 1.6, $3.70 deepseek-v4-pro: 47.0 +/- 1.6, $3.70 deepseek-v4-pro@parasail: 47.0 +/- 1.6, $3.70 deepseek-v4-pro@together: 47.0 +/- 1.6, $3.70 glm-5.3: 41.0 +/- 1.6, $4.00 glm-5.3@alibaba-intl: 41.0 +/- 1.6, $4.00 glm-5.3@parasail: 41.0 +/- 1.6, $4.00 glm-5.1: 34.0 +/- 1.5, $4.00 glm-5.2: 34.2 +/- 1.5, $4.00 glm-5.2@alibaba-intl: 34.2 +/- 1.5, $4.00 glm-5.2@alibaba-us: 34.2 +/- 1.5, $4.00 qwen3.6-max-preview: 52.0 +/- 1.6, $5.20 deepseek-v4-pro-fast@parasail: 47.0 +/- 1.6, $5.54 qwen3.8-max-0902: 47.3 +/- 1.6, $5.60 glm-5.3-fast@parasail: 41.0 +/- 1.6, $6.00 gpt-5.1: 48.0 +/- 1.6, $6.00 gpt-5: 50.1 +/- 1.6, $6.00 grok-4.6: 49.3 +/- 1.6, $6.16 grok-4.7: 56.0 +/- 1.6, $6.16 o3: 49.4 +/- 1.6, $6.40 gpt-4.1: 31.1 +/- 1.5, $6.40 qwen3.7-max: 55.8 +/- 1.6, $7.00 qwen3.7-max@alibaba-us: 55.8 +/- 1.6, $7.00 gemini-4-carbon: 77.7 +/- 1.3, $7.20 gemini-4-argon: 76.0 +/- 1.4, $7.20 gpt-6.1-sol: 73.9 +/- 1.4, $7.20 gpt-6-sol: 60.7 +/- 1.6, $7.20 claude-sonnet-5-5: 46.5 +/- 1.6, $7.20 claude-sonnet-5: 33.7 +/- 1.5, $7.20 gemini-3.1-pro-preview: 73.5 +/- 1.4, $8.00 gpt-5.6-terra: 43.2 +/- 1.6, $8.00 gpt-4o: 26.0 +/- 1.4, $8.00 gpt-5.2: 32.8 +/- 1.5, $8.40 gpt-5.4: 45.1 +/- 1.6, $10.00 kimi-k3@deepinfra: 50.6 +/- 1.6, $10.26 kimi-k3: 50.6 +/- 1.6, $10.80 kimi-k3@moonshot: 50.6 +/- 1.6, $10.80 claude-sonnet-4-6: 32.8 +/- 1.5, $10.80 claude-sonnet-4-5: 23.7 +/- 1.4, $10.80 claude-opus-5-5: 72.2 +/- 1.4, $14.40 gpt-5.6-sol: 69.7 +/- 1.5, $14.40 claude-opus-4-7: 51.7 +/- 1.6, $18.00 claude-opus-4-8: 53.0 +/- 1.6, $18.00 claude-opus-5: 59.9 +/- 1.6, $18.00 claude-opus-4-6: 47.0 +/- 1.6, $18.00 claude-opus-4-5: 45.7 +/- 1.6, $18.00 gpt-5.5: 63.0 +/- 1.5, $20.00 claude-fable-5-2: 94.0 +/- 0.8, $28.80 claude-fable-5: 70.7 +/- 1.4, $36.00 gpt-6-astra: 75.6 +/- 1.4, $36.00 claude-fable-5-1: 70.8 +/- 1.4, $36.00 o1: 41.1 +/- 1.6, $48.00 gemini-4-carbon Best, Balanced, Cheapo1
Analysis and reasoning, Medium
0255075100$0.1$1$10 gpt-oss-120b@deepinfra: 22.1 +/- 2.9, $0.10 deepseek-v4-flash-0731@deepinfra: 61.1 +/- 3.5, $0.12 deepseek-v4-flash@deepinfra: 46.3 +/- 3.5, $0.14 llama-3.3-70b-instruct-turbo@deepinfra: 19.9 +/- 2.8, $0.21 deepseek-v4-flash-0731: 61.1 +/- 3.5, $0.22 deepseek-v4-flash-0731@parasail: 61.1 +/- 3.5, $0.22 deepseek-v4-flash@parasail: 46.3 +/- 3.5, $0.22 deepseek-flash@deepinfra: 66.7 +/- 3.3, $0.28 deepseek-v4-flash: 46.3 +/- 3.5, $0.32 gpt-oss-120b: 22.1 +/- 2.9, $0.36 gpt-4o-mini: 10.7 +/- 2.2, $0.36 llama-3.3-70b-instruct-turbo@parasail: 19.9 +/- 2.8, $0.38 gpt-5.6-luna: 46.8 +/- 3.5, $0.64 minimax-m3@deepinfra: 45.8 +/- 3.5, $0.66 deepseek-flash: 66.7 +/- 3.3, $0.72 deepseek-flash@alibaba-intl: 66.7 +/- 3.3, $0.72 deepseek-flash@parasail: 66.7 +/- 3.3, $0.72 deepseek-flash@together: 66.7 +/- 3.3, $0.72 minimax-m3: 45.8 +/- 3.5, $0.72 minimax-m3@parasail: 45.8 +/- 3.5, $0.72 qwen3.6-flash: 35.2 +/- 3.4, $0.80 qwen3.6-flash@alibaba-us: 35.2 +/- 3.4, $0.80 qwen3.8-27b@deepinfra: 60.2 +/- 3.5, $0.87 deepseek-v4-flash-0731@alibaba-intl: 61.1 +/- 3.5, $0.88 mistral-large-2512: 20.4 +/- 2.9, $1.00 glm-4.7: 47.7 +/- 3.5, $1.02 qwen3.8-27b@parasail: 60.2 +/- 3.5, $1.07 deepseek-flash-fast@parasail: 66.7 +/- 3.3, $1.08 minimax-m3-fast@parasail: 45.8 +/- 3.5, $1.08 glm-5.2@deepinfra: 58.8 +/- 3.5, $1.17 llama-3.3-70b-instruct-turbo: 19.9 +/- 2.8, $1.25 deepseek-r1-0528: 40.8 +/- 3.5, $1.26 glm-5.3@deepinfra: 66.2 +/- 3.3, $1.45 gemini-3-flash-preview: 61.1 +/- 3.5, $1.60 qwen3.8-27b: 60.2 +/- 3.5, $1.60 gemini-3.8-flash: 82.4 +/- 2.7, $2.10 gemini-3.5-flash: 76.7 +/- 3.0, $2.10 kimi-k2.7-code: 57.9 +/- 3.5, $2.36 inkling: 50.0 +/- 3.5, $2.42 o4-mini: 38.7 +/- 3.4, $2.64 o3-mini: 22.8 +/- 3.0, $2.64 gpt-6.1-terra: 72.2 +/- 3.2, $2.80 glm-5.3: 66.2 +/- 3.3, $2.88 glm-5.3@alibaba-intl: 66.2 +/- 3.3, $2.88 glm-5.3@parasail: 66.2 +/- 3.3, $2.88 glm-5.2: 58.8 +/- 3.5, $2.88 glm-5.2@alibaba-intl: 58.8 +/- 3.5, $2.88 glm-5.2@alibaba-us: 58.8 +/- 3.5, $2.88 glm-5.1: 55.1 +/- 3.5, $2.88 qwen3.6-max-preview: 63.0 +/- 3.4, $4.16 glm-5.3-fast@parasail: 66.2 +/- 3.3, $4.32 grok-4.6: 75.9 +/- 3.0, $4.40 o3: 53.1 +/- 3.5, $4.80 gpt-4.1: 27.0 +/- 3.1, $4.80 qwen3.7-max: 70.4 +/- 3.2, $5.00 qwen3.7-max@alibaba-us: 70.4 +/- 3.2, $5.00 gpt-5: 56.7 +/- 3.5, $5.00 gpt-5.1: 53.2 +/- 3.5, $5.00 gemini-4-carbon: 92.6 +/- 1.9, $5.60 gemini-4-argon: 91.7 +/- 2.0, $5.60 gpt-6.1-sol: 82.9 +/- 2.7, $5.60 claude-sonnet-5-5: 75.9 +/- 3.0, $5.60 gpt-6-sol: 73.1 +/- 3.1, $5.60 claude-sonnet-5: 60.6 +/- 3.5, $5.60 gpt-4o: 17.8 +/- 2.7, $6.00 gemini-3.1-pro-preview: 79.6 +/- 2.9, $6.40 gpt-5.6-terra: 48.9 +/- 3.5, $6.40 gpt-5.2: 45.8 +/- 3.5, $7.00 kimi-k3@deepinfra: 60.7 +/- 3.5, $7.98 kimi-k3: 60.7 +/- 3.5, $8.40 kimi-k3@moonshot: 60.7 +/- 3.5, $8.40 claude-sonnet-4-5: 54.3 +/- 3.5, $8.40 claude-opus-5-5: 88.4 +/- 2.3, $11.20 gpt-5.6-sol: 71.7 +/- 3.2, $11.20 claude-opus-5: 80.6 +/- 2.8, $14.00 claude-opus-4-6: 67.6 +/- 3.3, $14.00 claude-opus-4-8: 64.8 +/- 3.4, $14.00 claude-opus-4-5: 62.0 +/- 3.4, $14.00 claude-opus-4-7: 61.7 +/- 3.4, $14.00 gpt-5.5: 69.0 +/- 3.3, $16.00 gpt-4-turbo: 25.1 +/- 3.1, $20.00 claude-fable-5-2: 94.9 +/- 1.6, $22.40 claude-fable-5-1: 86.6 +/- 2.4, $28.00 gpt-6-astra: 86.5 +/- 2.4, $28.00 claude-fable-5: 81.9 +/- 2.7, $28.00 o1: 41.7 +/- 3.5, $36.00 gemini-4-carbon Best, Balanced, Cheapo1
Analysis and reasoning, Hard
0255075100$0.1$1$10$100 gpt-oss-20b: 53.2 +/- 2.5, $0.19 gemma-3-4b-it: 23.2 +/- 1.9, $0.19 gpt-oss-120b@deepinfra: 75.8 +/- 2.7, $0.23 gemma-3-12b-it: 39.5 +/- 2.5, $0.24 phi-4: 56.1 +/- 2.6, $0.27 deepseek-v4-flash-0731@deepinfra: 91.0 +/- 1.8, $0.28 gemma-3-27b-it: 47.7 +/- 2.7, $0.30 qwen3.5-9b@deepinfra: 79.0 +/- 2.4, $0.34 qwen3-32b: 65.7 +/- 2.8, $0.41 qwen3.5-9b@parasail: 79.0 +/- 2.4, $0.43 gemma-4-26b-a4b-it: 73.2 +/- 3.2, $0.45 qwen3-14b: 63.8 +/- 2.9, $0.46 gpt-5-nano: 57.6 +/- 3.5, $0.46 deepseek-v4.1-flash-1017@deepinfra: 87.0 +/- 3.1, $0.46 llama-4-scout-17b-16e-instruct: 51.8 +/- 3.2, $0.47 llama-3.3-70b-instruct-turbo@deepinfra: 47.4 +/- 2.9, $0.49 deepseek-v4-flash-0731: 91.0 +/- 1.8, $0.53 deepseek-v4-flash-0731@parasail: 91.0 +/- 1.8, $0.53 qwen3.5-flash-2026-02-23: 82.3 +/- 2.7, $0.56 gpt-4.1-nano: 48.9 +/- 2.5, $0.56 qwen3.5-9b: 79.0 +/- 2.4, $0.57 gpt-6-luna: 73.6 +/- 3.7, $0.65 claude-haiku-5-5: 85.5 +/- 2.8, $0.65 deepseek-flash@deepinfra: 89.8 +/- 1.7, $0.66 qwen3-30b-a3b: 61.7 +/- 3.0, $0.69 glm-5.3-flash: 80.6 +/- 3.3, $0.75 glm-5.3-flash@deepinfra: 80.6 +/- 3.3, $0.75 gemma-4-31b-it: 61.7 +/- 3.6, $0.76 minimax-m3.1-flash: 79.8 +/- 4.7, $0.84 minimax-m3.1-flash@parasail: 79.8 +/- 4.7, $0.84 gpt-oss-120b: 75.8 +/- 2.7, $0.84 gpt-4o-mini: 37.7 +/- 2.4, $0.84 llama-3.3-70b-instruct-turbo@parasail: 47.4 +/- 2.9, $0.89 qwen3.6-35b-a3b@parasail: 83.8 +/- 2.6, $1.01 qwen3.6-35b-a3b@deepinfra: 83.8 +/- 2.6, $1.06 qwen3-30b-a3b-instruct-2507: 55.6 +/- 2.9, $1.12 qwen3.5-35b-a3b@deepinfra: 83.5 +/- 2.2, $1.18 qwen3.5-35b-a3b@parasail: 83.5 +/- 2.2, $1.20 mimo-v2.6-pro: 82.2 +/- 4.1, $1.48 gpt-5.6-luna: 63.6 +/- 3.4, $1.48 gpt-5.4-nano: 55.6 +/- 3.5, $1.53 minimax-m3@deepinfra: 90.9 +/- 2.1, $1.55 deepseek-v4.1-flash-1017: 87.0 +/- 3.1, $1.68 deepseek-v4.1-flash-1017@parasail: 87.0 +/- 3.1, $1.68 deepseek-flash: 89.8 +/- 1.7, $1.68 deepseek-flash@alibaba-intl: 89.8 +/- 1.7, $1.68 deepseek-flash@parasail: 89.8 +/- 1.7, $1.68 deepseek-flash@together: 89.8 +/- 1.7, $1.68 minimax-m3: 90.9 +/- 2.1, $1.68 minimax-m3@parasail: 90.9 +/- 2.1, $1.68 qwen3.7-plus: 87.9 +/- 2.3, $1.79 qwen3.7-plus@alibaba-us: 87.9 +/- 2.3, $1.79 qwen3.6-flash: 83.3 +/- 2.7, $1.85 qwen3.6-flash@alibaba-us: 83.3 +/- 2.7, $1.85 gemini-3.1-flash-lite: 74.2 +/- 3.1, $1.85 qwen-plus-2025-01-25: 48.1 +/- 2.7, $1.88 inkling-small: 88.5 +/- 1.9, $1.98 qwen3.8-27b@deepinfra: 70.7 +/- 5.4, $1.99 deepseek-v4-flash-0731@alibaba-intl: 91.0 +/- 1.8, $2.07 gpt-4.1-mini: 65.9 +/- 2.7, $2.24 qwen3.5-35b-a3b: 83.5 +/- 2.2, $2.30 gpt-5-mini: 71.7 +/- 3.2, $2.30 gpt-3.5-turbo: 27.2 +/- 2.0, $2.35 glm-4.7: 83.3 +/- 2.4, $2.38 qwen3.8-27b@parasail: 70.7 +/- 5.4, $2.46 deepseek-flash-fast@parasail: 89.8 +/- 1.7, $2.52 minimax-m3-fast@parasail: 90.9 +/- 2.1, $2.52 qwen3-235b-a22b-thinking-2507: 80.1 +/- 2.6, $2.53 qwen3-30b-a3b-thinking-2507: 70.1 +/- 2.8, $2.56 glm-5.2@deepinfra: 71.2 +/- 3.2, $2.75 qwen3.6-35b-a3b: 83.8 +/- 2.6, $2.78 gemini-3.5-flash-lite: 75.8 +/- 3.1, $2.85 deepseek-r1-0528: 76.3 +/- 2.4, $2.94 llama-3.3-70b-instruct-turbo: 47.4 +/- 2.9, $3.02 mistral-large-4-0: 84.5 +/- 3.0, $3.29 glm-5.3@deepinfra: 90.9 +/- 1.6, $3.38 qwen3.6-27b@deepinfra: 85.9 +/- 2.5, $3.52 qwen3.5-397b-a17b@deepinfra: 85.9 +/- 2.5, $3.60 qwen3.6-plus: 88.4 +/- 2.3, $3.70 gemini-3-flash-preview: 83.2 +/- 1.9, $3.70 qwen3.8-27b: 70.7 +/- 5.4, $3.70 qwen3.5-397b-a17b@parasail: 85.9 +/- 2.5, $4.24 qwen3.6-27b: 85.9 +/- 2.5, $4.44 qwen3.5-397b-a17b: 85.9 +/- 2.5, $4.44 kimi-k2.6: 90.8 +/- 1.7, $4.65 gemini-3.8-flash: 88.0 +/- 3.0, $4.88 gemini-3.7-flash: 94.8 +/- 1.4, $4.88 gemini-3.5-flash: 88.9 +/- 2.2, $4.88 gemini-3.6-flash: 85.9 +/- 2.5, $4.88 deepseek-v4-pro-0813@deepinfra: 91.7 +/- 1.5, $4.94 deepseek-v4-pro@deepinfra: 73.2 +/- 3.2, $4.94 kimi-k2.7-code: 87.9 +/- 2.3, $5.50 gpt-5.4-mini: 64.1 +/- 3.4, $5.55 inkling: 75.6 +/- 3.7, $5.65 o4-mini: 75.3 +/- 3.1, $6.16 o3-mini: 68.2 +/- 3.3, $6.16 deepseek-v4-pro-0813: 91.7 +/- 1.5, $6.20 deepseek-v4-pro: 73.2 +/- 3.2, $6.20 deepseek-v4-pro@parasail: 73.2 +/- 3.2, $6.20 deepseek-v4-pro@together: 73.2 +/- 3.2, $6.20 gpt-6.1-terra: 87.4 +/- 3.0, $6.50 claude-haiku-4-5: 60.5 +/- 2.8, $6.50 glm-5.3: 90.9 +/- 1.6, $6.76 glm-5.3@alibaba-intl: 90.9 +/- 1.6, $6.76 glm-5.3@parasail: 90.9 +/- 1.6, $6.76 glm-5.1: 89.9 +/- 2.2, $6.76 glm-5.2: 71.2 +/- 3.2, $6.76 glm-5.2@alibaba-intl: 71.2 +/- 3.2, $6.76 glm-5.2@alibaba-us: 71.2 +/- 3.2, $6.76 deepseek-v4-pro-fast@parasail: 73.2 +/- 3.2, $9.31 qwen3.8-max-0902: 84.4 +/- 3.3, $9.40 qwen3.6-max-preview: 87.4 +/- 2.4, $9.62 gemini-4-hydrogen: 87.7 +/- 3.8, $9.75 glm-5.3-fast@parasail: 90.9 +/- 1.6, $10.14 grok-4.6: 91.1 +/- 2.4, $10.34 grok-4.7: 88.3 +/- 2.4, $10.34 o3: 79.8 +/- 2.9, $11.20 gpt-4.1: 66.9 +/- 2.8, $11.20 gpt-5: 71.7 +/- 3.2, $11.50 gpt-5.1: 66.7 +/- 3.4, $11.50 qwen3.7-max: 90.9 +/- 2.1, $11.75 qwen3.7-max@alibaba-us: 90.9 +/- 2.1, $11.75 kimi-k3.1-pro@deepinfra: 86.6 +/- 4.2, $12.35 gemini-4-argon: 92.8 +/- 2.5, $13.00 gemini-4-carbon: 93.2 +/- 2.0, $13.00 gpt-6.1-sol: 92.7 +/- 2.3, $13.00 claude-sonnet-5-5: 91.7 +/- 2.4, $13.00 gpt-6-sol: 94.3 +/- 1.5, $13.00 kimi-k3.1-pro: 86.6 +/- 4.2, $13.00 kimi-k3.1-pro@together: 86.6 +/- 4.2, $13.00 claude-sonnet-5: 80.3 +/- 2.8, $13.00 gpt-4o: 47.9 +/- 2.6, $14.00 gemini-3.1-pro-preview: 94.1 +/- 1.7, $14.80 gpt-5.6-terra: 81.9 +/- 2.9, $14.80 gpt-5.2: 73.2 +/- 3.2, $16.10 gpt-5.4: 74.8 +/- 3.1, $18.50 kimi-k3@deepinfra: 85.1 +/- 2.9, $18.53 kimi-k3: 85.1 +/- 2.9, $19.50 kimi-k3@moonshot: 85.1 +/- 2.9, $19.50 claude-sonnet-4-6: 78.8 +/- 2.9, $19.50 claude-sonnet-4-5: 73.7 +/- 3.1, $19.50 claude-opus-5-5: 90.6 +/- 2.4, $26.00 gpt-5.6-sol: 82.8 +/- 2.7, $26.00 claude-opus-5: 92.9 +/- 1.8, $32.50 claude-opus-4-6: 88.4 +/- 2.3, $32.50 claude-opus-4-7: 86.4 +/- 2.5, $32.50 claude-opus-4-8: 85.4 +/- 2.5, $32.50 claude-opus-4-5: 80.7 +/- 2.4, $32.50 gpt-5.5: 77.3 +/- 3.0, $37.00 gpt-4-turbo: 46.6 +/- 2.7, $47.00 claude-fable-5-2: 92.4 +/- 1.9, $52.00 gpt-6-astra: 93.0 +/- 2.2, $65.00 claude-fable-5-1: 90.1 +/- 3.5, $65.00 claude-fable-5: 78.8 +/- 2.9, $65.00 o1: 50.3 +/- 2.8, $84.00 gpt-4: 35.7 +/- 2.4, $114 gemini-3.8-flash Cheapgemini-4-argon Best, Balancedo1
Analysis and reasoning, Hardest
0255075100$1$10$100 deepseek-v4-flash-0731@deepinfra: 71.5 +/- 2.4, $0.45 deepseek-v4.1-flash-1017@deepinfra: 81.6 +/- 2.2, $0.74 gpt-5-nano: 9.6 +/- 1.2, $0.75 llama-4-scout-17b-16e-instruct: 0.3 +/- 0.3, $0.75 deepseek-v4-flash-0731: 71.5 +/- 2.4, $0.84 deepseek-v4-flash-0731@parasail: 71.5 +/- 2.4, $0.84 gpt-4.1-nano: 0.0 +/- 0.2, $0.90 deepseek-flash@deepinfra: 78.0 +/- 2.3, $1.05 gpt-6-luna: 50.9 +/- 2.4, $1.05 claude-haiku-5-5: 45.3 +/- 1.6, $1.05 qwen3-235b-a22b-instruct-2507@deepinfra: 6.1 +/- 0.9, $1.10 glm-5.3-flash: 61.0 +/- 2.5, $1.20 glm-5.3-flash@deepinfra: 61.0 +/- 2.5, $1.20 deepseek-v3.2: 30.5 +/- 1.5, $1.35 gpt-4o-mini: 0.0 +/- 0.4, $1.35 qwen3-235b-a22b-instruct-2507@parasail: 6.1 +/- 0.9, $1.62 qwen3-235b-a22b-instruct-2507: 6.1 +/- 0.9, $2.07 inkling-2-small@deepinfra: 89.1 +/- 1.8, $2.28 inkling-2-small: 89.1 +/- 1.8, $2.40 gpt-5.6-luna: 52.9 +/- 2.3, $2.40 gpt-5.4-nano: 20.9 +/- 1.5, $2.48 deepseek-v4.1-flash-1017: 81.6 +/- 2.2, $2.70 deepseek-v4.1-flash-1017@parasail: 81.6 +/- 2.2, $2.70 deepseek-flash: 78.0 +/- 2.3, $2.70 deepseek-flash@alibaba-intl: 78.0 +/- 2.3, $2.70 deepseek-flash@parasail: 78.0 +/- 2.3, $2.70 deepseek-flash@together: 78.0 +/- 2.3, $2.70 gemini-3.1-flash-lite: 8.6 +/- 1.1, $3.00 inkling-small: 55.5 +/- 2.4, $3.15 qwen3.8-27b@deepinfra: 17.8 +/- 1.3, $3.26 deepseek-v4-flash-0731@alibaba-intl: 71.5 +/- 2.4, $3.30 gpt-4.1-mini: 1.8 +/- 0.5, $3.60 gpt-5-mini: 25.2 +/- 1.1, $3.75 qwen3.8-27b@parasail: 17.8 +/- 1.3, $4.02 deepseek-flash-fast@parasail: 78.0 +/- 2.3, $4.05 glm-5.2@deepinfra: 49.9 +/- 2.2, $4.39 gemini-3.5-flash-lite: 31.9 +/- 1.9, $4.65 deepseek-r1-0528: 11.2 +/- 1.1, $4.73 gemini-3-flash-preview: 11.4 +/- 1.1, $6.00 qwen3.8-27b: 17.8 +/- 1.3, $6.00 deepseek-v4-pro-0813@deepinfra: 73.4 +/- 2.4, $7.80 gemini-3.8-flash: 73.3 +/- 1.2, $7.88 gemini-3.7-flash: 90.0 +/- 1.7, $7.88 gemini-3.5-flash: 82.3 +/- 2.2, $7.88 gemini-3.6-flash: 75.8 +/- 2.3, $7.88 gpt-5.4-mini: 35.6 +/- 2.0, $9.00 inkling: 58.0 +/- 2.4, $9.08 deepseek-v4-pro-0813: 73.4 +/- 2.4, $9.90 o4-mini: 27.6 +/- 1.2, $9.90 o3-mini: 18.8 +/- 1.4, $9.90 claude-haiku-4-5: 7.8 +/- 1.0, $10.50 glm-5.2: 49.9 +/- 2.2, $10.80 glm-5.2@alibaba-intl: 49.9 +/- 2.2, $10.80 glm-5.2@alibaba-us: 49.9 +/- 2.2, $10.80 grok-4.6: 76.1 +/- 2.3, $16.50 grok-4.7: 74.3 +/- 2.4, $16.50 o3: 29.2 +/- 1.2, $18.00 gpt-4.1: 3.8 +/- 0.5, $18.00 gpt-5: 32.6 +/- 1.3, $18.75 gpt-5.1: 36.3 +/- 1.4, $18.75 gemini-4-argon: 94.9 +/- 1.3, $21.00 gemini-4-carbon: 94.8 +/- 1.3, $21.00 gpt-6.1-sol: 95.1 +/- 1.3, $21.00 claude-sonnet-5-5: 68.5 +/- 1.6, $21.00 gpt-6-sol: 80.0 +/- 2.2, $21.00 gpt-4o: 2.4 +/- 0.4, $22.50 gemini-3.1-pro-preview: 73.8 +/- 1.5, $24.00 gpt-5.6-terra: 79.5 +/- 2.3, $24.00 gpt-5.2: 47.2 +/- 1.6, $26.25 kimi-k3@deepinfra: 70.8 +/- 2.4, $29.93 gpt-5.4: 65.5 +/- 1.6, $30.00 kimi-k3: 70.8 +/- 2.4, $31.50 kimi-k3@moonshot: 70.8 +/- 2.4, $31.50 claude-sonnet-4-6: 73.5 +/- 2.4, $31.50 claude-sonnet-4-5: 12.3 +/- 1.0, $31.50 claude-opus-5-5: 95.9 +/- 1.2, $42.00 gpt-5.6-sol: 91.2 +/- 1.7, $42.00 claude-opus-5: 92.9 +/- 1.5, $52.50 claude-opus-4-7: 62.3 +/- 1.5, $52.50 claude-opus-4-8: 82.0 +/- 2.2, $52.50 claude-opus-4-6: 51.8 +/- 1.5, $52.50 claude-opus-4-5: 20.7 +/- 1.2, $52.50 gpt-5.5: 88.9 +/- 1.8, $60.00 claude-fable-5-2: 94.0 +/- 1.4, $84.00 claude-fable-5: 91.5 +/- 1.6, $105 claude-fable-5-1: 77.1 +/- 1.2, $105 gpt-6-astra: 78.4 +/- 1.1, $105 o1: 13.0 +/- 1.1, $135 gemini-4-argon Best, Balancedo1
Math, Hard
0255075100$1$10$100 phi-4: 64.9 +/- 1.1, $0.27 deepseek-v4-flash-0731@deepinfra: 86.8 +/- 1.7, $0.28 gemma-3-27b-it: 74.0 +/- 1.1, $0.30 deepseek-v4-flash@deepinfra: 79.7 +/- 2.1, $0.34 gpt-5-nano: 94.9 +/- 0.5, $0.46 llama-4-scout-17b-16e-instruct: 62.3 +/- 1.2, $0.47 llama-3.3-70b-instruct-turbo@deepinfra: 41.6 +/- 1.2, $0.49 deepseek-v4-flash-0731: 86.8 +/- 1.7, $0.53 deepseek-v4-flash-0731@parasail: 86.8 +/- 1.7, $0.53 deepseek-v4-flash@parasail: 79.7 +/- 2.1, $0.53 gpt-4.1-nano: 70.0 +/- 1.0, $0.56 claude-haiku-5-5: 92.8 +/- 1.3, $0.65 gpt-6-luna: 89.1 +/- 1.6, $0.65 deepseek-flash@deepinfra: 93.3 +/- 1.3, $0.66 glm-5.3-flash: 81.2 +/- 2.0, $0.75 glm-5.3-flash@deepinfra: 81.2 +/- 2.0, $0.75 deepseek-v4-flash: 79.7 +/- 2.1, $0.76 gpt-4o-mini: 52.6 +/- 1.1, $0.84 llama-3.3-70b-instruct-turbo@parasail: 41.6 +/- 1.2, $0.89 deepseek-v4-flash-vision-exp: 87.8 +/- 1.7, $1.01 gpt-5.6-luna: 87.2 +/- 1.7, $1.48 gpt-5.4-nano: 91.0 +/- 1.5, $1.53 minimax-m3@deepinfra: 77.0 +/- 2.2, $1.55 deepseek-flash: 93.3 +/- 1.3, $1.68 deepseek-flash@alibaba-intl: 93.3 +/- 1.3, $1.68 deepseek-flash@parasail: 93.3 +/- 1.3, $1.68 deepseek-flash@together: 93.3 +/- 1.3, $1.68 minimax-m3: 77.0 +/- 2.2, $1.68 minimax-m3@parasail: 77.0 +/- 2.2, $1.68 qwen-plus-2025-01-25: 65.3 +/- 1.1, $1.88 qwen3.8-27b@deepinfra: 86.2 +/- 1.8, $1.99 deepseek-v4-flash-0731@alibaba-intl: 86.8 +/- 1.7, $2.07 gpt-4.1-mini: 87.3 +/- 0.7, $2.24 gpt-5-mini: 97.9 +/- 0.3, $2.30 gpt-3.5-turbo: 11.6 +/- 0.6, $2.35 qwen3.8-27b@parasail: 86.2 +/- 1.8, $2.46 deepseek-flash-fast@parasail: 93.3 +/- 1.3, $2.52 minimax-m3-fast@parasail: 77.0 +/- 2.2, $2.52 glm-5.2@deepinfra: 89.8 +/- 1.6, $2.75 gemini-3.5-flash-lite: 73.7 +/- 2.3, $2.85 deepseek-r1-0528: 96.6 +/- 0.4, $2.94 llama-3.3-70b-instruct-turbo: 41.6 +/- 1.2, $3.02 glm-5.3@deepinfra: 87.9 +/- 1.7, $3.38 qwen3.6-27b@deepinfra: 79.9 +/- 2.1, $3.52 qwen3.8-27b: 86.2 +/- 1.8, $3.70 qwen3.6-plus: 83.7 +/- 1.9, $3.70 qwen3.6-27b: 79.9 +/- 2.1, $4.44 kimi-k2.6: 84.3 +/- 1.9, $4.65 gemini-3.7-flash: 93.5 +/- 1.3, $4.88 gemini-3.8-flash: 91.6 +/- 1.4, $4.88 gemini-3.5-flash: 88.2 +/- 1.7, $4.88 gemini-3.6-flash: 86.4 +/- 1.8, $4.88 deepseek-v4-pro-0813@deepinfra: 95.1 +/- 1.1, $4.94 deepseek-v4-pro@deepinfra: 90.7 +/- 1.5, $4.94 kimi-k2.7-code: 79.6 +/- 2.1, $5.50 gpt-5.4-mini: 78.5 +/- 2.1, $5.55 inkling: 88.4 +/- 1.7, $5.65 o4-mini: 97.8 +/- 0.3, $6.16 o3-mini: 96.5 +/- 0.4, $6.16 deepseek-v4-pro-0813: 95.1 +/- 1.1, $6.20 deepseek-v4-pro: 90.7 +/- 1.5, $6.20 deepseek-v4-pro@parasail: 90.7 +/- 1.5, $6.20 deepseek-v4-pro@together: 90.7 +/- 1.5, $6.20 claude-haiku-4-5: 86.9 +/- 0.8, $6.50 glm-5.2: 89.8 +/- 1.6, $6.76 glm-5.2@alibaba-intl: 89.8 +/- 1.6, $6.76 glm-5.2@alibaba-us: 89.8 +/- 1.6, $6.76 glm-5.3: 87.9 +/- 1.7, $6.76 glm-5.3@alibaba-intl: 87.9 +/- 1.7, $6.76 glm-5.3@parasail: 87.9 +/- 1.7, $6.76 deepseek-v4-pro-fast@parasail: 90.7 +/- 1.5, $9.31 qwen3.8-max-0902: 91.3 +/- 1.5, $9.40 glm-5.3-fast@parasail: 87.9 +/- 1.7, $10.14 grok-4.7: 95.7 +/- 1.0, $10.34 grok-4.6: 92.6 +/- 1.4, $10.34 o3: 97.8 +/- 0.3, $11.20 gpt-4.1: 83.0 +/- 0.9, $11.20 gpt-5: 98.1 +/- 0.3, $11.50 qwen3.7-max: 85.3 +/- 1.8, $11.75 qwen3.7-max@alibaba-us: 85.3 +/- 1.8, $11.75 gpt-6.1-sol: 96.8 +/- 0.9, $13.00 gpt-6-sol: 96.4 +/- 1.0, $13.00 claude-sonnet-5-5: 96.1 +/- 1.0, $13.00 claude-sonnet-5: 92.9 +/- 1.3, $13.00 gpt-4o: 49.8 +/- 1.1, $14.00 gpt-5.6-terra: 94.9 +/- 1.1, $14.80 gemini-3.1-pro-preview: 91.0 +/- 1.5, $14.80 gpt-5.2: 93.2 +/- 1.3, $16.10 gpt-5.4: 94.2 +/- 1.2, $18.50 kimi-k3@deepinfra: 84.4 +/- 1.9, $18.53 claude-sonnet-4-5: 97.7 +/- 0.4, $19.50 claude-sonnet-4-6: 87.0 +/- 1.7, $19.50 kimi-k3: 84.4 +/- 1.9, $19.50 kimi-k3@moonshot: 84.4 +/- 1.9, $19.50 claude-opus-5-5: 97.1 +/- 0.9, $26.00 gpt-5.6-sol: 96.2 +/- 1.0, $26.00 claude-opus-5: 95.7 +/- 1.0, $32.50 claude-opus-4-8: 94.3 +/- 1.2, $32.50 claude-opus-4-7: 92.9 +/- 1.3, $32.50 claude-opus-4-5: 90.4 +/- 1.5, $32.50 claude-opus-4-6: 89.3 +/- 1.6, $32.50 gpt-5.5: 95.9 +/- 1.0, $37.00 gpt-4-turbo: 46.7 +/- 1.1, $47.00 claude-fable-5-1: 97.0 +/- 0.9, $65.00 gpt-6-astra: 96.8 +/- 0.9, $65.00 claude-fable-5: 96.0 +/- 1.0, $65.00 o1: 81.7 +/- 1.1, $84.00 gpt-4: 23.0 +/- 0.9, $114 gpt-5-mini Best, Balanced, Cheapo1
Math, Hardest
0255075100$1$10$100 gpt-oss-20b: 40.3 +/- 6.3, $0.30 gemma-3-4b-it: 7.5 +/- 2.8, $0.30 gpt-oss-120b@deepinfra: 88.9 +/- 4.5, $0.37 gemma-3-12b-it: 16.7 +/- 4.2, $0.38 phi-4: 13.8 +/- 3.7, $0.42 deepseek-v4-flash-0731@deepinfra: 76.0 +/- 1.8, $0.45 gemma-3-27b-it: 22.5 +/- 5.3, $0.48 qwen3.5-9b@deepinfra: 42.8 +/- 6.5, $0.53 qwen3-32b: 66.9 +/- 6.0, $0.66 qwen3.5-9b@parasail: 42.8 +/- 6.5, $0.68 gemma-4-26b-a4b-it: 82.2 +/- 5.8, $0.72 qwen3-14b: 66.4 +/- 6.2, $0.72 deepseek-v4.1-flash-1017@deepinfra: 85.3 +/- 1.4, $0.74 gpt-5-nano: 21.1 +/- 2.8, $0.75 llama-4-scout-17b-16e-instruct: 3.9 +/- 2.9, $0.75 llama-3.3-70b-instruct-turbo@deepinfra: 5.1 +/- 2.4, $0.78 deepseek-v4-flash-0731: 76.0 +/- 1.8, $0.84 deepseek-v4-flash-0731@parasail: 76.0 +/- 1.8, $0.84 qwen3.5-9b: 42.8 +/- 6.5, $0.89 qwen3.5-flash-2026-02-23: 33.4 +/- 2.0, $0.90 gpt-4.1-nano: 15.0 +/- 3.0, $0.90 gpt-6-luna: 88.9 +/- 1.3, $1.05 claude-haiku-5-5: 86.2 +/- 1.4, $1.05 deepseek-flash@deepinfra: 82.9 +/- 1.4, $1.05 qwen3-30b-a3b: 62.8 +/- 6.2, $1.11 glm-5.3-flash: 74.8 +/- 1.8, $1.20 glm-5.3-flash@deepinfra: 74.8 +/- 1.8, $1.20 gemma-4-31b-it: 73.3 +/- 6.7, $1.20 gpt-oss-120b: 88.9 +/- 4.5, $1.35 gpt-4o-mini: 3.8 +/- 1.4, $1.35 llama-3.3-70b-instruct-turbo@parasail: 5.1 +/- 2.4, $1.41 qwen3.6-35b-a3b@parasail: 53.5 +/- 2.8, $1.65 qwen3.6-35b-a3b@deepinfra: 53.5 +/- 2.8, $1.73 qwen3-30b-a3b-instruct-2507: 62.2 +/- 6.5, $1.80 qwen3.5-35b-a3b@deepinfra: 54.4 +/- 6.2, $1.92 qwen3.5-35b-a3b@parasail: 54.4 +/- 6.2, $1.95 gpt-5.6-luna: 57.3 +/- 4.4, $2.40 gpt-5.4-nano: 25.7 +/- 2.9, $2.48 minimax-m3@deepinfra: 71.1 +/- 6.8, $2.49 deepseek-v4.1-flash-1017: 85.3 +/- 1.4, $2.70 deepseek-v4.1-flash-1017@parasail: 85.3 +/- 1.4, $2.70 deepseek-flash: 82.9 +/- 1.4, $2.70 deepseek-flash@alibaba-intl: 82.9 +/- 1.4, $2.70 deepseek-flash@parasail: 82.9 +/- 1.4, $2.70 deepseek-flash@together: 82.9 +/- 1.4, $2.70 minimax-m3: 71.1 +/- 6.8, $2.70 minimax-m3@parasail: 71.1 +/- 6.8, $2.70 qwen3.7-plus: 63.9 +/- 2.4, $2.88 qwen3.7-plus@alibaba-us: 63.9 +/- 2.4, $2.88 qwen3.6-flash: 37.3 +/- 2.1, $3.00 qwen3.6-flash@alibaba-us: 37.3 +/- 2.1, $3.00 gemini-3.1-flash-lite: 33.5 +/- 3.9, $3.00 qwen-plus-2025-01-25: 17.8 +/- 4.4, $3.00 qwen-plus: 1.7 +/- 0.8, $3.00 inkling-small: 68.2 +/- 2.5, $3.15 deepseek-v4-flash-0731@alibaba-intl: 76.0 +/- 1.8, $3.30 gpt-4.1-mini: 18.6 +/- 2.2, $3.60 qwen3.5-35b-a3b: 54.4 +/- 6.2, $3.75 gpt-5-mini: 34.1 +/- 2.9, $3.75 gpt-3.5-turbo: 1.1 +/- 2.7, $3.75 glm-4.7: 42.9 +/- 2.4, $3.83 deepseek-flash-fast@parasail: 82.9 +/- 1.4, $4.05 minimax-m3-fast@parasail: 71.1 +/- 6.8, $4.05 qwen3-235b-a22b-thinking-2507: 47.6 +/- 2.7, $4.14 qwen3-30b-a3b-thinking-2507: 70.3 +/- 6.0, $4.20 glm-5.2@deepinfra: 41.4 +/- 4.2, $4.39 qwen3.6-35b-a3b: 53.5 +/- 2.8, $4.50 glm-4.6: 3.8 +/- 1.1, $4.50 gemini-3.5-flash-lite: 44.6 +/- 4.2, $4.65 llama-3.3-70b-instruct-turbo: 5.1 +/- 2.4, $4.68 deepseek-r1-0528: 66.4 +/- 4.9, $4.73 mistral-large-4-0: 78.5 +/- 1.5, $5.25 glm-5.3@deepinfra: 79.9 +/- 2.2, $5.44 qwen3.6-27b@deepinfra: 62.6 +/- 2.6, $5.76 qwen3.5-397b-a17b@deepinfra: 60.1 +/- 2.7, $5.85 gemini-3-flash-preview: 59.9 +/- 1.8, $6.00 qwen3.6-plus: 52.6 +/- 1.8, $6.00 qwen3.5-397b-a17b@parasail: 60.1 +/- 2.7, $6.90 qwen3.6-27b: 62.6 +/- 2.6, $7.20 qwen3.5-397b-a17b: 60.1 +/- 2.7, $7.20 kimi-k2.6: 64.1 +/- 1.6, $7.50 deepseek-v4-pro-0813@deepinfra: 81.6 +/- 1.5, $7.80 deepseek-v4-pro@deepinfra: 46.1 +/- 4.6, $7.80 gemini-3.8-flash: 83.7 +/- 1.5, $7.88 gemini-3.7-flash: 84.4 +/- 1.6, $7.88 gemini-3.5-flash: 65.3 +/- 2.2, $7.88 gemini-3.6-flash: 71.7 +/- 3.4, $7.88 kimi-k2.7-code: 74.8 +/- 2.1, $8.85 gpt-5.4-mini: 23.6 +/- 2.7, $9.00 inkling: 61.1 +/- 2.4, $9.08 deepseek-v4-pro-0813: 81.6 +/- 1.5, $9.90 deepseek-v4-pro: 46.1 +/- 4.6, $9.90 deepseek-v4-pro@parasail: 46.1 +/- 4.6, $9.90 deepseek-v4-pro@together: 46.1 +/- 4.6, $9.90 o4-mini: 28.2 +/- 2.7, $9.90 o3-mini: 20.7 +/- 2.7, $9.90 gpt-6.1-terra: 95.4 +/- 2.6, $10.50 claude-haiku-4-5: 20.0 +/- 3.2, $10.50 glm-5.3: 79.9 +/- 2.2, $10.80 glm-5.3@alibaba-intl: 79.9 +/- 2.2, $10.80 glm-5.3@parasail: 79.9 +/- 2.2, $10.80 glm-5.1: 54.5 +/- 1.8, $10.80 glm-5.2: 41.4 +/- 4.2, $10.80 glm-5.2@alibaba-intl: 41.4 +/- 4.2, $10.80 glm-5.2@alibaba-us: 41.4 +/- 4.2, $10.80 deepseek-v4-pro-fast@parasail: 46.1 +/- 4.6, $14.85 qwen3.8-max-0902: 82.8 +/- 2.9, $15.00 qwen3.6-max-preview: 57.1 +/- 2.5, $15.60 glm-5.3-fast@parasail: 79.9 +/- 2.2, $16.20 grok-4.6: 82.6 +/- 1.4, $16.50 grok-4.7: 75.5 +/- 1.6, $16.50 o3: 29.7 +/- 2.7, $18.00 gpt-4.1: 16.6 +/- 2.2, $18.00 qwen3.7-max: 80.1 +/- 2.1, $18.75 qwen3.7-max@alibaba-us: 80.1 +/- 2.1, $18.75 gpt-5: 39.2 +/- 3.0, $18.75 gpt-5.1: 19.9 +/- 3.7, $18.75 gpt-6.1-sol: 96.8 +/- 2.6, $21.00 gemini-4-carbon: 95.3 +/- 2.7, $21.00 gpt-6-sol: 94.9 +/- 2.7, $21.00 claude-sonnet-5-5: 94.4 +/- 2.7, $21.00 gemini-4-argon: 93.0 +/- 2.7, $21.00 claude-sonnet-5: 72.8 +/- 3.3, $21.00 gpt-4o: 2.3 +/- 0.8, $22.50 gemini-3.1-pro-preview: 64.1 +/- 1.7, $24.00 gpt-5.6-terra: 66.8 +/- 4.5, $24.00 gpt-5.2: 51.8 +/- 3.1, $26.25 kimi-k3@deepinfra: 70.4 +/- 4.0, $29.93 gpt-5.4: 60.9 +/- 3.3, $30.00 kimi-k3: 70.4 +/- 4.0, $31.50 kimi-k3@moonshot: 70.4 +/- 4.0, $31.50 claude-sonnet-4-6: 51.8 +/- 3.7, $31.50 claude-sonnet-4-5: 22.8 +/- 2.8, $31.50 claude-opus-5-5: 95.6 +/- 2.6, $42.00 gpt-5.6-sol: 77.2 +/- 4.2, $42.00 claude-opus-5: 92.8 +/- 1.6, $52.50 claude-opus-4-8: 72.3 +/- 2.5, $52.50 claude-opus-4-7: 66.9 +/- 2.2, $52.50 claude-opus-4-6: 61.8 +/- 1.9, $52.50 claude-opus-4-5: 34.4 +/- 2.6, $52.50 gpt-5.5: 69.1 +/- 4.5, $60.00 gpt-4-turbo: 3.7 +/- 1.2, $75.00 claude-fable-5-2: 97.4 +/- 0.7, $84.00 gpt-6-astra: 96.8 +/- 2.6, $105 claude-fable-5-1: 95.1 +/- 2.7, $105 claude-fable-5: 92.9 +/- 1.5, $105 o1: 17.6 +/- 2.8, $135 gpt-4: 1.1 +/- 0.5, $180 gpt-oss-120b@deepinfra Cheapdeepseek-v4.1-flash-1017@deepinfra Balancedgpt-6.1-sol Besto1
Writing code, Hard
0255075100$1$10$100 deepseek-v4.1-flash-1017@deepinfra: 38.8 +/- 11.5, $0.59 deepseek-flash@deepinfra: 64.4 +/- 11.4, $0.85 claude-haiku-5-5: 63.4 +/- 7.1, $0.88 gpt-6-luna: 25.7 +/- 6.8, $0.88 gemma-4-31b-it: 8.5 +/- 6.6, $0.94 glm-5.3-flash: 20.7 +/- 9.6, $0.98 glm-5.3-flash@deepinfra: 20.7 +/- 9.6, $0.98 minimax-m3.1-flash: 25.7 +/- 10.3, $1.11 minimax-m3.1-flash@parasail: 25.7 +/- 10.3, $1.11 mimo-v2.6-pro: 36.4 +/- 11.3, $1.83 gpt-5.6-luna: 21.1 +/- 9.8, $2.02 deepseek-flash: 64.4 +/- 11.4, $2.22 deepseek-flash@alibaba-intl: 64.4 +/- 11.4, $2.22 deepseek-flash@parasail: 64.4 +/- 11.4, $2.22 deepseek-flash@together: 64.4 +/- 11.4, $2.22 deepseek-v4.1-flash-1017: 38.8 +/- 11.5, $2.22 deepseek-v4.1-flash-1017@parasail: 38.8 +/- 11.5, $2.22 qwen3.8-27b@deepinfra: 15.9 +/- 8.6, $2.83 deepseek-flash-fast@parasail: 64.4 +/- 11.4, $3.33 qwen3.8-27b@parasail: 15.9 +/- 8.6, $3.45 glm-5.2@deepinfra: 36.7 +/- 11.5, $3.56 mistral-large-4-0: 32.4 +/- 7.4, $4.24 qwen3.8-27b: 15.9 +/- 8.6, $5.05 deepseek-v4-pro-0813@deepinfra: 33.3 +/- 10.0, $6.11 gemini-3.8-flash: 27.7 +/- 7.4, $6.56 gemini-3.7-flash: 41.1 +/- 12.2, $6.56 gemini-3.6-flash: 23.3 +/- 9.3, $6.56 inkling: 9.8 +/- 7.0, $7.47 deepseek-v4-pro-0813: 33.3 +/- 10.0, $7.99 glm-5.2: 36.7 +/- 11.5, $8.74 glm-5.2@alibaba-intl: 36.7 +/- 11.5, $8.74 glm-5.2@alibaba-us: 36.7 +/- 11.5, $8.74 gpt-6.1-terra: 66.4 +/- 7.5, $8.75 qwen3.8-max-0902: 37.2 +/- 7.3, $12.10 gemini-4-hydrogen: 61.8 +/- 11.5, $13.13 grok-4.7: 49.9 +/- 8.2, $13.31 grok-4.6: 29.9 +/- 7.2, $13.31 gpt-5: 4.4 +/- 3.4, $16.00 kimi-k3.1-pro@deepinfra: 38.7 +/- 11.5, $16.63 gemini-4-carbon: 72.7 +/- 10.5, $17.50 gpt-6.1-sol: 63.4 +/- 8.0, $17.50 gemini-4-argon: 58.7 +/- 8.3, $17.50 claude-sonnet-5-5: 80.6 +/- 5.7, $17.50 kimi-k3.1-pro: 38.7 +/- 11.5, $17.50 kimi-k3.1-pro@together: 38.7 +/- 11.5, $17.50 gpt-6-sol: 48.9 +/- 11.2, $17.50 claude-sonnet-5: 63.3 +/- 10.7, $17.50 gpt-5.6-terra: 33.5 +/- 8.1, $20.20 gemini-3.1-pro-preview: 8.9 +/- 4.6, $20.20 gpt-5.2: 32.2 +/- 10.0, $22.40 kimi-k3@deepinfra: 45.2 +/- 8.0, $24.94 gpt-5.4: 23.3 +/- 8.1, $25.25 kimi-k3: 45.2 +/- 8.0, $26.25 kimi-k3@moonshot: 45.2 +/- 8.0, $26.25 claude-opus-5-5: 84.2 +/- 5.4, $35.00 gpt-5.6-sol: 20.0 +/- 9.2, $35.00 claude-opus-5: 87.8 +/- 7.2, $43.75 claude-opus-4-7: 43.3 +/- 9.6, $43.75 claude-opus-4-8: 65.6 +/- 10.9, $43.75 claude-opus-4-6: 33.3 +/- 10.0, $43.75 gpt-5.5: 23.3 +/- 10.1, $50.50 claude-fable-5-2: 82.9 +/- 8.9, $70.00 claude-fable-5-1: 83.8 +/- 5.5, $87.50 gpt-6-astra: 67.8 +/- 7.8, $87.50 claude-fable-5: 70.0 +/- 10.4, $87.50 deepseek-flash@deepinfra Cheapgemini-4-carbon Best, Balanced
Fixing code, Hard
0255075100$1$10$100 gpt-oss-20b: 3.1 +/- 1.5, $0.25 gpt-oss-120b@deepinfra: 25.6 +/- 1.9, $0.30 gemma-3-27b-it: 4.9 +/- 1.4, $0.38 qwen3.5-9b@deepinfra: 9.2 +/- 2.4, $0.40 qwen3.5-9b@parasail: 9.2 +/- 2.4, $0.54 qwen3-32b: 40.0 +/- 3.3, $0.54 deepseek-v4.1-flash-1017@deepinfra: 80.4 +/- 8.1, $0.59 gpt-5-nano: 7.0 +/- 1.9, $0.64 qwen3.5-9b: 9.2 +/- 2.4, $0.68 gpt-4.1-nano: 8.9 +/- 1.9, $0.74 claude-haiku-5-5: 77.6 +/- 8.5, $0.88 gpt-6-luna: 70.8 +/- 9.3, $0.88 qwen3-235b-a22b-instruct-2507@deepinfra: 59.6 +/- 3.3, $0.92 gemma-4-31b-it: 20.3 +/- 8.2, $0.94 glm-5.3-flash: 70.9 +/- 9.3, $0.98 glm-5.3-flash@deepinfra: 70.9 +/- 9.3, $0.98 deepseek-v3.2: 39.6 +/- 2.8, $1.03 minimax-m3.1-flash: 68.8 +/- 9.5, $1.11 minimax-m3.1-flash@parasail: 68.8 +/- 9.5, $1.11 gpt-oss-120b: 25.6 +/- 1.9, $1.11 gpt-4o-mini: 3.6 +/- 1.2, $1.11 qwen3-235b-a22b-instruct-2507@parasail: 59.6 +/- 3.3, $1.36 qwen3.6-35b-a3b@parasail: 24.6 +/- 3.2, $1.42 qwen3.6-35b-a3b@deepinfra: 24.6 +/- 3.2, $1.48 qwen3-235b-a22b-instruct-2507: 59.6 +/- 3.3, $1.70 mimo-v2.6-pro: 64.5 +/- 9.8, $1.83 deepseek-v4.1-flash-1017: 80.4 +/- 8.1, $2.22 deepseek-v4.1-flash-1017@parasail: 80.4 +/- 8.1, $2.22 qwen3.8-27b@deepinfra: 69.3 +/- 9.4, $2.83 gpt-4.1-mini: 32.4 +/- 3.1, $2.96 glm-4.7: 33.3 +/- 2.5, $3.16 gpt-5-mini: 47.0 +/- 1.7, $3.20 qwen3.8-27b@parasail: 69.3 +/- 9.4, $3.45 glm-5.2@deepinfra: 78.7 +/- 1.9, $3.56 glm-4.6: 24.5 +/- 2.4, $3.70 qwen3.6-35b-a3b: 24.6 +/- 3.2, $3.79 deepseek-r1-0528: 71.4 +/- 3.0, $3.90 mistral-large-4-0: 79.4 +/- 8.3, $4.24 gemini-3-flash-preview: 66.1 +/- 1.7, $5.05 qwen3.8-27b: 69.3 +/- 9.4, $5.05 qwen3.6-plus: 57.9 +/- 2.3, $5.05 deepseek-v4-pro@deepinfra: 77.6 +/- 1.9, $6.11 kimi-k2.6: 76.7 +/- 1.9, $6.23 gemini-3.5-flash: 79.3 +/- 1.8, $6.56 gemini-3.8-flash: 73.3 +/- 9.0, $6.56 inkling: 48.7 +/- 10.2, $7.47 deepseek-v4-pro: 77.6 +/- 1.9, $7.99 deepseek-v4-pro@parasail: 77.6 +/- 1.9, $7.99 deepseek-v4-pro@together: 77.6 +/- 1.9, $7.99 o4-mini: 72.0 +/- 3.0, $8.14 o3-mini: 60.4 +/- 3.3, $8.14 glm-5.2: 78.7 +/- 1.9, $8.74 glm-5.2@alibaba-intl: 78.7 +/- 1.9, $8.74 glm-5.2@alibaba-us: 78.7 +/- 1.9, $8.74 glm-5.1: 74.2 +/- 2.0, $8.74 gpt-6.1-terra: 89.2 +/- 6.3, $8.75 claude-haiku-4-5: 13.9 +/- 2.7, $8.75 deepseek-v4-pro-fast@parasail: 77.6 +/- 1.9, $11.98 qwen3.8-max-0902: 60.9 +/- 10.0, $12.10 gemini-4-hydrogen: 78.5 +/- 8.4, $13.13 qwen3.6-max-preview: 76.7 +/- 1.9, $13.13 grok-4.6: 76.7 +/- 8.6, $13.31 grok-4.7: 73.1 +/- 9.1, $13.31 o3: 68.4 +/- 1.7, $14.80 gpt-4.1: 49.8 +/- 1.9, $14.80 qwen3.7-max: 77.3 +/- 1.9, $15.13 qwen3.7-max@alibaba-us: 77.3 +/- 1.9, $15.13 gpt-5: 63.4 +/- 1.4, $16.00 gpt-5.1: 59.5 +/- 1.7, $16.00 kimi-k3.1-pro@deepinfra: 67.8 +/- 9.5, $16.63 gemini-4-carbon: 91.0 +/- 5.8, $17.50 gemini-4-argon: 85.7 +/- 7.2, $17.50 gpt-6.1-sol: 82.1 +/- 7.8, $17.50 claude-sonnet-5-5: 77.2 +/- 8.6, $17.50 kimi-k3.1-pro: 67.8 +/- 9.5, $17.50 kimi-k3.1-pro@together: 67.8 +/- 9.5, $17.50 gpt-4o: 28.4 +/- 1.7, $18.50 gpt-5.6-terra: 84.2 +/- 7.5, $20.20 gemini-3.1-pro-preview: 61.4 +/- 4.1, $20.20 gpt-5.2: 65.5 +/- 1.7, $22.40 kimi-k3@deepinfra: 53.1 +/- 10.2, $24.94 gpt-5.4: 78.9 +/- 1.4, $25.25 claude-sonnet-4-6: 67.9 +/- 1.6, $26.25 claude-sonnet-4-5: 60.9 +/- 1.7, $26.25 kimi-k3: 53.1 +/- 10.2, $26.25 kimi-k3@moonshot: 53.1 +/- 10.2, $26.25 claude-opus-5-5: 87.1 +/- 6.8, $35.00 claude-opus-4-7: 82.1 +/- 1.3, $43.75 claude-opus-4-6: 75.8 +/- 1.5, $43.75 claude-opus-4-5: 72.1 +/- 1.6, $43.75 gpt-5.5: 66.1 +/- 2.5, $50.50 claude-fable-5-2: 79.2 +/- 8.3, $70.00 claude-fable-5-1: 89.7 +/- 6.2, $87.50 gpt-6-astra: 87.5 +/- 6.8, $87.50 o1: 61.7 +/- 3.2, $111 deepseek-v4.1-flash-1017@deepinfra Cheapgpt-6.1-terra Best, Balancedo1
Questions about a document, Hard
0255075100$1$10$100 gpt-oss-120b@deepinfra: 36.1 +/- 5.0, $0.45 gpt-5-nano: 44.4 +/- 5.0, $0.76 gemma-3-27b-it: 33.3 +/- 5.0, $0.78 qwen3-32b: 74.2 +/- 5.0, $0.89 llama-4-scout-17b-16e-instruct: 36.0 +/- 5.0, $1.07 llama-3.3-70b-instruct-turbo@deepinfra: 33.3 +/- 5.0, $1.09 gpt-4.1-nano: 25.0 +/- 5.0, $1.16 qwen3-14b: 62.5 +/- 5.0, $1.18 qwen3-30b-a3b: 40.6 +/- 5.0, $1.41 qwen3-next-80b-a3b-instruct@deepinfra: 41.7 +/- 5.0, $1.71 gpt-oss-120b: 36.1 +/- 5.0, $1.74 qwen3-next-80b-a3b-instruct@parasail: 41.7 +/- 5.0, $1.79 llama-3.3-70b-instruct-turbo@parasail: 33.3 +/- 5.0, $2.21 qwen3-next-80b-a3b-instruct: 41.7 +/- 5.0, $2.28 gpt-5-mini: 69.4 +/- 5.0, $3.80 qwen3-235b-a22b-thinking-2507: 75.0 +/- 5.0, $3.91 gpt-4.1-mini: 44.4 +/- 5.0, $4.64 deepseek-r1-0528: 75.0 +/- 5.0, $5.94 llama-3.3-70b-instruct-turbo: 33.3 +/- 5.0, $9.26 o4-mini: 77.8 +/- 5.0, $12.76 o3-mini: 50.0 +/- 5.0, $12.76 gpt-5: 97.2 +/- 5.0, $19.00 o3: 88.9 +/- 5.0, $23.20 gpt-4.1: 63.9 +/- 5.0, $23.20 o1: 83.3 +/- 5.0, $174 o1

Sources

  • Epoch AIEpoch AI, Capabilities and Benchmarking, epoch.ai/benchmarks, retrieved 2026-11-16License: CC BY 4.0Read on 16 November 2026
  • LMArenaLMArena leaderboard dataset (lmarena-ai/leaderboard-dataset on Hugging Face), retrieved 2026-11-16License: CC BY 4.0Read on 16 November 2026

What these numbers are

  • These numbers come from the sources named on each row, not from Zabio's own tests. Zabio's tests are shown separately and never combined with them into one figure.
  • Sources test models at their own settings. A settings mismatch against how Zabio calls the model is marked on the row and lowers its weight.
  • Scores are compared only inside one benchmark. Task scores combine benchmarks by evidence in standard errors, so a task score is a summary, not a benchmark result.
  • A model missing from every source is listed as such rather than left out.

Results from our own tests have their own section.

Models with no public results yet

These models are in the catalog, and no source we read publishes a result for them.

  • mistral-small-2603
  • codestral-2508
  • chat-latest
  • ministral-3b-2512
  • ministral-8b-2512
  • ministral-14b-2512
  • qwen3.8-2.4t-a95b
  • muse-glimmer-30b
  • qwen3.8-omni-flash
  • qwen3.8-flash
  • qwen3.8-2.4t-a95b@alibaba-intl
  • qwen3.5-plus-2026-04-20
  • qwen3.5-plus-2026-02-15
  • kimi-k3.1-pro-highspeed
  • ling-3.0-flash
  • ling-3.0-flash-vl
  • ling-3.1-flash
  • llama-guard-4-12b
  • muse-glimmer-30b@deepinfra
  • mistral-small-3.2-24b-instruct-2506
  • qwen3-vl-30b-a3b-instruct
  • gpt-audio-1.5
  • gpt-audio
  • gpt-audio-mini
  • ripple-65b-preview
  • glm-5.3-prime

Where to next

How routing picks between these models

Model and company names are trademarks of their owners. Zabio uses these models through their vendors' APIs and is not affiliated with the vendors.