Price and right answers Each dot is a model at one thinking setting: cost per 1,000 tasks across, right answers up. Up and to the left is better.
deepseek-flash deepseek-v4-pro Ringed: a best, balanced or cheap choiceMath Easy 50 60 70 80 90 100 $0.2 $0.5 $1 $2 deepseek-flash, Thinking high: 100.0%, $0.32 high deepseek-flash, Thinking low: 100.0%, $0.27 low deepseek-flash, Thinking off: 100.0%, $0.36 off deepseek-v4-pro, Thinking low: 100.0%, $0.81 low deepseek-v4-pro, Thinking high: 100.0%, $1.28 high deepseek-v4-pro, Thinking off: 96.0%, $1.04 off Medium 50 60 70 80 90 100 $0.5 $1 $2 $5 deepseek-flash, Thinking high: 96.4%, $1.04 high deepseek-flash, Thinking low: 98.2%, $0.62 low deepseek-flash, Thinking off: 94.5%, $0.59 off deepseek-v4-pro, Thinking low: 98.2%, $1.57 low deepseek-v4-pro, Thinking high: 98.2%, $3.44 high deepseek-v4-pro, Thinking off: 94.5%, $1.60 off Hard 50 60 70 80 90 100 $1 $2 $5 $10 $20 deepseek-flash, Thinking off: 96.3%, $1.66 off deepseek-flash, Thinking low: 100.0%, $2.26 low deepseek-flash, Thinking high: 100.0%, $3.00 high deepseek-v4-pro, Thinking off: 86.4%, $3.94 off deepseek-v4-pro, Thinking low: 98.1%, $5.98 low deepseek-v4-pro, Thinking high: 98.2%, $15.63 high Hardest 0 25 50 75 100 $2 $5 $10 $20 $50 deepseek-flash, Thinking low: 96.4%, $6.38 low deepseek-flash, Thinking high: 98.2%, $9.18 high deepseek-flash, Thinking off: 73.3%, $3.16 off deepseek-v4-pro, Thinking off: 44.2%, $6.52 off deepseek-v4-pro, Thinking low: 89.1%, $25.10 low deepseek-v4-pro, Thinking high: 98.2%, $48.68 high
SQL and data queries Easy 50 60 70 80 90 100 $0.2 $0.5 $1 $2 $5 deepseek-flash, Thinking off: 100.0%, $0.24 off deepseek-flash, Thinking low: 100.0%, $0.35 low deepseek-flash, Thinking high: 100.0%, $0.42 high deepseek-v4-pro, Thinking off: 100.0%, $1.03 off deepseek-v4-pro, Thinking low: 100.0%, $1.76 low deepseek-v4-pro, Thinking high: 100.0%, $2.35 high Medium 50 60 70 80 90 100 $1 $10 deepseek-flash, Thinking off: 96.4%, $0.45 off deepseek-flash, Thinking high: 98.2%, $1.87 high deepseek-flash, Thinking low: 96.4%, $1.26 low deepseek-v4-pro, Thinking off: 90.9%, $1.76 off deepseek-v4-pro, Thinking high: 98.2%, $12.51 high deepseek-v4-pro, Thinking low: 98.2%, $8.92 low Hard 50 60 70 80 90 100 $1 $10 deepseek-flash, Thinking off: 93.9%, $0.58 off deepseek-flash, Thinking high: 100.0%, $3.48 high deepseek-flash, Thinking low: 100.0%, $2.10 low deepseek-v4-pro, Thinking off: 84.8%, $2.24 off deepseek-v4-pro, Thinking low: 97.6%, $13.45 low deepseek-v4-pro, Thinking high: 100.0%, $22.80 high Hardest 50 60 70 80 90 100 $1 $10 deepseek-flash, Thinking off: 88.5%, $0.75 off deepseek-flash, Thinking low: 100.0%, $4.29 low deepseek-flash, Thinking high: 100.0%, $7.42 high deepseek-v4-pro, Thinking off: 84.8%, $3.06 off deepseek-v4-pro, Thinking low: 94.5%, $26.46 low deepseek-v4-pro, Thinking high: 98.2%, $41.95 high
Cost per 1,000 tasks, list price, log scale. Right answers, %.
How every model compares on public benchmarks Weekly results from Epoch AI, LMArena and LiveBench, shown as price against score for each kind of job. Zabio uses these same results to pick the model for each job.
Aggregated from public results Read on 16 November 2026
Writing Price against score, Medium 50 60 70 80 90 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 70.4 +/- 3.8, $0.12
deepseek-v4-flash@deepinfra: 66.3 +/- 3.9, $0.14
deepseek-v4-flash@parasail: 66.3 +/- 3.9, $0.22
deepseek-v4-flash-0731: 70.4 +/- 3.8, $0.22
deepseek-v4-flash-0731@parasail: 70.4 +/- 3.8, $0.22
deepseek-flash@deepinfra: 76.0 +/- 3.6, $0.28
gpt-6-luna: 61.8 +/- 4.0, $0.28
claude-haiku-5-5: 63.0 +/- 4.0, $0.28
glm-5.3-flash: 62.7 +/- 3.9, $0.32
glm-5.3-flash@deepinfra: 62.7 +/- 3.9, $0.32
deepseek-v4-flash: 66.3 +/- 3.9, $0.32
deepseek-v4-flash-vision-exp: 76.6 +/- 3.5, $0.43
gpt-5.6-luna: 60.2 +/- 4.0, $0.64
gpt-5.4-nano: 64.6 +/- 3.9, $0.66
minimax-m3@deepinfra: 66.4 +/- 3.9, $0.66
deepseek-flash: 76.0 +/- 3.6, $0.72
deepseek-flash@alibaba-intl: 76.0 +/- 3.6, $0.72
deepseek-flash@parasail: 76.0 +/- 3.6, $0.72
deepseek-flash@together: 76.0 +/- 3.6, $0.72
minimax-m3: 66.4 +/- 3.9, $0.72
minimax-m3@parasail: 66.4 +/- 3.9, $0.72
qwen3.8-27b@deepinfra: 74.6 +/- 3.6, $0.87
deepseek-v4-flash-0731@alibaba-intl: 70.4 +/- 3.8, $0.88
qwen3.8-27b@parasail: 74.6 +/- 3.6, $1.07
deepseek-flash-fast@parasail: 76.0 +/- 3.6, $1.08
minimax-m3-fast@parasail: 66.4 +/- 3.9, $1.08
glm-5.2@deepinfra: 69.3 +/- 3.9, $1.17
gemini-3.5-flash-lite: 68.4 +/- 3.8, $1.24
glm-5.3@deepinfra: 72.3 +/- 3.8, $1.45
qwen3.6-27b@deepinfra: 56.4 +/- 4.1, $1.54
qwen3.8-27b: 74.6 +/- 3.6, $1.60
qwen3.6-plus: 67.8 +/- 3.9, $1.60
qwen3.6-27b: 56.4 +/- 4.1, $1.92
kimi-k2.6: 68.4 +/- 3.9, $2.00
deepseek-v4-pro-0813@deepinfra: 75.4 +/- 3.6, $2.08
deepseek-v4-pro@deepinfra: 67.2 +/- 3.9, $2.08
gemini-3.8-flash: 83.9 +/- 3.1, $2.10
gemini-3.7-flash: 81.8 +/- 3.3, $2.10
gemini-3.6-flash: 79.1 +/- 3.4, $2.10
gemini-3.5-flash: 80.1 +/- 3.4, $2.10
kimi-k2.7-code: 63.2 +/- 3.9, $2.36
gpt-5.4-mini: 64.9 +/- 3.9, $2.40
inkling: 72.2 +/- 3.7, $2.42
deepseek-v4-pro-0813: 75.4 +/- 3.6, $2.64
deepseek-v4-pro: 67.2 +/- 3.9, $2.64
deepseek-v4-pro@parasail: 67.2 +/- 3.9, $2.64
deepseek-v4-pro@together: 67.2 +/- 3.9, $2.64
glm-5.3: 72.3 +/- 3.8, $2.88
glm-5.3@alibaba-intl: 72.3 +/- 3.8, $2.88
glm-5.3@parasail: 72.3 +/- 3.8, $2.88
glm-5.2: 69.3 +/- 3.9, $2.88
glm-5.2@alibaba-intl: 69.3 +/- 3.9, $2.88
glm-5.2@alibaba-us: 69.3 +/- 3.9, $2.88
deepseek-v4-pro-fast@parasail: 67.2 +/- 3.9, $3.96
qwen3.8-max-0902: 76.5 +/- 3.5, $4.00
glm-5.3-fast@parasail: 72.3 +/- 3.8, $4.32
grok-4.6: 77.7 +/- 3.5, $4.40
grok-4.7: 79.1 +/- 3.4, $4.40
qwen3.7-max: 75.9 +/- 3.6, $5.00
qwen3.7-max@alibaba-us: 75.9 +/- 3.6, $5.00
gpt-6.1-sol: 81.4 +/- 3.4, $5.60
gpt-6-sol: 77.7 +/- 3.6, $5.60
claude-sonnet-5-5: 63.1 +/- 3.9, $5.60
claude-sonnet-5: 68.1 +/- 3.9, $5.60
gemini-3.1-pro-preview: 82.6 +/- 3.2, $6.40
gpt-5.6-terra: 70.0 +/- 3.8, $6.40
gpt-5.2: 72.0 +/- 3.8, $7.00
kimi-k3@deepinfra: 80.0 +/- 3.4, $7.98
gpt-5.4: 73.0 +/- 3.7, $8.00
kimi-k3: 80.0 +/- 3.4, $8.40
kimi-k3@moonshot: 80.0 +/- 3.4, $8.40
claude-sonnet-4-6: 67.8 +/- 3.9, $8.40
gpt-5.6-sol: 79.6 +/- 3.5, $11.20
claude-opus-5-5: 75.3 +/- 3.7, $11.20
claude-opus-5: 77.2 +/- 3.6, $14.00
claude-opus-4-6: 73.1 +/- 3.7, $14.00
claude-opus-4-7: 71.8 +/- 3.8, $14.00
claude-opus-4-8: 73.8 +/- 3.7, $14.00
claude-opus-4-5: 73.5 +/- 3.7, $14.00
gpt-5.5: 79.6 +/- 3.5, $16.00
claude-fable-5: 84.2 +/- 3.2, $28.00
claude-fable-5-1: 80.5 +/- 3.4, $28.00
gpt-6-astra: 83.8 +/- 3.2, $28.00
gemini-3.8-flash Cheap claude-fable-5 Best, Balanced Ranking, Medium 1. claude-fable-5: 84.2 +/- 3.2 1 claude-fable-5 84.2 4. gemini-3.8-flash: 83.9 +/- 3.1 4 gemini-3.8-flash 83.9 5. claude-fable-5-1: 80.5 +/- 3.4 5 claude-fable-5-1 80.5 7. gpt-6-astra: 83.8 +/- 3.2 7 gpt-6-astra 83.8 8. gpt-6.1-sol: 81.4 +/- 3.4 8 gpt-6.1-sol 81.4 9. gemini-3.7-flash: 81.8 +/- 3.3 9 gemini-3.7-flash 81.8 10. gpt-5.6-sol: 79.6 +/- 3.5 10 gpt-5.6-sol 79.6 11. gemini-3.1-pro-preview: 82.6 +/- 3.2 11 gemini-3.1-pro-preview 82.6 12. claude-opus-5: 77.2 +/- 3.6 12 claude-opus-5 77.2 13. kimi-k3: 80.0 +/- 3.4 13 kimi-k3 80.0 14. kimi-k3@deepinfra: 80.0 +/- 3.4 14 kimi-k3@deepinfra 80.0 15. kimi-k3@moonshot: 80.0 +/- 3.4 15 kimi-k3@moonshot 80.0 16. claude-opus-5-5: 75.3 +/- 3.7 16 claude-opus-5-5 75.3 17. gpt-5.5: 79.6 +/- 3.5 17 gpt-5.5 79.6 21. claude-opus-4-6: 73.1 +/- 3.7 21 claude-opus-4-6 73.1 22. gemini-3.6-flash: 79.1 +/- 3.4 22 gemini-3.6-flash 79.1 23. gemini-3.5-flash: 80.1 +/- 3.4 23 gemini-3.5-flash 80.1 24. qwen3.8-max-0902: 76.5 +/- 3.5 24 qwen3.8-max-0902 76.5 25. claude-opus-4-7: 71.8 +/- 3.8 25 claude-opus-4-7 71.8 26. deepseek-flash: 76.0 +/- 3.6 26 deepseek-flash 76.0 27. deepseek-flash-fast@parasail: 76.0 +/- 3.6 27 deepseek-flash-fast@parasail 76.0 28. deepseek-flash@alibaba-intl: 76.0 +/- 3.6 28 deepseek-flash@alibaba-intl 76.0 29. deepseek-flash@deepinfra: 76.0 +/- 3.6 29 deepseek-flash@deepinfra 76.0 30. deepseek-flash@parasail: 76.0 +/- 3.6 30 deepseek-flash@parasail 76.0 31. deepseek-flash@together: 76.0 +/- 3.6 31 deepseek-flash@together 76.0 32. claude-opus-4-8: 73.8 +/- 3.7 32 claude-opus-4-8 73.8 33. claude-opus-4-5: 73.5 +/- 3.7 33 claude-opus-4-5 73.5 35. glm-5.3: 72.3 +/- 3.8 35 glm-5.3 72.3 36. glm-5.3-fast@parasail: 72.3 +/- 3.8 36 glm-5.3-fast@parasail 72.3 37. glm-5.3@alibaba-intl: 72.3 +/- 3.8 37 glm-5.3@alibaba-intl 72.3 38. glm-5.3@deepinfra: 72.3 +/- 3.8 38 glm-5.3@deepinfra 72.3 39. glm-5.3@parasail: 72.3 +/- 3.8 39 glm-5.3@parasail 72.3 40. qwen3.7-max: 75.9 +/- 3.6 40 qwen3.7-max 75.9 41. qwen3.7-max@alibaba-us: 75.9 +/- 3.6 41 qwen3.7-max@alibaba-us 75.9 42. grok-4.6: 77.7 +/- 3.5 42 grok-4.6 77.7 43. gpt-6-sol: 77.7 +/- 3.6 43 gpt-6-sol 77.7 44. deepseek-v4-pro-0813: 75.4 +/- 3.6 44 deepseek-v4-pro-0813 75.4 45. deepseek-v4-pro-0813@deepinfra: 75.4 +/- 3.6 45 deepseek-v4-pro-0813@deepinfra 75.4 46. gpt-5.4: 73.0 +/- 3.7 46 gpt-5.4 73.0 47. gpt-5.6-terra: 70.0 +/- 3.8 47 gpt-5.6-terra 70.0 48. grok-4.7: 79.1 +/- 3.4 48 grok-4.7 79.1 49. claude-sonnet-4-6: 67.8 +/- 3.9 49 claude-sonnet-4-6 67.8 52. glm-5.2: 69.3 +/- 3.9 52 glm-5.2 69.3 53. glm-5.2@alibaba-intl: 69.3 +/- 3.9 53 glm-5.2@alibaba-intl 69.3 54. glm-5.2@alibaba-us: 69.3 +/- 3.9 54 glm-5.2@alibaba-us 69.3 55. glm-5.2@deepinfra: 69.3 +/- 3.9 55 glm-5.2@deepinfra 69.3 56. claude-sonnet-5-5: 63.1 +/- 3.9 56 claude-sonnet-5-5 63.1 58. claude-sonnet-5: 68.1 +/- 3.9 58 claude-sonnet-5 68.1 59. deepseek-v4-flash-vision-exp: 76.6 +/- 3.5 59 deepseek-v4-flash-vision-exp 76.6 61. glm-5.3-flash: 62.7 +/- 3.9 61 glm-5.3-flash 62.7 62. glm-5.3-flash@deepinfra: 62.7 +/- 3.9 62 glm-5.3-flash@deepinfra 62.7 64. kimi-k2.6: 68.4 +/- 3.9 64 kimi-k2.6 68.4 65. deepseek-v4-pro: 67.2 +/- 3.9 65 deepseek-v4-pro 67.2 66. deepseek-v4-pro-fast@parasail: 67.2 +/- 3.9 66 deepseek-v4-pro-fast@parasail 67.2 67. deepseek-v4-pro@deepinfra: 67.2 +/- 3.9 67 deepseek-v4-pro@deepinfra 67.2 68. deepseek-v4-pro@parasail: 67.2 +/- 3.9 68 deepseek-v4-pro@parasail 67.2 69. deepseek-v4-pro@together: 67.2 +/- 3.9 69 deepseek-v4-pro@together 67.2 72. gemini-3.5-flash-lite: 68.4 +/- 3.8 72 gemini-3.5-flash-lite 68.4 73. gpt-5.2: 72.0 +/- 3.8 73 gpt-5.2 72.0 74. qwen3.8-27b: 74.6 +/- 3.6 74 qwen3.8-27b 74.6 75. qwen3.8-27b@deepinfra: 74.6 +/- 3.6 75 qwen3.8-27b@deepinfra 74.6 76. qwen3.8-27b@parasail: 74.6 +/- 3.6 76 qwen3.8-27b@parasail 74.6 77. inkling: 72.2 +/- 3.7 77 inkling 72.2 78. qwen3.6-plus: 67.8 +/- 3.9 78 qwen3.6-plus 67.8 84. minimax-m3: 66.4 +/- 3.9 84 minimax-m3 66.4 85. minimax-m3-fast@parasail: 66.4 +/- 3.9 85 minimax-m3-fast@parasail 66.4 86. minimax-m3@deepinfra: 66.4 +/- 3.9 86 minimax-m3@deepinfra 66.4 87. minimax-m3@parasail: 66.4 +/- 3.9 87 minimax-m3@parasail 66.4 88. gpt-5.4-mini: 64.9 +/- 3.9 88 gpt-5.4-mini 64.9 89. deepseek-v4-flash: 66.3 +/- 3.9 89 deepseek-v4-flash 66.3 90. deepseek-v4-flash@deepinfra: 66.3 +/- 3.9 90 deepseek-v4-flash@deepinfra 66.3 91. deepseek-v4-flash@parasail: 66.3 +/- 3.9 91 deepseek-v4-flash@parasail 66.3 92. gpt-5.6-luna: 60.2 +/- 4.0 92 gpt-5.6-luna 60.2 93. deepseek-v4-flash-0731: 70.4 +/- 3.8 93 deepseek-v4-flash-0731 70.4 94. deepseek-v4-flash-0731@alibaba-intl: 70.4 +/- 3.8 94 deepseek-v4-flash-0731@alibaba-intl 70.4 95. deepseek-v4-flash-0731@deepinfra: 70.4 +/- 3.8 95 deepseek-v4-flash-0731@deepinfra 70.4 96. deepseek-v4-flash-0731@parasail: 70.4 +/- 3.8 96 deepseek-v4-flash-0731@parasail 70.4 97. gpt-6-luna: 61.8 +/- 4.0 97 gpt-6-luna 61.8 102. claude-haiku-5-5: 63.0 +/- 4.0 102 claude-haiku-5-5 63.0 124. kimi-k2.7-code: 63.2 +/- 3.9 124 kimi-k2.7-code 63.2 125. gpt-5.4-nano: 64.6 +/- 3.9 125 gpt-5.4-nano 64.6 163. qwen3.6-27b: 56.4 +/- 4.1 163 qwen3.6-27b 56.4 164. qwen3.6-27b@deepinfra: 56.4 +/- 4.1 164 qwen3.6-27b@deepinfra 56.4 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Rating only, not on the chart (82) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
mistral-small-24b-instruct-2501Rating 1,255.2 $0.07 gpt-oss-20bRating 1,281.3 $0.08 gemma-3-4b-itRating 1,267.5 $0.08 gpt-oss-120b@deepinfraRating 1,323.7 $0.10 gemma-3-12b-itRating 1,321.0 $0.10 phi-4Rating 1,245.4 $0.11 gemma-3-27b-itRating 1,343.4 $0.13 granite-4.2-8bRating 1,281.2 $0.15 qwen3-32bRating 1,331.0 $0.18 gpt-5-nanoRating 1,326.1 $0.20 llama-4-scout-17b-16e-instructRating 1,299.3 $0.20 llama-3.3-70b-instruct-turbo@deepinfraRating 1,292.7 $0.21 mimo-v2.6-flashRating 1,435.7 $0.22 qwen3.5-flash-2026-02-23Rating 1,381.8 $0.24 gpt-4.1-nanoRating 1,300.3 $0.24 qwen3-235b-a22b-instruct-2507@deepinfraRating 1,414.1 $0.29 qwen3-30b-a3bRating 1,309.0 $0.30 gemma-4-31b-itRating 1,453.1 $0.32 deepseek-v3.2Rating 1,420.8 $0.36 gpt-oss-120bRating 1,323.7 $0.36 gpt-4o-miniRating 1,292.9 $0.36 llama-3.3-70b-instruct-turbo@parasailRating 1,292.7 $0.38 qwen3-235b-a22b-instruct-2507@parasailRating 1,414.1 $0.43 qwen3-30b-a3b-instruct-2507Rating 1,365.8 $0.48 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,408.7 $0.51 qwen3.5-35b-a3b@deepinfraRating 1,386.7 $0.51 qwen3-next-80b-a3b-instruct@deepinfraRating 1,376.5 $0.51 qwen3.5-35b-a3b@parasailRating 1,386.7 $0.52 qwen3-next-80b-a3b-instruct@parasailRating 1,376.5 $0.52 qwen3-235b-a22b-instruct-2507Rating 1,414.1 $0.55 qwen3-next-80b-a3b-instructRating 1,376.5 $0.60 qwen3-next-80b-a3b-thinkingRating 1,358.7 $0.60 inkling-2-small@deepinfraRating 1,409.9 $0.61 mimo-v2.6-proRating 1,472.1 $0.62 inkling-2-smallRating 1,409.9 $0.64 qwen3.7-plusRating 1,446.9 $0.77 qwen3.7-plus@alibaba-usRating 1,446.9 $0.77 beam-1.0-501b-a23bRating 1,433.2 $0.80 gemini-3.1-flash-liteRating 1,409.2 $0.80 qwen-plus-2025-01-25Rating 1,331.5 $0.80 inkling-smallRating 1,396.5 $0.84 qwen3-vl-235b-a22b-instruct@parasailRating 1,408.7 $0.93 qwen3-vl-235b-a22b-instructRating 1,408.7 $0.96 gpt-4.1-miniRating 1,373.1 $0.96 mistral-large-2512Rating 1,402.9 $1.00 qwen3.5-35b-a3bRating 1,386.7 $1.00 gpt-5-miniRating 1,372.3 $1.00 gpt-3.5-turboRating 1,215.9 $1.00 glm-4.7Rating 1,426.2 $1.02 qwen3-235b-a22b-thinking-2507Rating 1,385.6 $1.10 glm-4.6Rating 1,412.8 $1.20 qwen3.5-27bRating 1,400.3 $1.20 qwen3.5-27b@deepinfraRating 1,400.3 $1.25 llama-3.3-70b-instruct-turboRating 1,292.7 $1.25 deepseek-r1-0528Rating 1,397.3 $1.26 qwen3.5-397b-a17b@deepinfraRating 1,434.5 $1.56 gemini-3-flash-previewRating 1,457.4 $1.60 qwen3.5-122b-a10bRating 1,405.7 $1.60 qwen3.5-397b-a17b@parasailRating 1,434.5 $1.84 qwen3.5-397b-a17bRating 1,434.5 $1.92 qwen3-vl-235b-a22b-thinkingRating 1,381.9 $1.92 o4-miniRating 1,368.0 $2.64 o3-miniRating 1,343.6 $2.64 claude-haiku-4-5Rating 1,414.6 $2.80 glm-5.1Rating 1,461.3 $2.88 qwen3.6-max-previewRating 1,450.5 $4.16 gemini-4-hydrogenRating 1,502.9 $4.20 mistral-medium-3-5Rating 1,422.9 $4.20 gpt-4.1Rating 1,403.0 $4.80 o3Rating 1,402.6 $4.80 gpt-5.1Rating 1,426.0 $5.00 gpt-5Rating 1,409.9 $5.00 kimi-k3.1-pro@deepinfraRating 1,487.4 $5.32 gemini-4-argonRating 1,527.6 $5.60 gemini-4-carbonRating 1,525.2 $5.60 kimi-k3.1-proRating 1,487.4 $5.60 kimi-k3.1-pro@togetherRating 1,487.4 $5.60 gpt-4oRating 1,325.5 $6.00 claude-sonnet-4-5Rating 1,462.0 $8.40 gpt-4-turboRating 1,303.3 $20.00 o1Rating 1,407.0 $36.00 gpt-4Rating 1,285.3 $48.00 Creative writing Price against score, Medium 0 25 50 75 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 70.5 +/- 6.5, $0.12
deepseek-v4-flash@deepinfra: 67.3 +/- 6.6, $0.14
deepseek-v4-flash@parasail: 67.3 +/- 6.6, $0.22
deepseek-v4-flash-0731: 70.5 +/- 6.5, $0.22
deepseek-v4-flash-0731@parasail: 70.5 +/- 6.5, $0.22
deepseek-flash@deepinfra: 72.7 +/- 6.3, $0.28
gpt-6-luna: 63.3 +/- 6.8, $0.28
claude-haiku-5-5: 73.6 +/- 6.2, $0.28
glm-5.3-flash: 59.9 +/- 6.9, $0.32
glm-5.3-flash@deepinfra: 59.9 +/- 6.9, $0.32
deepseek-v4-flash: 67.3 +/- 6.6, $0.32
deepseek-v4-flash-vision-exp: 73.8 +/- 6.2, $0.43
gpt-5.6-luna: 69.1 +/- 6.5, $0.64
gpt-5.4-nano: 67.7 +/- 6.6, $0.66
minimax-m3@deepinfra: 63.6 +/- 6.8, $0.66
deepseek-flash: 72.7 +/- 6.3, $0.72
deepseek-flash@alibaba-intl: 72.7 +/- 6.3, $0.72
deepseek-flash@parasail: 72.7 +/- 6.3, $0.72
deepseek-flash@together: 72.7 +/- 6.3, $0.72
minimax-m3: 63.6 +/- 6.8, $0.72
minimax-m3@parasail: 63.6 +/- 6.8, $0.72
qwen3.8-27b@deepinfra: 73.1 +/- 6.3, $0.87
deepseek-v4-flash-0731@alibaba-intl: 70.5 +/- 6.5, $0.88
qwen3.8-27b@parasail: 73.1 +/- 6.3, $1.07
deepseek-flash-fast@parasail: 72.7 +/- 6.3, $1.08
minimax-m3-fast@parasail: 63.6 +/- 6.8, $1.08
glm-5.2@deepinfra: 62.9 +/- 6.8, $1.17
gemini-3.5-flash-lite: 67.5 +/- 6.6, $1.24
glm-5.3@deepinfra: 72.1 +/- 6.3, $1.45
qwen3.6-27b@deepinfra: 60.1 +/- 6.9, $1.54
qwen3.6-plus: 56.9 +/- 7.0, $1.60
qwen3.8-27b: 73.1 +/- 6.3, $1.60
qwen3.6-27b: 60.1 +/- 6.9, $1.92
kimi-k2.6: 66.9 +/- 6.7, $2.00
deepseek-v4-pro-0813@deepinfra: 74.9 +/- 6.1, $2.08
deepseek-v4-pro@deepinfra: 67.8 +/- 6.6, $2.08
gemini-3.8-flash: 82.8 +/- 5.3, $2.10
gemini-3.7-flash: 82.3 +/- 5.4, $2.10
gemini-3.6-flash: 75.6 +/- 6.1, $2.10
gemini-3.5-flash: 75.2 +/- 6.1, $2.10
kimi-k2.7-code: 68.6 +/- 6.6, $2.36
gpt-5.4-mini: 66.6 +/- 6.7, $2.40
inkling: 74.2 +/- 6.2, $2.42
deepseek-v4-pro-0813: 74.9 +/- 6.1, $2.64
deepseek-v4-pro: 67.8 +/- 6.6, $2.64
deepseek-v4-pro@parasail: 67.8 +/- 6.6, $2.64
deepseek-v4-pro@together: 67.8 +/- 6.6, $2.64
glm-5.3: 72.1 +/- 6.3, $2.88
glm-5.3@alibaba-intl: 72.1 +/- 6.3, $2.88
glm-5.3@parasail: 72.1 +/- 6.3, $2.88
glm-5.2: 62.9 +/- 6.8, $2.88
glm-5.2@alibaba-intl: 62.9 +/- 6.8, $2.88
glm-5.2@alibaba-us: 62.9 +/- 6.8, $2.88
deepseek-v4-pro-fast@parasail: 67.8 +/- 6.6, $3.96
qwen3.8-max-0902: 76.5 +/- 6.0, $4.00
glm-5.3-fast@parasail: 72.1 +/- 6.3, $4.32
grok-4.6: 78.3 +/- 5.8, $4.40
grok-4.7: 74.9 +/- 6.1, $4.40
qwen3.7-max: 76.3 +/- 6.0, $5.00
qwen3.7-max@alibaba-us: 76.3 +/- 6.0, $5.00
claude-sonnet-5-5: 69.9 +/- 6.5, $5.60
gpt-6.1-sol: 70.8 +/- 6.4, $5.60
gpt-6-sol: 66.9 +/- 6.7, $5.60
claude-sonnet-5: 62.3 +/- 6.9, $5.60
gemini-3.1-pro-preview: 77.3 +/- 5.9, $6.40
gpt-5.6-terra: 66.8 +/- 6.7, $6.40
gpt-5.2: 65.7 +/- 6.7, $7.00
kimi-k3@deepinfra: 75.4 +/- 6.1, $7.98
gpt-5.4: 79.0 +/- 5.8, $8.00
kimi-k3: 75.4 +/- 6.1, $8.40
kimi-k3@moonshot: 75.4 +/- 6.1, $8.40
claude-sonnet-4-6: 68.9 +/- 6.6, $8.40
claude-opus-5-5: 70.9 +/- 6.4, $11.20
gpt-5.6-sol: 75.9 +/- 6.1, $11.20
claude-opus-4-8: 80.6 +/- 5.6, $14.00
claude-opus-4-7: 69.4 +/- 6.5, $14.00
claude-opus-4-6: 66.1 +/- 6.7, $14.00
claude-opus-4-5: 65.9 +/- 6.7, $14.00
claude-opus-5: 61.3 +/- 6.9, $14.00
gpt-5.5: 72.6 +/- 6.3, $16.00
claude-fable-5-1: 78.1 +/- 5.9, $28.00
claude-fable-5: 71.7 +/- 6.4, $28.00
gpt-6-astra: 74.0 +/- 6.2, $28.00
deepseek-v4-flash-0731@deepinfra Cheap gemini-3.8-flash Best, Balanced Ranking, Medium 4. gemini-3.8-flash: 82.8 +/- 5.3 4 gemini-3.8-flash 82.8 5. gemini-3.7-flash: 82.3 +/- 5.4 5 gemini-3.7-flash 82.3 6. claude-fable-5-1: 78.1 +/- 5.9 6 claude-fable-5-1 78.1 7. gemini-3.1-pro-preview: 77.3 +/- 5.9 7 gemini-3.1-pro-preview 77.3 8. claude-opus-4-8: 80.6 +/- 5.6 8 claude-opus-4-8 80.6 9. claude-opus-5-5: 70.9 +/- 6.4 9 claude-opus-5-5 70.9 10. claude-fable-5: 71.7 +/- 6.4 10 claude-fable-5 71.7 11. qwen3.8-max-0902: 76.5 +/- 6.0 11 qwen3.8-max-0902 76.5 12. gpt-5.6-sol: 75.9 +/- 6.1 12 gpt-5.6-sol 75.9 13. gemini-3.6-flash: 75.6 +/- 6.1 13 gemini-3.6-flash 75.6 14. claude-opus-4-7: 69.4 +/- 6.5 14 claude-opus-4-7 69.4 15. gemini-3.5-flash: 75.2 +/- 6.1 15 gemini-3.5-flash 75.2 17. claude-opus-4-6: 66.1 +/- 6.7 17 claude-opus-4-6 66.1 18. kimi-k3: 75.4 +/- 6.1 18 kimi-k3 75.4 19. kimi-k3@deepinfra: 75.4 +/- 6.1 19 kimi-k3@deepinfra 75.4 20. kimi-k3@moonshot: 75.4 +/- 6.1 20 kimi-k3@moonshot 75.4 22. claude-sonnet-5-5: 69.9 +/- 6.5 22 claude-sonnet-5-5 69.9 26. glm-5.3: 72.1 +/- 6.3 26 glm-5.3 72.1 27. glm-5.3-fast@parasail: 72.1 +/- 6.3 27 glm-5.3-fast@parasail 72.1 28. glm-5.3@alibaba-intl: 72.1 +/- 6.3 28 glm-5.3@alibaba-intl 72.1 29. glm-5.3@deepinfra: 72.1 +/- 6.3 29 glm-5.3@deepinfra 72.1 30. glm-5.3@parasail: 72.1 +/- 6.3 30 glm-5.3@parasail 72.1 31. grok-4.6: 78.3 +/- 5.8 31 grok-4.6 78.3 32. gpt-6-astra: 74.0 +/- 6.2 32 gpt-6-astra 74.0 33. claude-opus-4-5: 65.9 +/- 6.7 33 claude-opus-4-5 65.9 34. claude-opus-5: 61.3 +/- 6.9 34 claude-opus-5 61.3 36. gpt-5.5: 72.6 +/- 6.3 36 gpt-5.5 72.6 37. deepseek-v4-pro-0813: 74.9 +/- 6.1 37 deepseek-v4-pro-0813 74.9 38. deepseek-v4-pro-0813@deepinfra: 74.9 +/- 6.1 38 deepseek-v4-pro-0813@deepinfra 74.9 39. qwen3.7-max: 76.3 +/- 6.0 39 qwen3.7-max 76.3 40. qwen3.7-max@alibaba-us: 76.3 +/- 6.0 40 qwen3.7-max@alibaba-us 76.3 42. gpt-6.1-sol: 70.8 +/- 6.4 42 gpt-6.1-sol 70.8 43. gpt-5.4: 79.0 +/- 5.8 43 gpt-5.4 79.0 44. claude-sonnet-4-6: 68.9 +/- 6.6 44 claude-sonnet-4-6 68.9 45. deepseek-flash: 72.7 +/- 6.3 45 deepseek-flash 72.7 46. deepseek-flash-fast@parasail: 72.7 +/- 6.3 46 deepseek-flash-fast@parasail 72.7 47. deepseek-flash@alibaba-intl: 72.7 +/- 6.3 47 deepseek-flash@alibaba-intl 72.7 48. deepseek-flash@deepinfra: 72.7 +/- 6.3 48 deepseek-flash@deepinfra 72.7 49. deepseek-flash@parasail: 72.7 +/- 6.3 49 deepseek-flash@parasail 72.7 50. deepseek-flash@together: 72.7 +/- 6.3 50 deepseek-flash@together 72.7 51. deepseek-v4-pro: 67.8 +/- 6.6 51 deepseek-v4-pro 67.8 52. deepseek-v4-pro-fast@parasail: 67.8 +/- 6.6 52 deepseek-v4-pro-fast@parasail 67.8 53. deepseek-v4-pro@deepinfra: 67.8 +/- 6.6 53 deepseek-v4-pro@deepinfra 67.8 54. deepseek-v4-pro@parasail: 67.8 +/- 6.6 54 deepseek-v4-pro@parasail 67.8 55. deepseek-v4-pro@together: 67.8 +/- 6.6 55 deepseek-v4-pro@together 67.8 57. glm-5.2: 62.9 +/- 6.8 57 glm-5.2 62.9 58. glm-5.2@alibaba-intl: 62.9 +/- 6.8 58 glm-5.2@alibaba-intl 62.9 59. glm-5.2@alibaba-us: 62.9 +/- 6.8 59 glm-5.2@alibaba-us 62.9 60. glm-5.2@deepinfra: 62.9 +/- 6.8 60 glm-5.2@deepinfra 62.9 61. grok-4.7: 74.9 +/- 6.1 61 grok-4.7 74.9 62. gemini-3.5-flash-lite: 67.5 +/- 6.6 62 gemini-3.5-flash-lite 67.5 63. kimi-k2.6: 66.9 +/- 6.7 63 kimi-k2.6 66.9 64. gpt-5.6-terra: 66.8 +/- 6.7 64 gpt-5.6-terra 66.8 67. gpt-6-sol: 66.9 +/- 6.7 67 gpt-6-sol 66.9 68. claude-sonnet-5: 62.3 +/- 6.9 68 claude-sonnet-5 62.3 69. glm-5.3-flash: 59.9 +/- 6.9 69 glm-5.3-flash 59.9 70. glm-5.3-flash@deepinfra: 59.9 +/- 6.9 70 glm-5.3-flash@deepinfra 59.9 71. gpt-5.6-luna: 69.1 +/- 6.5 71 gpt-5.6-luna 69.1 72. deepseek-v4-flash-vision-exp: 73.8 +/- 6.2 72 deepseek-v4-flash-vision-exp 73.8 74. deepseek-v4-flash: 67.3 +/- 6.6 74 deepseek-v4-flash 67.3 75. deepseek-v4-flash@deepinfra: 67.3 +/- 6.6 75 deepseek-v4-flash@deepinfra 67.3 76. deepseek-v4-flash@parasail: 67.3 +/- 6.6 76 deepseek-v4-flash@parasail 67.3 78. gpt-6-luna: 63.3 +/- 6.8 78 gpt-6-luna 63.3 79. gpt-5.4-mini: 66.6 +/- 6.7 79 gpt-5.4-mini 66.6 80. inkling: 74.2 +/- 6.2 80 inkling 74.2 81. deepseek-v4-flash-0731: 70.5 +/- 6.5 81 deepseek-v4-flash-0731 70.5 82. deepseek-v4-flash-0731@alibaba-intl: 70.5 +/- 6.5 82 deepseek-v4-flash-0731@alibaba-intl 70.5 83. deepseek-v4-flash-0731@deepinfra: 70.5 +/- 6.5 83 deepseek-v4-flash-0731@deepinfra 70.5 84. deepseek-v4-flash-0731@parasail: 70.5 +/- 6.5 84 deepseek-v4-flash-0731@parasail 70.5 91. minimax-m3: 63.6 +/- 6.8 91 minimax-m3 63.6 92. minimax-m3-fast@parasail: 63.6 +/- 6.8 92 minimax-m3-fast@parasail 63.6 93. minimax-m3@deepinfra: 63.6 +/- 6.8 93 minimax-m3@deepinfra 63.6 94. minimax-m3@parasail: 63.6 +/- 6.8 94 minimax-m3@parasail 63.6 95. kimi-k2.7-code: 68.6 +/- 6.6 95 kimi-k2.7-code 68.6 97. qwen3.6-plus: 56.9 +/- 7.0 97 qwen3.6-plus 56.9 99. gpt-5.2: 65.7 +/- 6.7 99 gpt-5.2 65.7 101. claude-haiku-5-5: 73.6 +/- 6.2 101 claude-haiku-5-5 73.6 102. qwen3.8-27b: 73.1 +/- 6.3 102 qwen3.8-27b 73.1 103. qwen3.8-27b@deepinfra: 73.1 +/- 6.3 103 qwen3.8-27b@deepinfra 73.1 104. qwen3.8-27b@parasail: 73.1 +/- 6.3 104 qwen3.8-27b@parasail 73.1 117. qwen3.6-27b: 60.1 +/- 6.9 117 qwen3.6-27b 60.1 118. qwen3.6-27b@deepinfra: 60.1 +/- 6.9 118 qwen3.6-27b@deepinfra 60.1 119. gpt-5.4-nano: 67.7 +/- 6.6 119 gpt-5.4-nano 67.7 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Rating only, not on the chart (82) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
mistral-small-24b-instruct-2501Rating 1,226.6 $0.07 gemma-3-4b-itRating 1,274.4 $0.08 gpt-oss-20bRating 1,241.0 $0.08 gpt-oss-120b@deepinfraRating 1,278.6 $0.10 gemma-3-12b-itRating 1,332.4 $0.10 phi-4Rating 1,210.0 $0.11 gemma-3-27b-itRating 1,348.1 $0.13 granite-4.2-8bRating 1,199.1 $0.15 qwen3-32bRating 1,303.7 $0.18 llama-4-scout-17b-16e-instructRating 1,288.9 $0.20 gpt-5-nanoRating 1,246.0 $0.20 llama-3.3-70b-instruct-turbo@deepinfraRating 1,285.4 $0.21 mimo-v2.6-flashRating 1,391.8 $0.22 qwen3.5-flash-2026-02-23Rating 1,337.7 $0.24 gpt-4.1-nanoRating 1,307.0 $0.24 qwen3-235b-a22b-instruct-2507@deepinfraRating 1,378.2 $0.29 qwen3-30b-a3bRating 1,283.6 $0.30 gemma-4-31b-itRating 1,418.4 $0.32 deepseek-v3.2Rating 1,400.6 $0.36 gpt-4o-miniRating 1,293.8 $0.36 gpt-oss-120bRating 1,278.6 $0.36 llama-3.3-70b-instruct-turbo@parasailRating 1,285.4 $0.38 qwen3-235b-a22b-instruct-2507@parasailRating 1,378.2 $0.43 qwen3-30b-a3b-instruct-2507Rating 1,318.8 $0.48 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,355.4 $0.51 qwen3.5-35b-a3b@deepinfraRating 1,342.2 $0.51 qwen3-next-80b-a3b-instruct@deepinfraRating 1,313.1 $0.51 qwen3.5-35b-a3b@parasailRating 1,342.2 $0.52 qwen3-next-80b-a3b-instruct@parasailRating 1,313.1 $0.52 qwen3-235b-a22b-instruct-2507Rating 1,378.2 $0.55 qwen3-next-80b-a3b-thinkingRating 1,326.1 $0.60 qwen3-next-80b-a3b-instructRating 1,313.1 $0.60 inkling-2-small@deepinfraRating 1,345.0 $0.61 mimo-v2.6-proRating 1,448.0 $0.62 inkling-2-smallRating 1,345.0 $0.64 qwen3.7-plusRating 1,431.0 $0.77 qwen3.7-plus@alibaba-usRating 1,431.0 $0.77 gemini-3.1-flash-liteRating 1,414.6 $0.80 beam-1.0-501b-a23bRating 1,387.8 $0.80 qwen-plus-2025-01-25Rating 1,316.3 $0.80 inkling-smallRating 1,319.0 $0.84 qwen3-vl-235b-a22b-instruct@parasailRating 1,355.4 $0.93 qwen3-vl-235b-a22b-instructRating 1,355.4 $0.96 gpt-4.1-miniRating 1,348.8 $0.96 mistral-large-2512Rating 1,374.5 $1.00 qwen3.5-35b-a3bRating 1,342.2 $1.00 gpt-5-miniRating 1,322.9 $1.00 gpt-3.5-turboRating 1,190.9 $1.00 glm-4.7Rating 1,405.5 $1.02 qwen3-235b-a22b-thinking-2507Rating 1,369.5 $1.10 glm-4.6Rating 1,401.1 $1.20 qwen3.5-27bRating 1,357.6 $1.20 qwen3.5-27b@deepinfraRating 1,357.6 $1.25 llama-3.3-70b-instruct-turboRating 1,285.4 $1.25 deepseek-r1-0528Rating 1,388.5 $1.26 qwen3.5-397b-a17b@deepinfraRating 1,406.1 $1.56 gemini-3-flash-previewRating 1,456.3 $1.60 qwen3.5-122b-a10bRating 1,365.4 $1.60 qwen3.5-397b-a17b@parasailRating 1,406.1 $1.84 qwen3.5-397b-a17bRating 1,406.1 $1.92 qwen3-vl-235b-a22b-thinkingRating 1,338.1 $1.92 o4-miniRating 1,337.2 $2.64 o3-miniRating 1,301.0 $2.64 claude-haiku-4-5Rating 1,386.9 $2.80 glm-5.1Rating 1,446.2 $2.88 qwen3.6-max-previewRating 1,438.2 $4.16 gemini-4-hydrogenRating 1,491.8 $4.20 mistral-medium-3-5Rating 1,399.4 $4.20 gpt-4.1Rating 1,402.9 $4.80 o3Rating 1,383.0 $4.80 gpt-5.1Rating 1,407.6 $5.00 gpt-5Rating 1,372.5 $5.00 kimi-k3.1-pro@deepinfraRating 1,453.1 $5.32 gemini-4-argonRating 1,522.6 $5.60 gemini-4-carbonRating 1,516.1 $5.60 kimi-k3.1-proRating 1,453.1 $5.60 kimi-k3.1-pro@togetherRating 1,453.1 $5.60 gpt-4oRating 1,337.6 $6.00 claude-sonnet-4-5Rating 1,454.6 $8.40 gpt-4-turboRating 1,323.2 $20.00 o1Rating 1,381.3 $36.00 gpt-4Rating 1,266.6 $48.00 Summarizing Price against score, Medium 0 25 50 75 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 71.5 +/- 6.4, $0.18
deepseek-v4-flash@deepinfra: 68.9 +/- 6.6, $0.25
claude-haiku-5-5: 67.0 +/- 6.7, $0.34
gpt-6-luna: 55.3 +/- 7.0, $0.34
deepseek-v4-flash-0731: 71.5 +/- 6.4, $0.39
deepseek-v4-flash-0731@parasail: 71.5 +/- 6.4, $0.39
deepseek-v4-flash@parasail: 68.9 +/- 6.6, $0.39
deepseek-flash@deepinfra: 69.7 +/- 6.5, $0.42
glm-5.3-flash: 56.5 +/- 7.0, $0.46
glm-5.3-flash@deepinfra: 56.5 +/- 7.0, $0.46
deepseek-v4-flash: 68.9 +/- 6.6, $0.56
deepseek-v4-flash-vision-exp: 72.1 +/- 6.3, $0.65
gpt-5.6-luna: 64.9 +/- 6.8, $0.72
gpt-5.4-nano: 71.8 +/- 6.4, $0.73
qwen3.8-27b@deepinfra: 75.9 +/- 6.1, $0.74
minimax-m3@deepinfra: 53.9 +/- 7.1, $0.89
deepseek-flash: 69.7 +/- 6.5, $0.96
deepseek-flash@alibaba-intl: 69.7 +/- 6.5, $0.96
deepseek-flash@parasail: 69.7 +/- 6.5, $0.96
deepseek-flash@together: 69.7 +/- 6.5, $0.96
minimax-m3: 53.9 +/- 7.1, $0.96
minimax-m3@parasail: 53.9 +/- 7.1, $0.96
qwen3.8-27b@parasail: 75.9 +/- 6.1, $1.02
gemini-3.5-flash-lite: 68.5 +/- 6.6, $1.22
deepseek-v4-flash-0731@alibaba-intl: 71.5 +/- 6.4, $1.32
qwen3.6-27b@deepinfra: 52.6 +/- 7.1, $1.41
deepseek-flash-fast@parasail: 69.7 +/- 6.5, $1.44
minimax-m3-fast@parasail: 53.9 +/- 7.1, $1.44
glm-5.2@deepinfra: 68.4 +/- 6.6, $1.71
qwen3.8-27b: 75.9 +/- 6.1, $1.80
qwen3.6-plus: 64.3 +/- 6.8, $1.80
glm-5.3@deepinfra: 74.6 +/- 6.2, $1.85
qwen3.6-27b: 52.6 +/- 7.1, $2.16
kimi-k2.6: 67.5 +/- 6.6, $2.50
gemini-3.8-flash: 87.5 +/- 4.7, $2.55
gemini-3.7-flash: 84.8 +/- 5.1, $2.55
gemini-3.5-flash: 82.6 +/- 5.4, $2.55
gemini-3.6-flash: 75.9 +/- 6.1, $2.55
gpt-5.4-mini: 59.6 +/- 6.9, $2.70
kimi-k2.7-code: 58.3 +/- 7.0, $3.08
inkling: 74.9 +/- 6.1, $3.21
deepseek-v4-pro@deepinfra: 69.5 +/- 6.5, $3.64
deepseek-v4-pro-0813@deepinfra: 65.7 +/- 6.7, $3.64
deepseek-v4-pro: 69.5 +/- 6.5, $3.96
deepseek-v4-pro@parasail: 69.5 +/- 6.5, $3.96
deepseek-v4-pro@together: 69.5 +/- 6.5, $3.96
deepseek-v4-pro-0813: 65.7 +/- 6.7, $3.96
glm-5.3: 74.6 +/- 6.2, $4.24
glm-5.3@alibaba-intl: 74.6 +/- 6.2, $4.24
glm-5.3@parasail: 74.6 +/- 6.2, $4.24
glm-5.2: 68.4 +/- 6.6, $4.24
glm-5.2@alibaba-intl: 68.4 +/- 6.6, $4.24
glm-5.2@alibaba-us: 68.4 +/- 6.6, $4.24
deepseek-v4-pro-fast@parasail: 69.5 +/- 6.5, $5.94
qwen3.8-max-0902: 79.2 +/- 5.8, $6.00
glm-5.3-fast@parasail: 74.6 +/- 6.2, $6.36
grok-4.7: 78.2 +/- 5.8, $6.60
grok-4.6: 68.1 +/- 6.6, $6.60
gpt-6.1-sol: 81.3 +/- 5.5, $6.80
claude-sonnet-5: 71.2 +/- 6.4, $6.80
gpt-6-sol: 68.0 +/- 6.6, $6.80
claude-sonnet-5-5: 58.2 +/- 7.0, $6.80
gpt-5.2: 63.2 +/- 6.8, $7.00
gemini-3.1-pro-preview: 88.2 +/- 4.6, $7.20
gpt-5.6-terra: 73.2 +/- 6.3, $7.20
qwen3.7-max: 83.8 +/- 5.2, $7.50
qwen3.7-max@alibaba-us: 83.8 +/- 5.2, $7.50
gpt-5.4: 68.4 +/- 6.6, $9.00
kimi-k3@deepinfra: 69.4 +/- 6.5, $9.69
kimi-k3: 69.4 +/- 6.5, $10.20
kimi-k3@moonshot: 69.4 +/- 6.5, $10.20
claude-sonnet-4-6: 61.7 +/- 6.9, $10.20
gpt-5.6-sol: 69.4 +/- 6.5, $13.60
claude-opus-5-5: 69.1 +/- 6.5, $13.60
claude-opus-4-8: 77.7 +/- 5.9, $17.00
claude-opus-5: 66.4 +/- 6.7, $17.00
claude-opus-4-7: 66.3 +/- 6.7, $17.00
claude-opus-4-6: 65.2 +/- 6.7, $17.00
claude-opus-4-5: 63.7 +/- 6.8, $17.00
gpt-5.5: 72.3 +/- 6.3, $18.00
claude-fable-5: 81.7 +/- 5.5, $34.00
gpt-6-astra: 79.6 +/- 5.7, $34.00
claude-fable-5-1: 72.2 +/- 6.3, $34.00
deepseek-v4-flash-vision-exp Cheap qwen3.8-27b@deepinfra Balanced gemini-3.8-flash Best Ranking, Medium 1. gemini-3.1-pro-preview: 88.2 +/- 4.6 1 gemini-3.1-pro-preview 88.2 2. gemini-3.8-flash: 87.5 +/- 4.7 2 gemini-3.8-flash 87.5 3. gemini-3.7-flash: 84.8 +/- 5.1 3 gemini-3.7-flash 84.8 4. qwen3.7-max: 83.8 +/- 5.2 4 qwen3.7-max 83.8 5. qwen3.7-max@alibaba-us: 83.8 +/- 5.2 5 qwen3.7-max@alibaba-us 83.8 6. gemini-3.5-flash: 82.6 +/- 5.4 6 gemini-3.5-flash 82.6 7. claude-fable-5: 81.7 +/- 5.5 7 claude-fable-5 81.7 8. gpt-6.1-sol: 81.3 +/- 5.5 8 gpt-6.1-sol 81.3 9. gpt-6-astra: 79.6 +/- 5.7 9 gpt-6-astra 79.6 10. qwen3.8-max-0902: 79.2 +/- 5.8 10 qwen3.8-max-0902 79.2 11. grok-4.7: 78.2 +/- 5.8 11 grok-4.7 78.2 12. claude-opus-4-8: 77.7 +/- 5.9 12 claude-opus-4-8 77.7 13. qwen3.8-27b: 75.9 +/- 6.1 13 qwen3.8-27b 75.9 14. qwen3.8-27b@deepinfra: 75.9 +/- 6.1 14 qwen3.8-27b@deepinfra 75.9 15. qwen3.8-27b@parasail: 75.9 +/- 6.1 15 qwen3.8-27b@parasail 75.9 16. gemini-3.6-flash: 75.9 +/- 6.1 16 gemini-3.6-flash 75.9 17. inkling: 74.9 +/- 6.1 17 inkling 74.9 18. glm-5.3: 74.6 +/- 6.2 18 glm-5.3 74.6 19. glm-5.3-fast@parasail: 74.6 +/- 6.2 19 glm-5.3-fast@parasail 74.6 20. glm-5.3@alibaba-intl: 74.6 +/- 6.2 20 glm-5.3@alibaba-intl 74.6 21. glm-5.3@deepinfra: 74.6 +/- 6.2 21 glm-5.3@deepinfra 74.6 22. glm-5.3@parasail: 74.6 +/- 6.2 22 glm-5.3@parasail 74.6 23. gpt-5.6-terra: 73.2 +/- 6.3 23 gpt-5.6-terra 73.2 24. gpt-5.5: 72.3 +/- 6.3 24 gpt-5.5 72.3 25. claude-fable-5-1: 72.2 +/- 6.3 25 claude-fable-5-1 72.2 26. deepseek-v4-flash-vision-exp: 72.1 +/- 6.3 26 deepseek-v4-flash-vision-exp 72.1 27. gpt-5.4-nano: 71.8 +/- 6.4 27 gpt-5.4-nano 71.8 28. deepseek-v4-flash-0731: 71.5 +/- 6.4 28 deepseek-v4-flash-0731 71.5 29. deepseek-v4-flash-0731@alibaba-intl: 71.5 +/- 6.4 29 deepseek-v4-flash-0731@alibaba-intl 71.5 30. deepseek-v4-flash-0731@deepinfra: 71.5 +/- 6.4 30 deepseek-v4-flash-0731@deepinfra 71.5 31. deepseek-v4-flash-0731@parasail: 71.5 +/- 6.4 31 deepseek-v4-flash-0731@parasail 71.5 32. claude-sonnet-5: 71.2 +/- 6.4 32 claude-sonnet-5 71.2 33. deepseek-flash: 69.7 +/- 6.5 33 deepseek-flash 69.7 34. deepseek-flash-fast@parasail: 69.7 +/- 6.5 34 deepseek-flash-fast@parasail 69.7 35. deepseek-flash@alibaba-intl: 69.7 +/- 6.5 35 deepseek-flash@alibaba-intl 69.7 36. deepseek-flash@deepinfra: 69.7 +/- 6.5 36 deepseek-flash@deepinfra 69.7 37. deepseek-flash@parasail: 69.7 +/- 6.5 37 deepseek-flash@parasail 69.7 38. deepseek-flash@together: 69.7 +/- 6.5 38 deepseek-flash@together 69.7 39. deepseek-v4-pro: 69.5 +/- 6.5 39 deepseek-v4-pro 69.5 40. deepseek-v4-pro-fast@parasail: 69.5 +/- 6.5 40 deepseek-v4-pro-fast@parasail 69.5 41. deepseek-v4-pro@deepinfra: 69.5 +/- 6.5 41 deepseek-v4-pro@deepinfra 69.5 42. deepseek-v4-pro@parasail: 69.5 +/- 6.5 42 deepseek-v4-pro@parasail 69.5 43. deepseek-v4-pro@together: 69.5 +/- 6.5 43 deepseek-v4-pro@together 69.5 44. gpt-5.6-sol: 69.4 +/- 6.5 44 gpt-5.6-sol 69.4 45. kimi-k3: 69.4 +/- 6.5 45 kimi-k3 69.4 46. kimi-k3@deepinfra: 69.4 +/- 6.5 46 kimi-k3@deepinfra 69.4 47. kimi-k3@moonshot: 69.4 +/- 6.5 47 kimi-k3@moonshot 69.4 48. claude-opus-5-5: 69.1 +/- 6.5 48 claude-opus-5-5 69.1 49. deepseek-v4-flash: 68.9 +/- 6.6 49 deepseek-v4-flash 68.9 50. deepseek-v4-flash@deepinfra: 68.9 +/- 6.6 50 deepseek-v4-flash@deepinfra 68.9 51. deepseek-v4-flash@parasail: 68.9 +/- 6.6 51 deepseek-v4-flash@parasail 68.9 52. gemini-3.5-flash-lite: 68.5 +/- 6.6 52 gemini-3.5-flash-lite 68.5 53. gpt-5.4: 68.4 +/- 6.6 53 gpt-5.4 68.4 54. glm-5.2: 68.4 +/- 6.6 54 glm-5.2 68.4 55. glm-5.2@alibaba-intl: 68.4 +/- 6.6 55 glm-5.2@alibaba-intl 68.4 56. glm-5.2@alibaba-us: 68.4 +/- 6.6 56 glm-5.2@alibaba-us 68.4 57. glm-5.2@deepinfra: 68.4 +/- 6.6 57 glm-5.2@deepinfra 68.4 58. grok-4.6: 68.1 +/- 6.6 58 grok-4.6 68.1 59. gpt-6-sol: 68.0 +/- 6.6 59 gpt-6-sol 68.0 60. kimi-k2.6: 67.5 +/- 6.6 60 kimi-k2.6 67.5 61. claude-haiku-5-5: 67.0 +/- 6.7 61 claude-haiku-5-5 67.0 62. claude-opus-5: 66.4 +/- 6.7 62 claude-opus-5 66.4 63. claude-opus-4-7: 66.3 +/- 6.7 63 claude-opus-4-7 66.3 64. deepseek-v4-pro-0813: 65.7 +/- 6.7 64 deepseek-v4-pro-0813 65.7 65. deepseek-v4-pro-0813@deepinfra: 65.7 +/- 6.7 65 deepseek-v4-pro-0813@deepinfra 65.7 66. claude-opus-4-6: 65.2 +/- 6.7 66 claude-opus-4-6 65.2 67. gpt-5.6-luna: 64.9 +/- 6.8 67 gpt-5.6-luna 64.9 68. qwen3.6-plus: 64.3 +/- 6.8 68 qwen3.6-plus 64.3 69. claude-opus-4-5: 63.7 +/- 6.8 69 claude-opus-4-5 63.7 70. gpt-5.2: 63.2 +/- 6.8 70 gpt-5.2 63.2 71. claude-sonnet-4-6: 61.7 +/- 6.9 71 claude-sonnet-4-6 61.7 72. gpt-5.4-mini: 59.6 +/- 6.9 72 gpt-5.4-mini 59.6 73. kimi-k2.7-code: 58.3 +/- 7.0 73 kimi-k2.7-code 58.3 74. claude-sonnet-5-5: 58.2 +/- 7.0 74 claude-sonnet-5-5 58.2 75. glm-5.3-flash: 56.5 +/- 7.0 75 glm-5.3-flash 56.5 76. glm-5.3-flash@deepinfra: 56.5 +/- 7.0 76 glm-5.3-flash@deepinfra 56.5 77. gpt-6-luna: 55.3 +/- 7.0 77 gpt-6-luna 55.3 78. minimax-m3: 53.9 +/- 7.1 78 minimax-m3 53.9 79. minimax-m3-fast@parasail: 53.9 +/- 7.1 79 minimax-m3-fast@parasail 53.9 80. minimax-m3@deepinfra: 53.9 +/- 7.1 80 minimax-m3@deepinfra 53.9 81. minimax-m3@parasail: 53.9 +/- 7.1 81 minimax-m3@parasail 53.9 82. qwen3.6-27b: 52.6 +/- 7.1 82 qwen3.6-27b 52.6 83. qwen3.6-27b@deepinfra: 52.6 +/- 7.1 83 qwen3.6-27b@deepinfra 52.6 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Pulling out data Price against score, Medium 50 60 70 80 90 100 $0.1 $1 $10
deepseek-v4.1-flash-1017@deepinfra: 89.8 +/- 1.7, $0.14
gpt-6-luna: 86.7 +/- 1.9, $0.18
claude-haiku-5-5: 75.6 +/- 2.3, $0.18
glm-5.3-flash: 86.3 +/- 2.2, $0.22
glm-5.3-flash@deepinfra: 86.3 +/- 2.2, $0.22
gemma-4-31b-it: 76.7 +/- 2.1, $0.24
minimax-m3.1-flash: 84.7 +/- 2.1, $0.24
minimax-m3.1-flash@parasail: 84.7 +/- 2.1, $0.24
mimo-v2.6-pro: 85.2 +/- 2.1, $0.47
deepseek-v4.1-flash-1017: 89.8 +/- 1.7, $0.48
deepseek-v4.1-flash-1017@parasail: 89.8 +/- 1.7, $0.48
qwen3.8-27b@deepinfra: 86.4 +/- 2.2, $0.50
qwen3.8-27b@parasail: 86.4 +/- 2.2, $0.63
mistral-large-4-0: 88.3 +/- 1.8, $0.98
qwen3.8-27b: 86.4 +/- 2.2, $1.00
gemini-3.8-flash: 75.1 +/- 2.4, $1.35
inkling: 86.4 +/- 1.9, $1.61
gpt-6.1-terra: 91.2 +/- 1.6, $1.80
gemini-4-hydrogen: 90.6 +/- 1.8, $2.70
qwen3.8-max-0902: 88.2 +/- 1.9, $2.80
grok-4.6: 86.7 +/- 2.0, $3.08
grok-4.7: 88.5 +/- 1.8, $3.08
kimi-k3.1-pro@deepinfra: 88.6 +/- 1.9, $3.42
gemini-4-carbon: 90.3 +/- 2.2, $3.60
gemini-4-argon: 89.7 +/- 2.2, $3.60
gpt-6.1-sol: 91.4 +/- 1.6, $3.60
kimi-k3.1-pro: 88.6 +/- 1.9, $3.60
kimi-k3.1-pro@together: 88.6 +/- 1.9, $3.60
claude-sonnet-5-5: 78.8 +/- 2.2, $3.60
gpt-5.6-terra: 89.7 +/- 1.7, $4.00
kimi-k3@deepinfra: 88.4 +/- 1.9, $5.13
kimi-k3: 88.4 +/- 1.9, $5.40
kimi-k3@moonshot: 88.4 +/- 1.9, $5.40
claude-opus-5-5: 89.2 +/- 1.9, $7.20
claude-fable-5-2: 89.8 +/- 1.9, $14.40
claude-fable-5-1: 87.2 +/- 2.3, $18.00
gpt-6-astra: 91.5 +/- 1.6, $18.00
deepseek-v4.1-flash-1017@deepinfra Cheap glm-5.3-flash Balanced gemini-4-hydrogen Best Ranking, Medium 3. gemini-4-carbon: 90.3 +/- 2.2 3 gemini-4-carbon 90.3 4. gemini-4-argon: 89.7 +/- 2.2 4 gemini-4-argon 89.7 7. claude-opus-5-5: 89.2 +/- 1.9 7 claude-opus-5-5 89.2 8. gemini-4-hydrogen: 90.6 +/- 1.8 8 gemini-4-hydrogen 90.6 10. claude-fable-5-1: 87.2 +/- 2.3 10 claude-fable-5-1 87.2 12. kimi-k3: 88.4 +/- 1.9 12 kimi-k3 88.4 13. kimi-k3@deepinfra: 88.4 +/- 1.9 13 kimi-k3@deepinfra 88.4 14. kimi-k3@moonshot: 88.4 +/- 1.9 14 kimi-k3@moonshot 88.4 15. gpt-6.1-sol: 91.4 +/- 1.6 15 gpt-6.1-sol 91.4 16. kimi-k3.1-pro: 88.6 +/- 1.9 16 kimi-k3.1-pro 88.6 17. kimi-k3.1-pro@deepinfra: 88.6 +/- 1.9 17 kimi-k3.1-pro@deepinfra 88.6 18. kimi-k3.1-pro@together: 88.6 +/- 1.9 18 kimi-k3.1-pro@together 88.6 26. gpt-6-astra: 91.5 +/- 1.6 26 gpt-6-astra 91.5 27. qwen3.8-max-0902: 88.2 +/- 1.9 27 qwen3.8-max-0902 88.2 34. gemini-3.8-flash: 75.1 +/- 2.4 34 gemini-3.8-flash 75.1 39. claude-sonnet-5-5: 78.8 +/- 2.2 39 claude-sonnet-5-5 78.8 40. glm-5.3-flash: 86.3 +/- 2.2 40 glm-5.3-flash 86.3 41. glm-5.3-flash@deepinfra: 86.3 +/- 2.2 41 glm-5.3-flash@deepinfra 86.3 46. mimo-v2.6-pro: 85.2 +/- 2.1 46 mimo-v2.6-pro 85.2 47. gpt-5.6-terra: 89.7 +/- 1.7 47 gpt-5.6-terra 89.7 50. gpt-6.1-terra: 91.2 +/- 1.6 50 gpt-6.1-terra 91.2 58. deepseek-v4.1-flash-1017: 89.8 +/- 1.7 58 deepseek-v4.1-flash-1017 89.8 59. deepseek-v4.1-flash-1017@deepinfra: 89.8 +/- 1.7 59 deepseek-v4.1-flash-1017@deepinfra 89.8 60. deepseek-v4.1-flash-1017@parasail: 89.8 +/- 1.7 60 deepseek-v4.1-flash-1017@parasail 89.8 68. grok-4.6: 86.7 +/- 2.0 68 grok-4.6 86.7 70. claude-fable-5-2: 89.8 +/- 1.9 70 claude-fable-5-2 89.8 72. gemma-4-31b-it: 76.7 +/- 2.1 72 gemma-4-31b-it 76.7 73. mistral-large-4-0: 88.3 +/- 1.8 73 mistral-large-4-0 88.3 78. grok-4.7: 88.5 +/- 1.8 78 grok-4.7 88.5 79. gpt-6-luna: 86.7 +/- 1.9 79 gpt-6-luna 86.7 86. inkling: 86.4 +/- 1.9 86 inkling 86.4 98. qwen3.8-27b: 86.4 +/- 2.2 98 qwen3.8-27b 86.4 99. qwen3.8-27b@deepinfra: 86.4 +/- 2.2 99 qwen3.8-27b@deepinfra 86.4 100. qwen3.8-27b@parasail: 86.4 +/- 2.2 100 qwen3.8-27b@parasail 86.4 102. claude-haiku-5-5: 75.6 +/- 2.3 102 claude-haiku-5-5 75.6 104. minimax-m3.1-flash: 84.7 +/- 2.1 104 minimax-m3.1-flash 84.7 105. minimax-m3.1-flash@parasail: 84.7 +/- 2.1 105 minimax-m3.1-flash@parasail 84.7 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Rating only, not on the chart (128) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
gpt-oss-20bRating 1,281.3 $0.05 mistral-small-24b-instruct-2501Rating 1,255.2 $0.06 gemma-3-4b-itRating 1,267.5 $0.06 gpt-oss-120b@deepinfraRating 1,323.7 $0.06 gemma-3-12b-itRating 1,321.0 $0.07 phi-4Rating 1,245.4 $0.08 gemma-3-27b-itRating 1,343.4 $0.10 granite-4.2-8bRating 1,281.2 $0.10 deepseek-v4-flash@deepinfraRating 1,428.0 $0.11 qwen3-32bRating 1,331.0 $0.12 gpt-5-nanoRating 1,326.1 $0.12 llama-4-scout-17b-16e-instructRating 1,299.3 $0.14 llama-3.3-70b-instruct-turbo@deepinfraRating 1,292.7 $0.14 qwen3.5-flash-2026-02-23Rating 1,381.8 $0.16 gpt-4.1-nanoRating 1,300.3 $0.16 mimo-v2.6-flashRating 1,435.7 $0.17 deepseek-v4-flash@parasailRating 1,428.0 $0.17 qwen3-235b-a22b-instruct-2507@deepinfraRating 1,414.1 $0.18 deepseek-flash@deepinfraRating 1,475.8 $0.20 qwen3-30b-a3bRating 1,309.0 $0.20 deepseek-v4-flashRating 1,428.0 $0.24 gpt-oss-120bRating 1,323.7 $0.24 gpt-4o-miniRating 1,292.9 $0.24 qwen3-235b-a22b-instruct-2507@parasailRating 1,414.1 $0.27 llama-3.3-70b-instruct-turbo@parasailRating 1,292.7 $0.28 deepseek-v3.2Rating 1,420.8 $0.28 qwen3-next-80b-a3b-instruct@deepinfraRating 1,376.5 $0.29 qwen3-next-80b-a3b-instruct@parasailRating 1,376.5 $0.30 qwen3.5-35b-a3b@deepinfraRating 1,386.7 $0.31 qwen3.5-35b-a3b@parasailRating 1,386.7 $0.32 qwen3-30b-a3b-instruct-2507Rating 1,365.8 $0.32 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,408.7 $0.34 qwen3-next-80b-a3b-instructRating 1,376.5 $0.36 qwen3-next-80b-a3b-thinkingRating 1,358.7 $0.36 qwen3-235b-a22b-instruct-2507Rating 1,414.1 $0.37 gpt-5.6-lunaRating 1,443.1 $0.40 gpt-5.4-nanoRating 1,386.3 $0.41 inkling-2-small@deepinfraRating 1,409.9 $0.42 inkling-2-smallRating 1,409.9 $0.44 minimax-m3@deepinfraRating 1,428.2 $0.44 deepseek-flashRating 1,475.8 $0.48 deepseek-flash@alibaba-intlRating 1,475.8 $0.48 deepseek-flash@parasailRating 1,475.8 $0.48 deepseek-flash@togetherRating 1,475.8 $0.48 minimax-m3Rating 1,428.2 $0.48 minimax-m3@parasailRating 1,428.2 $0.48 gemini-3.1-flash-liteRating 1,409.2 $0.50 qwen3.7-plusRating 1,446.9 $0.51 qwen3.7-plus@alibaba-usRating 1,446.9 $0.51 qwen3-vl-235b-a22b-instruct@parasailRating 1,408.7 $0.55 beam-1.0-501b-a23bRating 1,433.2 $0.56 qwen-plus-2025-01-25Rating 1,331.5 $0.56 inkling-smallRating 1,396.5 $0.60 qwen3.5-35b-a3bRating 1,386.7 $0.60 gpt-5-miniRating 1,372.3 $0.60 qwen3-vl-235b-a22b-instructRating 1,408.7 $0.64 gpt-4.1-miniRating 1,373.1 $0.64 qwen3-235b-a22b-thinking-2507Rating 1,385.6 $0.64 glm-4.7Rating 1,426.2 $0.67 mistral-large-2512Rating 1,402.9 $0.70 gpt-3.5-turboRating 1,215.9 $0.70 deepseek-flash-fast@parasailRating 1,475.8 $0.72 minimax-m3-fast@parasailRating 1,428.2 $0.72 qwen3.5-27bRating 1,400.3 $0.72 qwen3.5-27b@deepinfraRating 1,400.3 $0.73 gemini-3.5-flash-liteRating 1,443.2 $0.74 glm-4.6Rating 1,412.8 $0.80 glm-5.2@deepinfraRating 1,470.0 $0.81 deepseek-r1-0528Rating 1,397.3 $0.83 glm-5.3@deepinfraRating 1,480.6 $0.95 qwen3.5-397b-a17b@deepinfraRating 1,434.5 $0.96 qwen3.5-122b-a10bRating 1,405.7 $0.96 gemini-3-flash-previewRating 1,457.4 $1.00 qwen3.6-plusRating 1,434.7 $1.00 llama-3.3-70b-instruct-turboRating 1,292.7 $1.04 qwen3.5-397b-a17b@parasailRating 1,434.5 $1.12 qwen3-vl-235b-a22b-thinkingRating 1,381.9 $1.12 qwen3.5-397b-a17bRating 1,434.5 $1.20 kimi-k2.6Rating 1,455.5 $1.30 gemini-3.7-flashRating 1,486.1 $1.35 gemini-3.6-flashRating 1,472.8 $1.35 gemini-3.5-flashRating 1,465.4 $1.35 gpt-5.4-miniRating 1,434.5 $1.50 deepseek-v4-pro-0813@deepinfraRating 1,460.5 $1.56 deepseek-v4-pro@deepinfraRating 1,453.2 $1.56 o4-miniRating 1,368.0 $1.76 o3-miniRating 1,343.6 $1.76 claude-haiku-4-5Rating 1,414.6 $1.80 deepseek-v4-pro-0813Rating 1,460.5 $1.85 deepseek-v4-proRating 1,453.2 $1.85 deepseek-v4-pro@parasailRating 1,453.2 $1.85 deepseek-v4-pro@togetherRating 1,453.2 $1.85 glm-5.3Rating 1,480.6 $2.00 glm-5.3@alibaba-intlRating 1,480.6 $2.00 glm-5.3@parasailRating 1,480.6 $2.00 glm-5.2Rating 1,470.0 $2.00 glm-5.2@alibaba-intlRating 1,470.0 $2.00 glm-5.2@alibaba-usRating 1,470.0 $2.00 glm-5.1Rating 1,461.3 $2.00 qwen3.6-max-previewRating 1,450.5 $2.60 mistral-medium-3-5Rating 1,422.9 $2.70 deepseek-v4-pro-fast@parasailRating 1,453.2 $2.77 glm-5.3-fast@parasailRating 1,480.6 $3.00 gpt-5.1Rating 1,426.0 $3.00 gpt-5Rating 1,409.9 $3.00 gpt-4.1Rating 1,403.0 $3.20 o3Rating 1,402.6 $3.20 qwen3.7-maxRating 1,465.8 $3.50 qwen3.7-max@alibaba-usRating 1,465.8 $3.50 claude-sonnet-5Rating 1,466.6 $3.60 gpt-6-solRating 1,448.6 $3.60 gemini-3.1-pro-previewRating 1,479.7 $4.00 gpt-4oRating 1,325.5 $4.00 gpt-5.2Rating 1,423.8 $4.20 gpt-5.4Rating 1,460.1 $5.00 claude-sonnet-4-6Rating 1,474.5 $5.40 claude-sonnet-4-5Rating 1,462.0 $5.40 gpt-5.6-solRating 1,490.3 $7.20 claude-opus-4-6Rating 1,498.5 $9.00 claude-opus-5Rating 1,494.1 $9.00 claude-opus-4-7Rating 1,493.0 $9.00 claude-opus-4-8Rating 1,478.5 $9.00 claude-opus-4-5Rating 1,474.3 $9.00 gpt-5.5Rating 1,474.5 $10.00 gpt-4-turboRating 1,303.3 $14.00 claude-fable-5Rating 1,508.7 $18.00 o1Rating 1,407.0 $24.00 gpt-4Rating 1,285.3 $36.00 Sorting into categories Price against score, Medium 0 25 50 75 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 65.5 +/- 3.4, $0.08
deepseek-v4-flash@deepinfra: 63.1 +/- 3.4, $0.11
deepseek-v4-flash-0731: 65.5 +/- 3.4, $0.17
deepseek-v4-flash-0731@parasail: 65.5 +/- 3.4, $0.17
deepseek-v4-flash@parasail: 63.1 +/- 3.4, $0.17
claude-haiku-5-5: 66.5 +/- 3.3, $0.18
gpt-6-luna: 55.9 +/- 3.5, $0.18
deepseek-flash@deepinfra: 70.0 +/- 3.2, $0.20
glm-5.3-flash: 52.8 +/- 3.5, $0.22
glm-5.3-flash@deepinfra: 52.8 +/- 3.5, $0.22
deepseek-v4-flash: 63.1 +/- 3.4, $0.24
deepseek-v4-flash-vision-exp: 71.0 +/- 3.2, $0.30
gpt-5.6-luna: 60.1 +/- 3.5, $0.40
gpt-5.4-nano: 67.2 +/- 3.3, $0.41
minimax-m3@deepinfra: 57.5 +/- 3.5, $0.44
deepseek-flash: 70.0 +/- 3.2, $0.48
deepseek-flash@alibaba-intl: 70.0 +/- 3.2, $0.48
deepseek-flash@parasail: 70.0 +/- 3.2, $0.48
deepseek-flash@together: 70.0 +/- 3.2, $0.48
minimax-m3: 57.5 +/- 3.5, $0.48
minimax-m3@parasail: 57.5 +/- 3.5, $0.48
qwen3.8-27b@deepinfra: 72.7 +/- 3.2, $0.50
deepseek-v4-flash-0731@alibaba-intl: 65.5 +/- 3.4, $0.62
qwen3.8-27b@parasail: 72.7 +/- 3.2, $0.63
deepseek-flash-fast@parasail: 70.0 +/- 3.2, $0.72
minimax-m3-fast@parasail: 57.5 +/- 3.5, $0.72
gemini-3.5-flash-lite: 67.2 +/- 3.3, $0.74
glm-5.2@deepinfra: 62.3 +/- 3.4, $0.81
qwen3.6-27b@deepinfra: 53.2 +/- 3.5, $0.90
glm-5.3@deepinfra: 69.3 +/- 3.3, $0.95
qwen3.8-27b: 72.7 +/- 3.2, $1.00
qwen3.6-plus: 58.3 +/- 3.5, $1.00
qwen3.6-27b: 53.2 +/- 3.5, $1.20
kimi-k2.6: 64.4 +/- 3.4, $1.30
gemini-3.8-flash: 81.4 +/- 2.8, $1.35
gemini-3.7-flash: 79.9 +/- 2.8, $1.35
gemini-3.5-flash: 75.6 +/- 3.0, $1.35
gemini-3.6-flash: 75.4 +/- 3.1, $1.35
gpt-5.4-mini: 59.8 +/- 3.5, $1.50
deepseek-v4-pro-0813@deepinfra: 67.7 +/- 3.3, $1.56
deepseek-v4-pro@deepinfra: 62.4 +/- 3.4, $1.56
kimi-k2.7-code: 56.3 +/- 3.5, $1.56
inkling: 70.1 +/- 3.2, $1.61
deepseek-v4-pro-0813: 67.7 +/- 3.3, $1.85
deepseek-v4-pro: 62.4 +/- 3.4, $1.85
deepseek-v4-pro@parasail: 62.4 +/- 3.4, $1.85
deepseek-v4-pro@together: 62.4 +/- 3.4, $1.85
glm-5.3: 69.3 +/- 3.3, $2.00
glm-5.3@alibaba-intl: 69.3 +/- 3.3, $2.00
glm-5.3@parasail: 69.3 +/- 3.3, $2.00
glm-5.2: 62.3 +/- 3.4, $2.00
glm-5.2@alibaba-intl: 62.3 +/- 3.4, $2.00
glm-5.2@alibaba-us: 62.3 +/- 3.4, $2.00
deepseek-v4-pro-fast@parasail: 62.4 +/- 3.4, $2.77
qwen3.8-max-0902: 74.1 +/- 3.1, $2.80
glm-5.3-fast@parasail: 69.3 +/- 3.3, $3.00
grok-4.7: 75.3 +/- 3.1, $3.08
grok-4.6: 71.9 +/- 3.2, $3.08
qwen3.7-max: 74.0 +/- 3.1, $3.50
qwen3.7-max@alibaba-us: 74.0 +/- 3.1, $3.50
gpt-6.1-sol: 74.2 +/- 3.1, $3.60
gpt-6-sol: 68.6 +/- 3.3, $3.60
claude-sonnet-5: 63.9 +/- 3.4, $3.60
claude-sonnet-5-5: 56.8 +/- 3.5, $3.60
gemini-3.1-pro-preview: 79.1 +/- 2.9, $4.00
gpt-5.6-terra: 64.6 +/- 3.4, $4.00
gpt-5.2: 61.8 +/- 3.4, $4.20
gpt-5.4: 70.2 +/- 3.2, $5.00
kimi-k3@deepinfra: 71.4 +/- 3.2, $5.13
kimi-k3: 71.4 +/- 3.2, $5.40
kimi-k3@moonshot: 71.4 +/- 3.2, $5.40
claude-sonnet-4-6: 63.2 +/- 3.4, $5.40
gpt-5.6-sol: 71.9 +/- 3.2, $7.20
claude-opus-5-5: 65.7 +/- 3.4, $7.20
claude-opus-4-8: 72.0 +/- 3.2, $9.00
claude-opus-4-7: 66.7 +/- 3.3, $9.00
claude-opus-5: 63.8 +/- 3.4, $9.00
claude-opus-4-6: 63.3 +/- 3.4, $9.00
claude-opus-4-5: 62.6 +/- 3.4, $9.00
gpt-5.5: 70.7 +/- 3.2, $10.00
claude-fable-5: 75.8 +/- 3.0, $18.00
gpt-6-astra: 75.6 +/- 3.0, $18.00
claude-fable-5-1: 73.0 +/- 3.1, $18.00
gemini-3.8-flash Best, Balanced, Cheap Ranking, Medium 1. gemini-3.8-flash: 81.4 +/- 2.8 1 gemini-3.8-flash 81.4 2. gemini-3.7-flash: 79.9 +/- 2.8 2 gemini-3.7-flash 79.9 3. gemini-3.1-pro-preview: 79.1 +/- 2.9 3 gemini-3.1-pro-preview 79.1 4. claude-fable-5: 75.8 +/- 3.0 4 claude-fable-5 75.8 5. gemini-3.5-flash: 75.6 +/- 3.0 5 gemini-3.5-flash 75.6 6. gpt-6-astra: 75.6 +/- 3.0 6 gpt-6-astra 75.6 7. gemini-3.6-flash: 75.4 +/- 3.1 7 gemini-3.6-flash 75.4 8. grok-4.7: 75.3 +/- 3.1 8 grok-4.7 75.3 9. gpt-6.1-sol: 74.2 +/- 3.1 9 gpt-6.1-sol 74.2 10. qwen3.8-max-0902: 74.1 +/- 3.1 10 qwen3.8-max-0902 74.1 11. qwen3.7-max: 74.0 +/- 3.1 11 qwen3.7-max 74.0 12. qwen3.7-max@alibaba-us: 74.0 +/- 3.1 12 qwen3.7-max@alibaba-us 74.0 13. claude-fable-5-1: 73.0 +/- 3.1 13 claude-fable-5-1 73.0 14. qwen3.8-27b: 72.7 +/- 3.2 14 qwen3.8-27b 72.7 15. qwen3.8-27b@deepinfra: 72.7 +/- 3.2 15 qwen3.8-27b@deepinfra 72.7 16. qwen3.8-27b@parasail: 72.7 +/- 3.2 16 qwen3.8-27b@parasail 72.7 17. claude-opus-4-8: 72.0 +/- 3.2 17 claude-opus-4-8 72.0 18. grok-4.6: 71.9 +/- 3.2 18 grok-4.6 71.9 19. gpt-5.6-sol: 71.9 +/- 3.2 19 gpt-5.6-sol 71.9 20. kimi-k3: 71.4 +/- 3.2 20 kimi-k3 71.4 21. kimi-k3@deepinfra: 71.4 +/- 3.2 21 kimi-k3@deepinfra 71.4 22. kimi-k3@moonshot: 71.4 +/- 3.2 22 kimi-k3@moonshot 71.4 23. deepseek-v4-flash-vision-exp: 71.0 +/- 3.2 23 deepseek-v4-flash-vision-exp 71.0 24. gpt-5.5: 70.7 +/- 3.2 24 gpt-5.5 70.7 25. gpt-5.4: 70.2 +/- 3.2 25 gpt-5.4 70.2 26. inkling: 70.1 +/- 3.2 26 inkling 70.1 27. deepseek-flash: 70.0 +/- 3.2 27 deepseek-flash 70.0 28. deepseek-flash-fast@parasail: 70.0 +/- 3.2 28 deepseek-flash-fast@parasail 70.0 29. deepseek-flash@alibaba-intl: 70.0 +/- 3.2 29 deepseek-flash@alibaba-intl 70.0 30. deepseek-flash@deepinfra: 70.0 +/- 3.2 30 deepseek-flash@deepinfra 70.0 31. deepseek-flash@parasail: 70.0 +/- 3.2 31 deepseek-flash@parasail 70.0 32. deepseek-flash@together: 70.0 +/- 3.2 32 deepseek-flash@together 70.0 33. glm-5.3: 69.3 +/- 3.3 33 glm-5.3 69.3 34. glm-5.3-fast@parasail: 69.3 +/- 3.3 34 glm-5.3-fast@parasail 69.3 35. glm-5.3@alibaba-intl: 69.3 +/- 3.3 35 glm-5.3@alibaba-intl 69.3 36. glm-5.3@deepinfra: 69.3 +/- 3.3 36 glm-5.3@deepinfra 69.3 37. glm-5.3@parasail: 69.3 +/- 3.3 37 glm-5.3@parasail 69.3 38. gpt-6-sol: 68.6 +/- 3.3 38 gpt-6-sol 68.6 39. deepseek-v4-pro-0813: 67.7 +/- 3.3 39 deepseek-v4-pro-0813 67.7 40. deepseek-v4-pro-0813@deepinfra: 67.7 +/- 3.3 40 deepseek-v4-pro-0813@deepinfra 67.7 41. gemini-3.5-flash-lite: 67.2 +/- 3.3 41 gemini-3.5-flash-lite 67.2 42. gpt-5.4-nano: 67.2 +/- 3.3 42 gpt-5.4-nano 67.2 43. claude-opus-4-7: 66.7 +/- 3.3 43 claude-opus-4-7 66.7 44. claude-haiku-5-5: 66.5 +/- 3.3 44 claude-haiku-5-5 66.5 45. claude-opus-5-5: 65.7 +/- 3.4 45 claude-opus-5-5 65.7 46. deepseek-v4-flash-0731: 65.5 +/- 3.4 46 deepseek-v4-flash-0731 65.5 47. deepseek-v4-flash-0731@alibaba-intl: 65.5 +/- 3.4 47 deepseek-v4-flash-0731@alibaba-intl 65.5 48. deepseek-v4-flash-0731@deepinfra: 65.5 +/- 3.4 48 deepseek-v4-flash-0731@deepinfra 65.5 49. deepseek-v4-flash-0731@parasail: 65.5 +/- 3.4 49 deepseek-v4-flash-0731@parasail 65.5 50. gpt-5.6-terra: 64.6 +/- 3.4 50 gpt-5.6-terra 64.6 51. kimi-k2.6: 64.4 +/- 3.4 51 kimi-k2.6 64.4 52. claude-sonnet-5: 63.9 +/- 3.4 52 claude-sonnet-5 63.9 53. claude-opus-5: 63.8 +/- 3.4 53 claude-opus-5 63.8 54. claude-opus-4-6: 63.3 +/- 3.4 54 claude-opus-4-6 63.3 55. claude-sonnet-4-6: 63.2 +/- 3.4 55 claude-sonnet-4-6 63.2 56. deepseek-v4-flash: 63.1 +/- 3.4 56 deepseek-v4-flash 63.1 57. deepseek-v4-flash@deepinfra: 63.1 +/- 3.4 57 deepseek-v4-flash@deepinfra 63.1 58. deepseek-v4-flash@parasail: 63.1 +/- 3.4 58 deepseek-v4-flash@parasail 63.1 59. claude-opus-4-5: 62.6 +/- 3.4 59 claude-opus-4-5 62.6 60. deepseek-v4-pro: 62.4 +/- 3.4 60 deepseek-v4-pro 62.4 61. deepseek-v4-pro-fast@parasail: 62.4 +/- 3.4 61 deepseek-v4-pro-fast@parasail 62.4 62. deepseek-v4-pro@deepinfra: 62.4 +/- 3.4 62 deepseek-v4-pro@deepinfra 62.4 63. deepseek-v4-pro@parasail: 62.4 +/- 3.4 63 deepseek-v4-pro@parasail 62.4 64. deepseek-v4-pro@together: 62.4 +/- 3.4 64 deepseek-v4-pro@together 62.4 65. glm-5.2: 62.3 +/- 3.4 65 glm-5.2 62.3 66. glm-5.2@alibaba-intl: 62.3 +/- 3.4 66 glm-5.2@alibaba-intl 62.3 67. glm-5.2@alibaba-us: 62.3 +/- 3.4 67 glm-5.2@alibaba-us 62.3 68. glm-5.2@deepinfra: 62.3 +/- 3.4 68 glm-5.2@deepinfra 62.3 69. gpt-5.2: 61.8 +/- 3.4 69 gpt-5.2 61.8 70. gpt-5.6-luna: 60.1 +/- 3.5 70 gpt-5.6-luna 60.1 71. gpt-5.4-mini: 59.8 +/- 3.5 71 gpt-5.4-mini 59.8 72. qwen3.6-plus: 58.3 +/- 3.5 72 qwen3.6-plus 58.3 73. minimax-m3: 57.5 +/- 3.5 73 minimax-m3 57.5 74. minimax-m3-fast@parasail: 57.5 +/- 3.5 74 minimax-m3-fast@parasail 57.5 75. minimax-m3@deepinfra: 57.5 +/- 3.5 75 minimax-m3@deepinfra 57.5 76. minimax-m3@parasail: 57.5 +/- 3.5 76 minimax-m3@parasail 57.5 77. claude-sonnet-5-5: 56.8 +/- 3.5 77 claude-sonnet-5-5 56.8 78. kimi-k2.7-code: 56.3 +/- 3.5 78 kimi-k2.7-code 56.3 79. gpt-6-luna: 55.9 +/- 3.5 79 gpt-6-luna 55.9 80. qwen3.6-27b: 53.2 +/- 3.5 80 qwen3.6-27b 53.2 81. qwen3.6-27b@deepinfra: 53.2 +/- 3.5 81 qwen3.6-27b@deepinfra 53.2 82. glm-5.3-flash: 52.8 +/- 3.5 82 glm-5.3-flash 52.8 83. glm-5.3-flash@deepinfra: 52.8 +/- 3.5 83 glm-5.3-flash@deepinfra 52.8 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Conversation Price against score, Medium 0 25 50 75 100 $0.1 $1 $10
deepseek-v4-flash-0731@deepinfra: 33.6 +/- 1.5, $0.17
gpt-5-nano: 11.7 +/- 1.0, $0.24
deepseek-v4.1-flash-1017@deepinfra: 42.5 +/- 1.6, $0.27
qwen3.5-flash-2026-02-23: 20.3 +/- 1.3, $0.32
gpt-4.1-nano: 6.0 +/- 0.8, $0.32
deepseek-v4-flash-0731: 33.6 +/- 1.5, $0.34
deepseek-v4-flash-0731@parasail: 33.6 +/- 1.5, $0.34
gpt-6-luna: 41.4 +/- 1.6, $0.36
claude-haiku-5-5: 23.8 +/- 1.4, $0.36
deepseek-flash@deepinfra: 42.9 +/- 1.6, $0.39
gemma-4-31b-it: 10.4 +/- 1.0, $0.48
gpt-4o-mini: 8.3 +/- 0.9, $0.48
gpt-5.6-luna: 41.0 +/- 1.6, $0.80
gpt-5.4-nano: 11.7 +/- 1.0, $0.82
deepseek-flash: 42.9 +/- 1.6, $0.96
deepseek-flash@alibaba-intl: 42.9 +/- 1.6, $0.96
deepseek-flash@parasail: 42.9 +/- 1.6, $0.96
deepseek-flash@together: 42.9 +/- 1.6, $0.96
deepseek-v4.1-flash-1017: 42.5 +/- 1.6, $0.96
deepseek-v4.1-flash-1017@parasail: 42.5 +/- 1.6, $0.96
qwen3.6-flash: 15.9 +/- 1.2, $1.00
qwen3.6-flash@alibaba-us: 15.9 +/- 1.2, $1.00
inkling-small: 19.1 +/- 1.2, $1.20
gpt-5-mini: 21.6 +/- 1.3, $1.20
deepseek-v4-flash-0731@alibaba-intl: 33.6 +/- 1.5, $1.23
gpt-4.1-mini: 12.7 +/- 1.1, $1.28
qwen3-235b-a22b-thinking-2507: 40.4 +/- 1.6, $1.29
glm-4.7: 32.2 +/- 1.5, $1.34
deepseek-flash-fast@parasail: 42.9 +/- 1.6, $1.44
glm-5.2@deepinfra: 34.2 +/- 1.5, $1.62
glm-5.3@deepinfra: 41.0 +/- 1.6, $1.90
mistral-large-4-0: 20.0 +/- 1.3, $1.96
gemini-3-flash-preview: 66.8 +/- 1.5, $2.00
qwen3.6-plus: 44.1 +/- 1.6, $2.00
kimi-k2.6: 34.9 +/- 1.5, $2.60
gemini-3.8-flash: 69.7 +/- 1.5, $2.70
gemini-3.7-flash: 69.2 +/- 1.5, $2.70
gemini-3.6-flash: 66.2 +/- 1.5, $2.70
gemini-3.5-flash: 66.2 +/- 1.5, $2.70
gpt-5.4-mini: 29.4 +/- 1.4, $3.00
deepseek-v4-pro-0813@deepinfra: 52.9 +/- 1.6, $3.12
deepseek-v4-pro@deepinfra: 47.0 +/- 1.6, $3.12
kimi-k2.7-code: 36.5 +/- 1.5, $3.12
inkling: 40.3 +/- 1.6, $3.22
o4-mini: 19.6 +/- 1.3, $3.52
o3-mini: 15.3 +/- 1.1, $3.52
gpt-6.1-terra: 57.3 +/- 1.6, $3.60
claude-haiku-4-5: 13.2 +/- 1.1, $3.60
deepseek-v4-pro-0813: 52.9 +/- 1.6, $3.70
deepseek-v4-pro: 47.0 +/- 1.6, $3.70
deepseek-v4-pro@parasail: 47.0 +/- 1.6, $3.70
deepseek-v4-pro@together: 47.0 +/- 1.6, $3.70
glm-5.3: 41.0 +/- 1.6, $4.00
glm-5.3@alibaba-intl: 41.0 +/- 1.6, $4.00
glm-5.3@parasail: 41.0 +/- 1.6, $4.00
glm-5.1: 34.0 +/- 1.5, $4.00
glm-5.2: 34.2 +/- 1.5, $4.00
glm-5.2@alibaba-intl: 34.2 +/- 1.5, $4.00
glm-5.2@alibaba-us: 34.2 +/- 1.5, $4.00
qwen3.6-max-preview: 52.0 +/- 1.6, $5.20
deepseek-v4-pro-fast@parasail: 47.0 +/- 1.6, $5.54
qwen3.8-max-0902: 47.3 +/- 1.6, $5.60
glm-5.3-fast@parasail: 41.0 +/- 1.6, $6.00
gpt-5.1: 48.0 +/- 1.6, $6.00
gpt-5: 50.1 +/- 1.6, $6.00
grok-4.6: 49.3 +/- 1.6, $6.16
grok-4.7: 56.0 +/- 1.6, $6.16
o3: 49.4 +/- 1.6, $6.40
gpt-4.1: 31.1 +/- 1.5, $6.40
qwen3.7-max: 55.8 +/- 1.6, $7.00
qwen3.7-max@alibaba-us: 55.8 +/- 1.6, $7.00
gemini-4-carbon: 77.7 +/- 1.3, $7.20
gemini-4-argon: 76.0 +/- 1.4, $7.20
gpt-6.1-sol: 73.9 +/- 1.4, $7.20
gpt-6-sol: 60.7 +/- 1.6, $7.20
claude-sonnet-5-5: 46.5 +/- 1.6, $7.20
claude-sonnet-5: 33.7 +/- 1.5, $7.20
gemini-3.1-pro-preview: 73.5 +/- 1.4, $8.00
gpt-5.6-terra: 43.2 +/- 1.6, $8.00
gpt-4o: 26.0 +/- 1.4, $8.00
gpt-5.2: 32.8 +/- 1.5, $8.40
gpt-5.4: 45.1 +/- 1.6, $10.00
kimi-k3@deepinfra: 50.6 +/- 1.6, $10.26
kimi-k3: 50.6 +/- 1.6, $10.80
kimi-k3@moonshot: 50.6 +/- 1.6, $10.80
claude-sonnet-4-6: 32.8 +/- 1.5, $10.80
claude-sonnet-4-5: 23.7 +/- 1.4, $10.80
claude-opus-5-5: 72.2 +/- 1.4, $14.40
gpt-5.6-sol: 69.7 +/- 1.5, $14.40
claude-opus-4-7: 51.7 +/- 1.6, $18.00
claude-opus-4-8: 53.0 +/- 1.6, $18.00
claude-opus-5: 59.9 +/- 1.6, $18.00
claude-opus-4-6: 47.0 +/- 1.6, $18.00
claude-opus-4-5: 45.7 +/- 1.6, $18.00
gpt-5.5: 63.0 +/- 1.5, $20.00
claude-fable-5-2: 94.0 +/- 0.8, $28.80
claude-fable-5: 70.7 +/- 1.4, $36.00
gpt-6-astra: 75.6 +/- 1.4, $36.00
claude-fable-5-1: 70.8 +/- 1.4, $36.00
o1: 41.1 +/- 1.6, $48.00
gemini-4-carbon Best, Balanced, Cheap Ranking, Medium 1. gemini-4-carbon: 77.7 +/- 1.3 1 gemini-4-carbon 77.7 2. gemini-4-argon: 76.0 +/- 1.4 2 gemini-4-argon 76.0 3. claude-fable-5: 70.7 +/- 1.4 3 claude-fable-5 70.7 4. claude-opus-5-5: 72.2 +/- 1.4 4 claude-opus-5-5 72.2 5. gemini-3.1-pro-preview: 73.5 +/- 1.4 5 gemini-3.1-pro-preview 73.5 6. gemini-3.8-flash: 69.7 +/- 1.5 6 gemini-3.8-flash 69.7 7. claude-fable-5-2: 94.0 +/- 0.8 7 claude-fable-5-2 94.0 8. gpt-5.6-sol: 69.7 +/- 1.5 8 gpt-5.6-sol 69.7 9. gemini-3.7-flash: 69.2 +/- 1.5 9 gemini-3.7-flash 69.2 10. gpt-6.1-sol: 73.9 +/- 1.4 10 gpt-6.1-sol 73.9 11. gpt-6-astra: 75.6 +/- 1.4 11 gpt-6-astra 75.6 12. gemini-3.6-flash: 66.2 +/- 1.5 12 gemini-3.6-flash 66.2 13. claude-fable-5-1: 70.8 +/- 1.4 13 claude-fable-5-1 70.8 14. claude-opus-4-7: 51.7 +/- 1.6 14 claude-opus-4-7 51.7 15. gemini-3-flash-preview: 66.8 +/- 1.5 15 gemini-3-flash-preview 66.8 16. gpt-5.5: 63.0 +/- 1.5 16 gpt-5.5 63.0 17. claude-opus-4-8: 53.0 +/- 1.6 17 claude-opus-4-8 53.0 18. claude-opus-5: 59.9 +/- 1.6 18 claude-opus-5 59.9 19. gemini-3.5-flash: 66.2 +/- 1.5 19 gemini-3.5-flash 66.2 20. kimi-k3: 50.6 +/- 1.6 20 kimi-k3 50.6 21. kimi-k3@deepinfra: 50.6 +/- 1.6 21 kimi-k3@deepinfra 50.6 22. kimi-k3@moonshot: 50.6 +/- 1.6 22 kimi-k3@moonshot 50.6 23. claude-opus-4-6: 47.0 +/- 1.6 23 claude-opus-4-6 47.0 24. qwen3.8-max-0902: 47.3 +/- 1.6 24 qwen3.8-max-0902 47.3 25. qwen3.7-max: 55.8 +/- 1.6 25 qwen3.7-max 55.8 26. qwen3.7-max@alibaba-us: 55.8 +/- 1.6 26 qwen3.7-max@alibaba-us 55.8 27. gpt-6-sol: 60.7 +/- 1.6 27 gpt-6-sol 60.7 28. gpt-6.1-terra: 57.3 +/- 1.6 28 gpt-6.1-terra 57.3 29. deepseek-v4-pro-0813: 52.9 +/- 1.6 29 deepseek-v4-pro-0813 52.9 30. deepseek-v4-pro-0813@deepinfra: 52.9 +/- 1.6 30 deepseek-v4-pro-0813@deepinfra 52.9 31. claude-opus-4-5: 45.7 +/- 1.6 31 claude-opus-4-5 45.7 32. qwen3.6-max-preview: 52.0 +/- 1.6 32 qwen3.6-max-preview 52.0 33. gpt-5.4: 45.1 +/- 1.6 33 gpt-5.4 45.1 34. deepseek-v4-pro: 47.0 +/- 1.6 34 deepseek-v4-pro 47.0 35. deepseek-v4-pro-fast@parasail: 47.0 +/- 1.6 35 deepseek-v4-pro-fast@parasail 47.0 36. deepseek-v4-pro@deepinfra: 47.0 +/- 1.6 36 deepseek-v4-pro@deepinfra 47.0 37. deepseek-v4-pro@parasail: 47.0 +/- 1.6 37 deepseek-v4-pro@parasail 47.0 38. deepseek-v4-pro@together: 47.0 +/- 1.6 38 deepseek-v4-pro@together 47.0 42. claude-sonnet-5-5: 46.5 +/- 1.6 42 claude-sonnet-5-5 46.5 46. glm-5.3: 41.0 +/- 1.6 46 glm-5.3 41.0 47. glm-5.3-fast@parasail: 41.0 +/- 1.6 47 glm-5.3-fast@parasail 41.0 48. glm-5.3@alibaba-intl: 41.0 +/- 1.6 48 glm-5.3@alibaba-intl 41.0 49. glm-5.3@deepinfra: 41.0 +/- 1.6 49 glm-5.3@deepinfra 41.0 50. glm-5.3@parasail: 41.0 +/- 1.6 50 glm-5.3@parasail 41.0 51. gpt-5.6-terra: 43.2 +/- 1.6 51 gpt-5.6-terra 43.2 52. deepseek-flash: 42.9 +/- 1.6 52 deepseek-flash 42.9 53. deepseek-flash-fast@parasail: 42.9 +/- 1.6 53 deepseek-flash-fast@parasail 42.9 54. deepseek-flash@alibaba-intl: 42.9 +/- 1.6 54 deepseek-flash@alibaba-intl 42.9 55. deepseek-flash@deepinfra: 42.9 +/- 1.6 55 deepseek-flash@deepinfra 42.9 56. deepseek-flash@parasail: 42.9 +/- 1.6 56 deepseek-flash@parasail 42.9 57. deepseek-flash@together: 42.9 +/- 1.6 57 deepseek-flash@together 42.9 59. grok-4.6: 49.3 +/- 1.6 59 grok-4.6 49.3 60. grok-4.7: 56.0 +/- 1.6 60 grok-4.7 56.0 64. gpt-5.1: 48.0 +/- 1.6 64 gpt-5.1 48.0 65. claude-sonnet-4-6: 32.8 +/- 1.5 65 claude-sonnet-4-6 32.8 66. glm-5.1: 34.0 +/- 1.5 66 glm-5.1 34.0 67. glm-5.2: 34.2 +/- 1.5 67 glm-5.2 34.2 68. glm-5.2@alibaba-intl: 34.2 +/- 1.5 68 glm-5.2@alibaba-intl 34.2 69. glm-5.2@alibaba-us: 34.2 +/- 1.5 69 glm-5.2@alibaba-us 34.2 70. glm-5.2@deepinfra: 34.2 +/- 1.5 70 glm-5.2@deepinfra 34.2 71. gpt-5.6-luna: 41.0 +/- 1.6 71 gpt-5.6-luna 41.0 72. claude-sonnet-5: 33.7 +/- 1.5 72 claude-sonnet-5 33.7 73. qwen3.6-plus: 44.1 +/- 1.6 73 qwen3.6-plus 44.1 74. claude-sonnet-4-5: 23.7 +/- 1.4 74 claude-sonnet-4-5 23.7 81. gpt-5: 50.1 +/- 1.6 81 gpt-5 50.1 82. kimi-k2.6: 34.9 +/- 1.5 82 kimi-k2.6 34.9 83. inkling: 40.3 +/- 1.6 83 inkling 40.3 84. gpt-5.4-mini: 29.4 +/- 1.4 84 gpt-5.4-mini 29.4 85. o3: 49.4 +/- 1.6 85 o3 49.4 90. gpt-6-luna: 41.4 +/- 1.6 90 gpt-6-luna 41.4 91. deepseek-v4.1-flash-1017: 42.5 +/- 1.6 91 deepseek-v4.1-flash-1017 42.5 92. deepseek-v4.1-flash-1017@deepinfra: 42.5 +/- 1.6 92 deepseek-v4.1-flash-1017@deepinfra 42.5 93. deepseek-v4.1-flash-1017@parasail: 42.5 +/- 1.6 93 deepseek-v4.1-flash-1017@parasail 42.5 94. glm-4.7: 32.2 +/- 1.5 94 glm-4.7 32.2 96. gemma-4-31b-it: 10.4 +/- 1.0 96 gemma-4-31b-it 10.4 104. gpt-5.2: 32.8 +/- 1.5 104 gpt-5.2 32.8 111. kimi-k2.7-code: 36.5 +/- 1.5 111 kimi-k2.7-code 36.5 115. gpt-4.1: 31.1 +/- 1.5 115 gpt-4.1 31.1 118. qwen3-235b-a22b-thinking-2507: 40.4 +/- 1.6 118 qwen3-235b-a22b-thinking-2507 40.4 119. o1: 41.1 +/- 1.6 119 o1 41.1 123. deepseek-v4-flash-0731: 33.6 +/- 1.5 123 deepseek-v4-flash-0731 33.6 124. deepseek-v4-flash-0731@alibaba-intl: 33.6 +/- 1.5 124 deepseek-v4-flash-0731@alibaba-intl 33.6 125. deepseek-v4-flash-0731@deepinfra: 33.6 +/- 1.5 125 deepseek-v4-flash-0731@deepinfra 33.6 126. deepseek-v4-flash-0731@parasail: 33.6 +/- 1.5 126 deepseek-v4-flash-0731@parasail 33.6 133. claude-haiku-5-5: 23.8 +/- 1.4 133 claude-haiku-5-5 23.8 134. claude-haiku-4-5: 13.2 +/- 1.1 134 claude-haiku-4-5 13.2 135. inkling-small: 19.1 +/- 1.2 135 inkling-small 19.1 138. gpt-5.4-nano: 11.7 +/- 1.0 138 gpt-5.4-nano 11.7 140. qwen3.5-flash-2026-02-23: 20.3 +/- 1.3 140 qwen3.5-flash-2026-02-23 20.3 144. mistral-large-4-0: 20.0 +/- 1.3 144 mistral-large-4-0 20.0 145. o4-mini: 19.6 +/- 1.3 145 o4-mini 19.6 146. gpt-5-mini: 21.6 +/- 1.3 146 gpt-5-mini 21.6 148. gpt-4.1-mini: 12.7 +/- 1.1 148 gpt-4.1-mini 12.7 151. gpt-4o: 26.0 +/- 1.4 151 gpt-4o 26.0 152. qwen3.6-flash: 15.9 +/- 1.2 152 qwen3.6-flash 15.9 153. qwen3.6-flash@alibaba-us: 15.9 +/- 1.2 153 qwen3.6-flash@alibaba-us 15.9 160. o3-mini: 15.3 +/- 1.1 160 o3-mini 15.3 167. gpt-5-nano: 11.7 +/- 1.0 167 gpt-5-nano 11.7 169. gpt-4.1-nano: 6.0 +/- 0.8 169 gpt-4.1-nano 6.0 170. gpt-4o-mini: 8.3 +/- 0.9 170 gpt-4o-mini 8.3 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Rating only, not on the chart (70) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
gpt-oss-20bRating 1,289.2 $0.10 mistral-small-24b-instruct-2501Rating 1,249.7 $0.11 gemma-3-4b-itRating 1,272.2 $0.12 gpt-oss-120b@deepinfraRating 1,326.4 $0.13 gemma-3-12b-itRating 1,344.4 $0.14 phi-4Rating 1,241.8 $0.17 gemma-3-27b-itRating 1,358.0 $0.19 granite-4.2-8bRating 1,269.6 $0.20 deepseek-v4-flash@deepinfraRating 1,452.9 $0.22 qwen3-32bRating 1,336.9 $0.24 llama-4-scout-17b-16e-instructRating 1,319.2 $0.28 llama-3.3-70b-instruct-turbo@deepinfraRating 1,316.9 $0.29 deepseek-v4-flash@parasailRating 1,452.9 $0.34 mimo-v2.6-flashRating 1,444.7 $0.34 qwen3-235b-a22b-instruct-2507@deepinfraRating 1,436.2 $0.36 qwen3-30b-a3bRating 1,318.5 $0.39 glm-5.3-flashRating 1,476.4 $0.44 glm-5.3-flash@deepinfraRating 1,476.4 $0.44 deepseek-v4-flashRating 1,452.9 $0.48 gpt-oss-120bRating 1,326.4 $0.48 qwen3-235b-a22b-instruct-2507@parasailRating 1,436.2 $0.54 llama-3.3-70b-instruct-turbo@parasailRating 1,316.9 $0.55 deepseek-v3.2Rating 1,427.7 $0.57 qwen3-next-80b-a3b-instruct@deepinfraRating 1,400.0 $0.58 qwen3-next-80b-a3b-instruct@parasailRating 1,400.0 $0.60 qwen3.5-35b-a3b@deepinfraRating 1,394.3 $0.62 qwen3.5-35b-a3b@parasailRating 1,394.3 $0.64 qwen3-30b-a3b-instruct-2507Rating 1,379.1 $0.64 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,424.4 $0.67 qwen3-next-80b-a3b-instructRating 1,400.0 $0.72 qwen3-next-80b-a3b-thinkingRating 1,349.6 $0.72 qwen3-235b-a22b-instruct-2507Rating 1,436.2 $0.74 inkling-2-small@deepinfraRating 1,414.7 $0.84 inkling-2-smallRating 1,414.7 $0.88 minimax-m3@deepinfraRating 1,448.1 $0.89 mimo-v2.6-proRating 1,462.0 $0.93 minimax-m3Rating 1,448.1 $0.96 minimax-m3@parasailRating 1,448.1 $0.96 qwen3.8-27b@deepinfraRating 1,436.4 $0.99 gemini-3.1-flash-liteRating 1,435.4 $1.00 qwen3.7-plusRating 1,464.1 $1.02 qwen3.7-plus@alibaba-usRating 1,464.1 $1.02 qwen3-vl-235b-a22b-instruct@parasailRating 1,424.4 $1.10 beam-1.0-501b-a23bRating 1,432.3 $1.12 qwen-plus-2025-01-25Rating 1,352.4 $1.12 qwen3.5-35b-a3bRating 1,394.3 $1.20 qwen3.8-27b@parasailRating 1,436.4 $1.26 qwen3-vl-235b-a22b-instructRating 1,424.4 $1.28 mistral-large-2512Rating 1,419.1 $1.40 gpt-3.5-turboRating 1,201.4 $1.40 minimax-m3-fast@parasailRating 1,448.1 $1.44 qwen3.5-27bRating 1,413.7 $1.44 qwen3.5-27b@deepinfraRating 1,413.7 $1.46 gemini-3.5-flash-liteRating 1,466.8 $1.48 glm-4.6Rating 1,421.0 $1.60 deepseek-r1-0528Rating 1,409.4 $1.66 qwen3.5-397b-a17b@deepinfraRating 1,451.6 $1.92 qwen3.5-122b-a10bRating 1,419.3 $1.92 qwen3.8-27bRating 1,436.4 $2.00 llama-3.3-70b-instruct-turboRating 1,316.9 $2.08 qwen3.5-397b-a17b@parasailRating 1,451.6 $2.24 qwen3-vl-235b-a22b-thinkingRating 1,384.5 $2.24 qwen3.5-397b-a17bRating 1,451.6 $2.40 gemini-4-hydrogenRating 1,476.4 $5.40 mistral-medium-3-5Rating 1,432.7 $5.40 kimi-k3.1-pro@deepinfraRating 1,474.9 $6.84 kimi-k3.1-proRating 1,474.9 $7.20 kimi-k3.1-pro@togetherRating 1,474.9 $7.20 gpt-4-turboRating 1,312.9 $28.00 gpt-4Rating 1,274.2 $72.00 Analysis and reasoning Price against score, Medium 0 25 50 75 100 $0.1 $1 $10
gpt-oss-120b@deepinfra: 22.1 +/- 2.9, $0.10
deepseek-v4-flash-0731@deepinfra: 61.1 +/- 3.5, $0.12
deepseek-v4-flash@deepinfra: 46.3 +/- 3.5, $0.14
llama-3.3-70b-instruct-turbo@deepinfra: 19.9 +/- 2.8, $0.21
deepseek-v4-flash-0731: 61.1 +/- 3.5, $0.22
deepseek-v4-flash-0731@parasail: 61.1 +/- 3.5, $0.22
deepseek-v4-flash@parasail: 46.3 +/- 3.5, $0.22
deepseek-flash@deepinfra: 66.7 +/- 3.3, $0.28
deepseek-v4-flash: 46.3 +/- 3.5, $0.32
gpt-oss-120b: 22.1 +/- 2.9, $0.36
gpt-4o-mini: 10.7 +/- 2.2, $0.36
llama-3.3-70b-instruct-turbo@parasail: 19.9 +/- 2.8, $0.38
gpt-5.6-luna: 46.8 +/- 3.5, $0.64
minimax-m3@deepinfra: 45.8 +/- 3.5, $0.66
deepseek-flash: 66.7 +/- 3.3, $0.72
deepseek-flash@alibaba-intl: 66.7 +/- 3.3, $0.72
deepseek-flash@parasail: 66.7 +/- 3.3, $0.72
deepseek-flash@together: 66.7 +/- 3.3, $0.72
minimax-m3: 45.8 +/- 3.5, $0.72
minimax-m3@parasail: 45.8 +/- 3.5, $0.72
qwen3.6-flash: 35.2 +/- 3.4, $0.80
qwen3.6-flash@alibaba-us: 35.2 +/- 3.4, $0.80
qwen3.8-27b@deepinfra: 60.2 +/- 3.5, $0.87
deepseek-v4-flash-0731@alibaba-intl: 61.1 +/- 3.5, $0.88
mistral-large-2512: 20.4 +/- 2.9, $1.00
glm-4.7: 47.7 +/- 3.5, $1.02
qwen3.8-27b@parasail: 60.2 +/- 3.5, $1.07
deepseek-flash-fast@parasail: 66.7 +/- 3.3, $1.08
minimax-m3-fast@parasail: 45.8 +/- 3.5, $1.08
glm-5.2@deepinfra: 58.8 +/- 3.5, $1.17
llama-3.3-70b-instruct-turbo: 19.9 +/- 2.8, $1.25
deepseek-r1-0528: 40.8 +/- 3.5, $1.26
glm-5.3@deepinfra: 66.2 +/- 3.3, $1.45
gemini-3-flash-preview: 61.1 +/- 3.5, $1.60
qwen3.8-27b: 60.2 +/- 3.5, $1.60
gemini-3.8-flash: 82.4 +/- 2.7, $2.10
gemini-3.5-flash: 76.7 +/- 3.0, $2.10
kimi-k2.7-code: 57.9 +/- 3.5, $2.36
inkling: 50.0 +/- 3.5, $2.42
o4-mini: 38.7 +/- 3.4, $2.64
o3-mini: 22.8 +/- 3.0, $2.64
gpt-6.1-terra: 72.2 +/- 3.2, $2.80
glm-5.3: 66.2 +/- 3.3, $2.88
glm-5.3@alibaba-intl: 66.2 +/- 3.3, $2.88
glm-5.3@parasail: 66.2 +/- 3.3, $2.88
glm-5.2: 58.8 +/- 3.5, $2.88
glm-5.2@alibaba-intl: 58.8 +/- 3.5, $2.88
glm-5.2@alibaba-us: 58.8 +/- 3.5, $2.88
glm-5.1: 55.1 +/- 3.5, $2.88
qwen3.6-max-preview: 63.0 +/- 3.4, $4.16
glm-5.3-fast@parasail: 66.2 +/- 3.3, $4.32
grok-4.6: 75.9 +/- 3.0, $4.40
o3: 53.1 +/- 3.5, $4.80
gpt-4.1: 27.0 +/- 3.1, $4.80
qwen3.7-max: 70.4 +/- 3.2, $5.00
qwen3.7-max@alibaba-us: 70.4 +/- 3.2, $5.00
gpt-5: 56.7 +/- 3.5, $5.00
gpt-5.1: 53.2 +/- 3.5, $5.00
gemini-4-carbon: 92.6 +/- 1.9, $5.60
gemini-4-argon: 91.7 +/- 2.0, $5.60
gpt-6.1-sol: 82.9 +/- 2.7, $5.60
claude-sonnet-5-5: 75.9 +/- 3.0, $5.60
gpt-6-sol: 73.1 +/- 3.1, $5.60
claude-sonnet-5: 60.6 +/- 3.5, $5.60
gpt-4o: 17.8 +/- 2.7, $6.00
gemini-3.1-pro-preview: 79.6 +/- 2.9, $6.40
gpt-5.6-terra: 48.9 +/- 3.5, $6.40
gpt-5.2: 45.8 +/- 3.5, $7.00
kimi-k3@deepinfra: 60.7 +/- 3.5, $7.98
kimi-k3: 60.7 +/- 3.5, $8.40
kimi-k3@moonshot: 60.7 +/- 3.5, $8.40
claude-sonnet-4-5: 54.3 +/- 3.5, $8.40
claude-opus-5-5: 88.4 +/- 2.3, $11.20
gpt-5.6-sol: 71.7 +/- 3.2, $11.20
claude-opus-5: 80.6 +/- 2.8, $14.00
claude-opus-4-6: 67.6 +/- 3.3, $14.00
claude-opus-4-8: 64.8 +/- 3.4, $14.00
claude-opus-4-5: 62.0 +/- 3.4, $14.00
claude-opus-4-7: 61.7 +/- 3.4, $14.00
gpt-5.5: 69.0 +/- 3.3, $16.00
gpt-4-turbo: 25.1 +/- 3.1, $20.00
claude-fable-5-2: 94.9 +/- 1.6, $22.40
claude-fable-5-1: 86.6 +/- 2.4, $28.00
gpt-6-astra: 86.5 +/- 2.4, $28.00
claude-fable-5: 81.9 +/- 2.7, $28.00
o1: 41.7 +/- 3.5, $36.00
gemini-4-carbon Best, Balanced, Cheap Ranking, Medium 1. claude-fable-5-2: 94.9 +/- 1.6 1 claude-fable-5-2 94.9 2. gemini-4-carbon: 92.6 +/- 1.9 2 gemini-4-carbon 92.6 3. gemini-4-argon: 91.7 +/- 2.0 3 gemini-4-argon 91.7 4. claude-opus-5-5: 88.4 +/- 2.3 4 claude-opus-5-5 88.4 5. claude-fable-5-1: 86.6 +/- 2.4 5 claude-fable-5-1 86.6 6. gpt-6-astra: 86.5 +/- 2.4 6 gpt-6-astra 86.5 7. gpt-6.1-sol: 82.9 +/- 2.7 7 gpt-6.1-sol 82.9 8. gemini-3.8-flash: 82.4 +/- 2.7 8 gemini-3.8-flash 82.4 9. claude-fable-5: 81.9 +/- 2.7 9 claude-fable-5 81.9 10. claude-opus-5: 80.6 +/- 2.8 10 claude-opus-5 80.6 11. gemini-3.1-pro-preview: 79.6 +/- 2.9 11 gemini-3.1-pro-preview 79.6 12. gemini-3.5-flash: 76.7 +/- 3.0 12 gemini-3.5-flash 76.7 13. claude-sonnet-5-5: 75.9 +/- 3.0 13 claude-sonnet-5-5 75.9 14. grok-4.6: 75.9 +/- 3.0 14 grok-4.6 75.9 15. gpt-6-sol: 73.1 +/- 3.1 15 gpt-6-sol 73.1 16. gpt-6.1-terra: 72.2 +/- 3.2 16 gpt-6.1-terra 72.2 17. gpt-5.6-sol: 71.7 +/- 3.2 17 gpt-5.6-sol 71.7 18. qwen3.7-max: 70.4 +/- 3.2 18 qwen3.7-max 70.4 19. qwen3.7-max@alibaba-us: 70.4 +/- 3.2 19 qwen3.7-max@alibaba-us 70.4 20. gpt-5.5: 69.0 +/- 3.3 20 gpt-5.5 69.0 21. claude-opus-4-6: 67.6 +/- 3.3 21 claude-opus-4-6 67.6 22. deepseek-flash: 66.7 +/- 3.3 22 deepseek-flash 66.7 23. deepseek-flash-fast@parasail: 66.7 +/- 3.3 23 deepseek-flash-fast@parasail 66.7 24. deepseek-flash@alibaba-intl: 66.7 +/- 3.3 24 deepseek-flash@alibaba-intl 66.7 25. deepseek-flash@deepinfra: 66.7 +/- 3.3 25 deepseek-flash@deepinfra 66.7 26. deepseek-flash@parasail: 66.7 +/- 3.3 26 deepseek-flash@parasail 66.7 27. deepseek-flash@together: 66.7 +/- 3.3 27 deepseek-flash@together 66.7 28. glm-5.3: 66.2 +/- 3.3 28 glm-5.3 66.2 29. glm-5.3-fast@parasail: 66.2 +/- 3.3 29 glm-5.3-fast@parasail 66.2 30. glm-5.3@alibaba-intl: 66.2 +/- 3.3 30 glm-5.3@alibaba-intl 66.2 31. glm-5.3@deepinfra: 66.2 +/- 3.3 31 glm-5.3@deepinfra 66.2 32. glm-5.3@parasail: 66.2 +/- 3.3 32 glm-5.3@parasail 66.2 33. claude-opus-4-8: 64.8 +/- 3.4 33 claude-opus-4-8 64.8 34. qwen3.6-max-preview: 63.0 +/- 3.4 34 qwen3.6-max-preview 63.0 35. claude-opus-4-5: 62.0 +/- 3.4 35 claude-opus-4-5 62.0 36. claude-opus-4-7: 61.7 +/- 3.4 36 claude-opus-4-7 61.7 37. deepseek-v4-flash-0731: 61.1 +/- 3.5 37 deepseek-v4-flash-0731 61.1 38. deepseek-v4-flash-0731@alibaba-intl: 61.1 +/- 3.5 38 deepseek-v4-flash-0731@alibaba-intl 61.1 39. deepseek-v4-flash-0731@deepinfra: 61.1 +/- 3.5 39 deepseek-v4-flash-0731@deepinfra 61.1 40. deepseek-v4-flash-0731@parasail: 61.1 +/- 3.5 40 deepseek-v4-flash-0731@parasail 61.1 41. gemini-3-flash-preview: 61.1 +/- 3.5 41 gemini-3-flash-preview 61.1 42. kimi-k3: 60.7 +/- 3.5 42 kimi-k3 60.7 43. kimi-k3@deepinfra: 60.7 +/- 3.5 43 kimi-k3@deepinfra 60.7 44. kimi-k3@moonshot: 60.7 +/- 3.5 44 kimi-k3@moonshot 60.7 45. claude-sonnet-5: 60.6 +/- 3.5 45 claude-sonnet-5 60.6 46. qwen3.8-27b: 60.2 +/- 3.5 46 qwen3.8-27b 60.2 47. qwen3.8-27b@deepinfra: 60.2 +/- 3.5 47 qwen3.8-27b@deepinfra 60.2 48. qwen3.8-27b@parasail: 60.2 +/- 3.5 48 qwen3.8-27b@parasail 60.2 49. glm-5.2: 58.8 +/- 3.5 49 glm-5.2 58.8 50. glm-5.2@alibaba-intl: 58.8 +/- 3.5 50 glm-5.2@alibaba-intl 58.8 51. glm-5.2@alibaba-us: 58.8 +/- 3.5 51 glm-5.2@alibaba-us 58.8 52. glm-5.2@deepinfra: 58.8 +/- 3.5 52 glm-5.2@deepinfra 58.8 53. kimi-k2.7-code: 57.9 +/- 3.5 53 kimi-k2.7-code 57.9 54. gpt-5: 56.7 +/- 3.5 54 gpt-5 56.7 55. glm-5.1: 55.1 +/- 3.5 55 glm-5.1 55.1 56. claude-sonnet-4-5: 54.3 +/- 3.5 56 claude-sonnet-4-5 54.3 57. gpt-5.1: 53.2 +/- 3.5 57 gpt-5.1 53.2 58. o3: 53.1 +/- 3.5 58 o3 53.1 59. inkling: 50.0 +/- 3.5 59 inkling 50.0 60. gpt-5.6-terra: 48.9 +/- 3.5 60 gpt-5.6-terra 48.9 61. glm-4.7: 47.7 +/- 3.5 61 glm-4.7 47.7 62. gpt-5.6-luna: 46.8 +/- 3.5 62 gpt-5.6-luna 46.8 63. deepseek-v4-flash: 46.3 +/- 3.5 63 deepseek-v4-flash 46.3 64. deepseek-v4-flash@deepinfra: 46.3 +/- 3.5 64 deepseek-v4-flash@deepinfra 46.3 65. deepseek-v4-flash@parasail: 46.3 +/- 3.5 65 deepseek-v4-flash@parasail 46.3 66. gpt-5.2: 45.8 +/- 3.5 66 gpt-5.2 45.8 67. minimax-m3: 45.8 +/- 3.5 67 minimax-m3 45.8 68. minimax-m3-fast@parasail: 45.8 +/- 3.5 68 minimax-m3-fast@parasail 45.8 69. minimax-m3@deepinfra: 45.8 +/- 3.5 69 minimax-m3@deepinfra 45.8 70. minimax-m3@parasail: 45.8 +/- 3.5 70 minimax-m3@parasail 45.8 71. o1: 41.7 +/- 3.5 71 o1 41.7 72. deepseek-r1-0528: 40.8 +/- 3.5 72 deepseek-r1-0528 40.8 73. o4-mini: 38.7 +/- 3.4 73 o4-mini 38.7 74. qwen3.6-flash: 35.2 +/- 3.4 74 qwen3.6-flash 35.2 75. qwen3.6-flash@alibaba-us: 35.2 +/- 3.4 75 qwen3.6-flash@alibaba-us 35.2 76. gpt-4.1: 27.0 +/- 3.1 76 gpt-4.1 27.0 77. gpt-4-turbo: 25.1 +/- 3.1 77 gpt-4-turbo 25.1 78. o3-mini: 22.8 +/- 3.0 78 o3-mini 22.8 79. gpt-oss-120b: 22.1 +/- 2.9 79 gpt-oss-120b 22.1 80. gpt-oss-120b@deepinfra: 22.1 +/- 2.9 80 gpt-oss-120b@deepinfra 22.1 81. mistral-large-2512: 20.4 +/- 2.9 81 mistral-large-2512 20.4 82. llama-3.3-70b-instruct-turbo: 19.9 +/- 2.8 82 llama-3.3-70b-instruct-turbo 19.9 83. llama-3.3-70b-instruct-turbo@deepinfra: 19.9 +/- 2.8 83 llama-3.3-70b-instruct-turbo@deepinfra 19.9 84. llama-3.3-70b-instruct-turbo@parasail: 19.9 +/- 2.8 84 llama-3.3-70b-instruct-turbo@parasail 19.9 85. gpt-4o: 17.8 +/- 2.7 85 gpt-4o 17.8 86. gpt-4o-mini: 10.7 +/- 2.2 86 gpt-4o-mini 10.7 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Price against score, Hard 0 25 50 75 100 $0.1 $1 $10 $100
gpt-oss-20b: 53.2 +/- 2.5, $0.19
gemma-3-4b-it: 23.2 +/- 1.9, $0.19
gpt-oss-120b@deepinfra: 75.8 +/- 2.7, $0.23
gemma-3-12b-it: 39.5 +/- 2.5, $0.24
phi-4: 56.1 +/- 2.6, $0.27
deepseek-v4-flash-0731@deepinfra: 91.0 +/- 1.8, $0.28
gemma-3-27b-it: 47.7 +/- 2.7, $0.30
qwen3.5-9b@deepinfra: 79.0 +/- 2.4, $0.34
qwen3-32b: 65.7 +/- 2.8, $0.41
qwen3.5-9b@parasail: 79.0 +/- 2.4, $0.43
gemma-4-26b-a4b-it: 73.2 +/- 3.2, $0.45
qwen3-14b: 63.8 +/- 2.9, $0.46
gpt-5-nano: 57.6 +/- 3.5, $0.46
deepseek-v4.1-flash-1017@deepinfra: 87.0 +/- 3.1, $0.46
llama-4-scout-17b-16e-instruct: 51.8 +/- 3.2, $0.47
llama-3.3-70b-instruct-turbo@deepinfra: 47.4 +/- 2.9, $0.49
deepseek-v4-flash-0731: 91.0 +/- 1.8, $0.53
deepseek-v4-flash-0731@parasail: 91.0 +/- 1.8, $0.53
qwen3.5-flash-2026-02-23: 82.3 +/- 2.7, $0.56
gpt-4.1-nano: 48.9 +/- 2.5, $0.56
qwen3.5-9b: 79.0 +/- 2.4, $0.57
gpt-6-luna: 73.6 +/- 3.7, $0.65
claude-haiku-5-5: 85.5 +/- 2.8, $0.65
deepseek-flash@deepinfra: 89.8 +/- 1.7, $0.66
qwen3-30b-a3b: 61.7 +/- 3.0, $0.69
glm-5.3-flash: 80.6 +/- 3.3, $0.75
glm-5.3-flash@deepinfra: 80.6 +/- 3.3, $0.75
gemma-4-31b-it: 61.7 +/- 3.6, $0.76
minimax-m3.1-flash: 79.8 +/- 4.7, $0.84
minimax-m3.1-flash@parasail: 79.8 +/- 4.7, $0.84
gpt-oss-120b: 75.8 +/- 2.7, $0.84
gpt-4o-mini: 37.7 +/- 2.4, $0.84
llama-3.3-70b-instruct-turbo@parasail: 47.4 +/- 2.9, $0.89
qwen3.6-35b-a3b@parasail: 83.8 +/- 2.6, $1.01
qwen3.6-35b-a3b@deepinfra: 83.8 +/- 2.6, $1.06
qwen3-30b-a3b-instruct-2507: 55.6 +/- 2.9, $1.12
qwen3.5-35b-a3b@deepinfra: 83.5 +/- 2.2, $1.18
qwen3.5-35b-a3b@parasail: 83.5 +/- 2.2, $1.20
mimo-v2.6-pro: 82.2 +/- 4.1, $1.48
gpt-5.6-luna: 63.6 +/- 3.4, $1.48
gpt-5.4-nano: 55.6 +/- 3.5, $1.53
minimax-m3@deepinfra: 90.9 +/- 2.1, $1.55
deepseek-v4.1-flash-1017: 87.0 +/- 3.1, $1.68
deepseek-v4.1-flash-1017@parasail: 87.0 +/- 3.1, $1.68
deepseek-flash: 89.8 +/- 1.7, $1.68
deepseek-flash@alibaba-intl: 89.8 +/- 1.7, $1.68
deepseek-flash@parasail: 89.8 +/- 1.7, $1.68
deepseek-flash@together: 89.8 +/- 1.7, $1.68
minimax-m3: 90.9 +/- 2.1, $1.68
minimax-m3@parasail: 90.9 +/- 2.1, $1.68
qwen3.7-plus: 87.9 +/- 2.3, $1.79
qwen3.7-plus@alibaba-us: 87.9 +/- 2.3, $1.79
qwen3.6-flash: 83.3 +/- 2.7, $1.85
qwen3.6-flash@alibaba-us: 83.3 +/- 2.7, $1.85
gemini-3.1-flash-lite: 74.2 +/- 3.1, $1.85
qwen-plus-2025-01-25: 48.1 +/- 2.7, $1.88
inkling-small: 88.5 +/- 1.9, $1.98
qwen3.8-27b@deepinfra: 70.7 +/- 5.4, $1.99
deepseek-v4-flash-0731@alibaba-intl: 91.0 +/- 1.8, $2.07
gpt-4.1-mini: 65.9 +/- 2.7, $2.24
qwen3.5-35b-a3b: 83.5 +/- 2.2, $2.30
gpt-5-mini: 71.7 +/- 3.2, $2.30
gpt-3.5-turbo: 27.2 +/- 2.0, $2.35
glm-4.7: 83.3 +/- 2.4, $2.38
qwen3.8-27b@parasail: 70.7 +/- 5.4, $2.46
deepseek-flash-fast@parasail: 89.8 +/- 1.7, $2.52
minimax-m3-fast@parasail: 90.9 +/- 2.1, $2.52
qwen3-235b-a22b-thinking-2507: 80.1 +/- 2.6, $2.53
qwen3-30b-a3b-thinking-2507: 70.1 +/- 2.8, $2.56
glm-5.2@deepinfra: 71.2 +/- 3.2, $2.75
qwen3.6-35b-a3b: 83.8 +/- 2.6, $2.78
gemini-3.5-flash-lite: 75.8 +/- 3.1, $2.85
deepseek-r1-0528: 76.3 +/- 2.4, $2.94
llama-3.3-70b-instruct-turbo: 47.4 +/- 2.9, $3.02
mistral-large-4-0: 84.5 +/- 3.0, $3.29
glm-5.3@deepinfra: 90.9 +/- 1.6, $3.38
qwen3.6-27b@deepinfra: 85.9 +/- 2.5, $3.52
qwen3.5-397b-a17b@deepinfra: 85.9 +/- 2.5, $3.60
qwen3.6-plus: 88.4 +/- 2.3, $3.70
gemini-3-flash-preview: 83.2 +/- 1.9, $3.70
qwen3.8-27b: 70.7 +/- 5.4, $3.70
qwen3.5-397b-a17b@parasail: 85.9 +/- 2.5, $4.24
qwen3.6-27b: 85.9 +/- 2.5, $4.44
qwen3.5-397b-a17b: 85.9 +/- 2.5, $4.44
kimi-k2.6: 90.8 +/- 1.7, $4.65
gemini-3.8-flash: 88.0 +/- 3.0, $4.88
gemini-3.7-flash: 94.8 +/- 1.4, $4.88
gemini-3.5-flash: 88.9 +/- 2.2, $4.88
gemini-3.6-flash: 85.9 +/- 2.5, $4.88
deepseek-v4-pro-0813@deepinfra: 91.7 +/- 1.5, $4.94
deepseek-v4-pro@deepinfra: 73.2 +/- 3.2, $4.94
kimi-k2.7-code: 87.9 +/- 2.3, $5.50
gpt-5.4-mini: 64.1 +/- 3.4, $5.55
inkling: 75.6 +/- 3.7, $5.65
o4-mini: 75.3 +/- 3.1, $6.16
o3-mini: 68.2 +/- 3.3, $6.16
deepseek-v4-pro-0813: 91.7 +/- 1.5, $6.20
deepseek-v4-pro: 73.2 +/- 3.2, $6.20
deepseek-v4-pro@parasail: 73.2 +/- 3.2, $6.20
deepseek-v4-pro@together: 73.2 +/- 3.2, $6.20
gpt-6.1-terra: 87.4 +/- 3.0, $6.50
claude-haiku-4-5: 60.5 +/- 2.8, $6.50
glm-5.3: 90.9 +/- 1.6, $6.76
glm-5.3@alibaba-intl: 90.9 +/- 1.6, $6.76
glm-5.3@parasail: 90.9 +/- 1.6, $6.76
glm-5.1: 89.9 +/- 2.2, $6.76
glm-5.2: 71.2 +/- 3.2, $6.76
glm-5.2@alibaba-intl: 71.2 +/- 3.2, $6.76
glm-5.2@alibaba-us: 71.2 +/- 3.2, $6.76
deepseek-v4-pro-fast@parasail: 73.2 +/- 3.2, $9.31
qwen3.8-max-0902: 84.4 +/- 3.3, $9.40
qwen3.6-max-preview: 87.4 +/- 2.4, $9.62
gemini-4-hydrogen: 87.7 +/- 3.8, $9.75
glm-5.3-fast@parasail: 90.9 +/- 1.6, $10.14
grok-4.6: 91.1 +/- 2.4, $10.34
grok-4.7: 88.3 +/- 2.4, $10.34
o3: 79.8 +/- 2.9, $11.20
gpt-4.1: 66.9 +/- 2.8, $11.20
gpt-5: 71.7 +/- 3.2, $11.50
gpt-5.1: 66.7 +/- 3.4, $11.50
qwen3.7-max: 90.9 +/- 2.1, $11.75
qwen3.7-max@alibaba-us: 90.9 +/- 2.1, $11.75
kimi-k3.1-pro@deepinfra: 86.6 +/- 4.2, $12.35
gemini-4-argon: 92.8 +/- 2.5, $13.00
gemini-4-carbon: 93.2 +/- 2.0, $13.00
gpt-6.1-sol: 92.7 +/- 2.3, $13.00
claude-sonnet-5-5: 91.7 +/- 2.4, $13.00
gpt-6-sol: 94.3 +/- 1.5, $13.00
kimi-k3.1-pro: 86.6 +/- 4.2, $13.00
kimi-k3.1-pro@together: 86.6 +/- 4.2, $13.00
claude-sonnet-5: 80.3 +/- 2.8, $13.00
gpt-4o: 47.9 +/- 2.6, $14.00
gemini-3.1-pro-preview: 94.1 +/- 1.7, $14.80
gpt-5.6-terra: 81.9 +/- 2.9, $14.80
gpt-5.2: 73.2 +/- 3.2, $16.10
gpt-5.4: 74.8 +/- 3.1, $18.50
kimi-k3@deepinfra: 85.1 +/- 2.9, $18.53
kimi-k3: 85.1 +/- 2.9, $19.50
kimi-k3@moonshot: 85.1 +/- 2.9, $19.50
claude-sonnet-4-6: 78.8 +/- 2.9, $19.50
claude-sonnet-4-5: 73.7 +/- 3.1, $19.50
claude-opus-5-5: 90.6 +/- 2.4, $26.00
gpt-5.6-sol: 82.8 +/- 2.7, $26.00
claude-opus-5: 92.9 +/- 1.8, $32.50
claude-opus-4-6: 88.4 +/- 2.3, $32.50
claude-opus-4-7: 86.4 +/- 2.5, $32.50
claude-opus-4-8: 85.4 +/- 2.5, $32.50
claude-opus-4-5: 80.7 +/- 2.4, $32.50
gpt-5.5: 77.3 +/- 3.0, $37.00
gpt-4-turbo: 46.6 +/- 2.7, $47.00
claude-fable-5-2: 92.4 +/- 1.9, $52.00
gpt-6-astra: 93.0 +/- 2.2, $65.00
claude-fable-5-1: 90.1 +/- 3.5, $65.00
claude-fable-5: 78.8 +/- 2.9, $65.00
o1: 50.3 +/- 2.8, $84.00
gpt-4: 35.7 +/- 2.4, $114
gemini-3.8-flash Cheap gemini-4-argon Best, Balanced Ranking, Hard 1. gemini-4-argon: 92.8 +/- 2.5 1 gemini-4-argon 92.8 2. gemini-4-carbon: 93.2 +/- 2.0 2 gemini-4-carbon 93.2 3. gemini-3.8-flash: 88.0 +/- 3.0 3 gemini-3.8-flash 88.0 4. gpt-6.1-sol: 92.7 +/- 2.3 4 gpt-6.1-sol 92.7 5. gpt-6-astra: 93.0 +/- 2.2 5 gpt-6-astra 93.0 6. gemini-3.7-flash: 94.8 +/- 1.4 6 gemini-3.7-flash 94.8 7. claude-sonnet-5-5: 91.7 +/- 2.4 7 claude-sonnet-5-5 91.7 8. gemini-3.1-pro-preview: 94.1 +/- 1.7 8 gemini-3.1-pro-preview 94.1 9. claude-opus-5-5: 90.6 +/- 2.4 9 claude-opus-5-5 90.6 10. claude-opus-5: 92.9 +/- 1.8 10 claude-opus-5 92.9 11. claude-fable-5-1: 90.1 +/- 3.5 11 claude-fable-5-1 90.1 12. qwen3.8-max-0902: 84.4 +/- 3.3 12 qwen3.8-max-0902 84.4 13. claude-opus-4-6: 88.4 +/- 2.3 13 claude-opus-4-6 88.4 14. glm-5.3: 90.9 +/- 1.6 14 glm-5.3 90.9 15. glm-5.3-fast@parasail: 90.9 +/- 1.6 15 glm-5.3-fast@parasail 90.9 16. glm-5.3@alibaba-intl: 90.9 +/- 1.6 16 glm-5.3@alibaba-intl 90.9 17. glm-5.3@deepinfra: 90.9 +/- 1.6 17 glm-5.3@deepinfra 90.9 18. glm-5.3@parasail: 90.9 +/- 1.6 18 glm-5.3@parasail 90.9 19. claude-fable-5-2: 92.4 +/- 1.9 19 claude-fable-5-2 92.4 20. grok-4.6: 91.1 +/- 2.4 20 grok-4.6 91.1 21. gpt-6-sol: 94.3 +/- 1.5 21 gpt-6-sol 94.3 22. gemini-4-hydrogen: 87.7 +/- 3.8 22 gemini-4-hydrogen 87.7 23. gpt-6.1-terra: 87.4 +/- 3.0 23 gpt-6.1-terra 87.4 24. claude-opus-4-7: 86.4 +/- 2.5 24 claude-opus-4-7 86.4 25. qwen3.7-max: 90.9 +/- 2.1 25 qwen3.7-max 90.9 26. qwen3.7-max@alibaba-us: 90.9 +/- 2.1 26 qwen3.7-max@alibaba-us 90.9 27. deepseek-v4.1-flash-1017: 87.0 +/- 3.1 27 deepseek-v4.1-flash-1017 87.0 28. deepseek-v4.1-flash-1017@deepinfra: 87.0 +/- 3.1 28 deepseek-v4.1-flash-1017@deepinfra 87.0 29. deepseek-v4.1-flash-1017@parasail: 87.0 +/- 3.1 29 deepseek-v4.1-flash-1017@parasail 87.0 30. deepseek-flash: 89.8 +/- 1.7 30 deepseek-flash 89.8 31. deepseek-flash-fast@parasail: 89.8 +/- 1.7 31 deepseek-flash-fast@parasail 89.8 32. deepseek-flash@alibaba-intl: 89.8 +/- 1.7 32 deepseek-flash@alibaba-intl 89.8 33. deepseek-flash@deepinfra: 89.8 +/- 1.7 33 deepseek-flash@deepinfra 89.8 34. deepseek-flash@parasail: 89.8 +/- 1.7 34 deepseek-flash@parasail 89.8 35. deepseek-flash@together: 89.8 +/- 1.7 35 deepseek-flash@together 89.8 36. kimi-k3.1-pro: 86.6 +/- 4.2 36 kimi-k3.1-pro 86.6 37. kimi-k3.1-pro@deepinfra: 86.6 +/- 4.2 37 kimi-k3.1-pro@deepinfra 86.6 38. kimi-k3.1-pro@together: 86.6 +/- 4.2 38 kimi-k3.1-pro@together 86.6 39. glm-5.3-flash: 80.6 +/- 3.3 39 glm-5.3-flash 80.6 40. glm-5.3-flash@deepinfra: 80.6 +/- 3.3 40 glm-5.3-flash@deepinfra 80.6 41. deepseek-v4-pro-0813: 91.7 +/- 1.5 41 deepseek-v4-pro-0813 91.7 42. deepseek-v4-pro-0813@deepinfra: 91.7 +/- 1.5 42 deepseek-v4-pro-0813@deepinfra 91.7 43. kimi-k3: 85.1 +/- 2.9 43 kimi-k3 85.1 44. kimi-k3@deepinfra: 85.1 +/- 2.9 44 kimi-k3@deepinfra 85.1 45. kimi-k3@moonshot: 85.1 +/- 2.9 45 kimi-k3@moonshot 85.1 46. kimi-k2.6: 90.8 +/- 1.7 46 kimi-k2.6 90.8 47. gemini-3.5-flash: 88.9 +/- 2.2 47 gemini-3.5-flash 88.9 48. glm-5.1: 89.9 +/- 2.2 48 glm-5.1 89.9 49. grok-4.7: 88.3 +/- 2.4 49 grok-4.7 88.3 50. deepseek-v4-flash-0731: 91.0 +/- 1.8 50 deepseek-v4-flash-0731 91.0 51. deepseek-v4-flash-0731@alibaba-intl: 91.0 +/- 1.8 51 deepseek-v4-flash-0731@alibaba-intl 91.0 52. deepseek-v4-flash-0731@deepinfra: 91.0 +/- 1.8 52 deepseek-v4-flash-0731@deepinfra 91.0 53. deepseek-v4-flash-0731@parasail: 91.0 +/- 1.8 53 deepseek-v4-flash-0731@parasail 91.0 54. claude-opus-4-8: 85.4 +/- 2.5 54 claude-opus-4-8 85.4 55. mimo-v2.6-pro: 82.2 +/- 4.1 55 mimo-v2.6-pro 82.2 56. gemini-3.6-flash: 85.9 +/- 2.5 56 gemini-3.6-flash 85.9 57. gpt-5.6-sol: 82.8 +/- 2.7 57 gpt-5.6-sol 82.8 58. qwen3.6-max-preview: 87.4 +/- 2.4 58 qwen3.6-max-preview 87.4 59. minimax-m3: 90.9 +/- 2.1 59 minimax-m3 90.9 60. minimax-m3-fast@parasail: 90.9 +/- 2.1 60 minimax-m3-fast@parasail 90.9 61. minimax-m3@deepinfra: 90.9 +/- 2.1 61 minimax-m3@deepinfra 90.9 62. minimax-m3@parasail: 90.9 +/- 2.1 62 minimax-m3@parasail 90.9 63. qwen3.7-plus: 87.9 +/- 2.3 63 qwen3.7-plus 87.9 64. qwen3.7-plus@alibaba-us: 87.9 +/- 2.3 64 qwen3.7-plus@alibaba-us 87.9 65. claude-fable-5: 78.8 +/- 2.9 65 claude-fable-5 78.8 66. qwen3.6-plus: 88.4 +/- 2.3 66 qwen3.6-plus 88.4 67. kimi-k2.7-code: 87.9 +/- 2.3 67 kimi-k2.7-code 87.9 68. mistral-large-4-0: 84.5 +/- 3.0 68 mistral-large-4-0 84.5 69. gpt-6-luna: 73.6 +/- 3.7 69 gpt-6-luna 73.6 70. gemini-3-flash-preview: 83.2 +/- 1.9 70 gemini-3-flash-preview 83.2 71. claude-sonnet-4-6: 78.8 +/- 2.9 71 claude-sonnet-4-6 78.8 72. claude-haiku-5-5: 85.5 +/- 2.8 72 claude-haiku-5-5 85.5 73. claude-opus-4-5: 80.7 +/- 2.4 73 claude-opus-4-5 80.7 74. qwen3.6-27b: 85.9 +/- 2.5 74 qwen3.6-27b 85.9 75. qwen3.6-27b@deepinfra: 85.9 +/- 2.5 75 qwen3.6-27b@deepinfra 85.9 76. qwen3.5-397b-a17b: 85.9 +/- 2.5 76 qwen3.5-397b-a17b 85.9 77. qwen3.5-397b-a17b@deepinfra: 85.9 +/- 2.5 77 qwen3.5-397b-a17b@deepinfra 85.9 78. qwen3.5-397b-a17b@parasail: 85.9 +/- 2.5 78 qwen3.5-397b-a17b@parasail 85.9 79. inkling: 75.6 +/- 3.7 79 inkling 75.6 80. claude-sonnet-5: 80.3 +/- 2.8 80 claude-sonnet-5 80.3 81. gpt-5.5: 77.3 +/- 3.0 81 gpt-5.5 77.3 82. gpt-5.6-terra: 81.9 +/- 2.9 82 gpt-5.6-terra 81.9 84. qwen3.6-35b-a3b: 83.8 +/- 2.6 84 qwen3.6-35b-a3b 83.8 85. qwen3.6-35b-a3b@deepinfra: 83.8 +/- 2.6 85 qwen3.6-35b-a3b@deepinfra 83.8 86. qwen3.6-35b-a3b@parasail: 83.8 +/- 2.6 86 qwen3.6-35b-a3b@parasail 83.8 87. glm-4.7: 83.3 +/- 2.4 87 glm-4.7 83.3 88. qwen3.6-flash: 83.3 +/- 2.7 88 qwen3.6-flash 83.3 89. qwen3.6-flash@alibaba-us: 83.3 +/- 2.7 89 qwen3.6-flash@alibaba-us 83.3 90. minimax-m3.1-flash: 79.8 +/- 4.7 90 minimax-m3.1-flash 79.8 91. minimax-m3.1-flash@parasail: 79.8 +/- 4.7 91 minimax-m3.1-flash@parasail 79.8 92. inkling-small: 88.5 +/- 1.9 92 inkling-small 88.5 93. gpt-5.4: 74.8 +/- 3.1 93 gpt-5.4 74.8 95. glm-5.2: 71.2 +/- 3.2 95 glm-5.2 71.2 96. glm-5.2@alibaba-intl: 71.2 +/- 3.2 96 glm-5.2@alibaba-intl 71.2 97. glm-5.2@alibaba-us: 71.2 +/- 3.2 97 glm-5.2@alibaba-us 71.2 98. glm-5.2@deepinfra: 71.2 +/- 3.2 98 glm-5.2@deepinfra 71.2 99. gemini-3.5-flash-lite: 75.8 +/- 3.1 99 gemini-3.5-flash-lite 75.8 103. claude-sonnet-4-5: 73.7 +/- 3.1 103 claude-sonnet-4-5 73.7 104. deepseek-v4-pro: 73.2 +/- 3.2 104 deepseek-v4-pro 73.2 105. deepseek-v4-pro-fast@parasail: 73.2 +/- 3.2 105 deepseek-v4-pro-fast@parasail 73.2 106. deepseek-v4-pro@deepinfra: 73.2 +/- 3.2 106 deepseek-v4-pro@deepinfra 73.2 107. deepseek-v4-pro@parasail: 73.2 +/- 3.2 107 deepseek-v4-pro@parasail 73.2 108. deepseek-v4-pro@together: 73.2 +/- 3.2 108 deepseek-v4-pro@together 73.2 109. qwen3.8-27b: 70.7 +/- 5.4 109 qwen3.8-27b 70.7 110. qwen3.8-27b@deepinfra: 70.7 +/- 5.4 110 qwen3.8-27b@deepinfra 70.7 111. qwen3.8-27b@parasail: 70.7 +/- 5.4 111 qwen3.8-27b@parasail 70.7 112. qwen3.5-9b: 79.0 +/- 2.4 112 qwen3.5-9b 79.0 113. qwen3.5-9b@deepinfra: 79.0 +/- 2.4 113 qwen3.5-9b@deepinfra 79.0 114. qwen3.5-9b@parasail: 79.0 +/- 2.4 114 qwen3.5-9b@parasail 79.0 115. gemma-4-31b-it: 61.7 +/- 3.6 115 gemma-4-31b-it 61.7 116. gpt-5.2: 73.2 +/- 3.2 116 gpt-5.2 73.2 121. o3: 79.8 +/- 2.9 121 o3 79.8 123. qwen3.5-35b-a3b: 83.5 +/- 2.2 123 qwen3.5-35b-a3b 83.5 124. qwen3.5-35b-a3b@deepinfra: 83.5 +/- 2.2 124 qwen3.5-35b-a3b@deepinfra 83.5 125. qwen3.5-35b-a3b@parasail: 83.5 +/- 2.2 125 qwen3.5-35b-a3b@parasail 83.5 127. qwen3.5-flash-2026-02-23: 82.3 +/- 2.7 127 qwen3.5-flash-2026-02-23 82.3 130. gpt-5.6-luna: 63.6 +/- 3.4 130 gpt-5.6-luna 63.6 134. gpt-5.4-mini: 64.1 +/- 3.4 134 gpt-5.4-mini 64.1 135. gemini-3.1-flash-lite: 74.2 +/- 3.1 135 gemini-3.1-flash-lite 74.2 136. gemma-4-26b-a4b-it: 73.2 +/- 3.2 136 gemma-4-26b-a4b-it 73.2 137. qwen3-235b-a22b-thinking-2507: 80.1 +/- 2.6 137 qwen3-235b-a22b-thinking-2507 80.1 140. gpt-5: 71.7 +/- 3.2 140 gpt-5 71.7 141. gpt-5.1: 66.7 +/- 3.4 141 gpt-5.1 66.7 142. deepseek-r1-0528: 76.3 +/- 2.4 142 deepseek-r1-0528 76.3 145. qwen3-30b-a3b-thinking-2507: 70.1 +/- 2.8 145 qwen3-30b-a3b-thinking-2507 70.1 150. o4-mini: 75.3 +/- 3.1 150 o4-mini 75.3 151. gpt-4.1: 66.9 +/- 2.8 151 gpt-4.1 66.9 152. qwen3-14b: 63.8 +/- 2.9 152 qwen3-14b 63.8 153. claude-haiku-4-5: 60.5 +/- 2.8 153 claude-haiku-4-5 60.5 154. gpt-oss-120b: 75.8 +/- 2.7 154 gpt-oss-120b 75.8 155. gpt-oss-120b@deepinfra: 75.8 +/- 2.7 155 gpt-oss-120b@deepinfra 75.8 156. gpt-5-mini: 71.7 +/- 3.2 156 gpt-5-mini 71.7 158. o3-mini: 68.2 +/- 3.3 158 o3-mini 68.2 159. gpt-4.1-mini: 65.9 +/- 2.7 159 gpt-4.1-mini 65.9 160. qwen3-32b: 65.7 +/- 2.8 160 qwen3-32b 65.7 161. gpt-5.4-nano: 55.6 +/- 3.5 161 gpt-5.4-nano 55.6 162. o1: 50.3 +/- 2.8 162 o1 50.3 163. qwen3-30b-a3b-instruct-2507: 55.6 +/- 2.9 163 qwen3-30b-a3b-instruct-2507 55.6 164. qwen3-30b-a3b: 61.7 +/- 3.0 164 qwen3-30b-a3b 61.7 166. gpt-5-nano: 57.6 +/- 3.5 166 gpt-5-nano 57.6 168. gemma-3-27b-it: 47.7 +/- 2.7 168 gemma-3-27b-it 47.7 169. qwen-plus-2025-01-25: 48.1 +/- 2.7 169 qwen-plus-2025-01-25 48.1 170. llama-4-scout-17b-16e-instruct: 51.8 +/- 3.2 170 llama-4-scout-17b-16e-instruct 51.8 171. gpt-oss-20b: 53.2 +/- 2.5 171 gpt-oss-20b 53.2 172. gpt-4o: 47.9 +/- 2.6 172 gpt-4o 47.9 173. gpt-4.1-nano: 48.9 +/- 2.5 173 gpt-4.1-nano 48.9 174. phi-4: 56.1 +/- 2.6 174 phi-4 56.1 175. llama-3.3-70b-instruct-turbo: 47.4 +/- 2.9 175 llama-3.3-70b-instruct-turbo 47.4 176. llama-3.3-70b-instruct-turbo@deepinfra: 47.4 +/- 2.9 176 llama-3.3-70b-instruct-turbo@deepinfra 47.4 177. llama-3.3-70b-instruct-turbo@parasail: 47.4 +/- 2.9 177 llama-3.3-70b-instruct-turbo@parasail 47.4 178. gpt-4-turbo: 46.6 +/- 2.7 178 gpt-4-turbo 46.6 179. gemma-3-12b-it: 39.5 +/- 2.5 179 gemma-3-12b-it 39.5 180. gpt-4o-mini: 37.7 +/- 2.4 180 gpt-4o-mini 37.7 181. gpt-4: 35.7 +/- 2.4 181 gpt-4 35.7 182. gemma-3-4b-it: 23.2 +/- 1.9 182 gemma-3-4b-it 23.2 183. gpt-3.5-turbo: 27.2 +/- 2.0 183 gpt-3.5-turbo 27.2 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Rating only, not on the chart (27) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
mistral-small-24b-instruct-2501Rating 1,284.9 $0.17 deepseek-v4-flash@deepinfraRating 1,458.8 $0.34 granite-4.2-8bRating 1,304.7 $0.35 mimo-v2.6-flashRating 1,472.1 $0.53 deepseek-v4-flash@parasailRating 1,458.8 $0.53 qwen3-235b-a22b-instruct-2507@deepinfraRating 1,447.1 $0.68 deepseek-v4-flashRating 1,458.8 $0.76 deepseek-v3.2Rating 1,447.2 $0.86 qwen3-235b-a22b-instruct-2507@parasailRating 1,447.1 $1.00 qwen3-next-80b-a3b-instruct@deepinfraRating 1,419.5 $1.17 qwen3-next-80b-a3b-instruct@parasailRating 1,419.5 $1.19 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,437.9 $1.19 qwen3-235b-a22b-instruct-2507Rating 1,447.1 $1.29 qwen3-next-80b-a3b-instructRating 1,419.5 $1.38 qwen3-next-80b-a3b-thinkingRating 1,385.3 $1.38 inkling-2-small@deepinfraRating 1,438.3 $1.43 inkling-2-smallRating 1,438.3 $1.50 beam-1.0-501b-a23bRating 1,461.3 $1.88 qwen3-vl-235b-a22b-instruct@parasailRating 1,437.9 $2.13 qwen3-vl-235b-a22b-instructRating 1,437.9 $2.24 mistral-large-2512Rating 1,433.0 $2.35 qwen3.5-27bRating 1,427.4 $2.76 glm-4.6Rating 1,441.9 $2.80 qwen3.5-27b@deepinfraRating 1,427.4 $2.86 qwen3.5-122b-a10bRating 1,432.6 $3.68 qwen3-vl-235b-a22b-thinkingRating 1,417.5 $4.40 mistral-medium-3-5Rating 1,445.0 $9.75 Price against score, Hardest 0 25 50 75 100 $1 $10 $100
deepseek-v4-flash-0731@deepinfra: 71.5 +/- 2.4, $0.45
deepseek-v4.1-flash-1017@deepinfra: 81.6 +/- 2.2, $0.74
gpt-5-nano: 9.6 +/- 1.2, $0.75
llama-4-scout-17b-16e-instruct: 0.3 +/- 0.3, $0.75
deepseek-v4-flash-0731: 71.5 +/- 2.4, $0.84
deepseek-v4-flash-0731@parasail: 71.5 +/- 2.4, $0.84
gpt-4.1-nano: 0.0 +/- 0.2, $0.90
deepseek-flash@deepinfra: 78.0 +/- 2.3, $1.05
gpt-6-luna: 50.9 +/- 2.4, $1.05
claude-haiku-5-5: 45.3 +/- 1.6, $1.05
qwen3-235b-a22b-instruct-2507@deepinfra: 6.1 +/- 0.9, $1.10
glm-5.3-flash: 61.0 +/- 2.5, $1.20
glm-5.3-flash@deepinfra: 61.0 +/- 2.5, $1.20
deepseek-v3.2: 30.5 +/- 1.5, $1.35
gpt-4o-mini: 0.0 +/- 0.4, $1.35
qwen3-235b-a22b-instruct-2507@parasail: 6.1 +/- 0.9, $1.62
qwen3-235b-a22b-instruct-2507: 6.1 +/- 0.9, $2.07
inkling-2-small@deepinfra: 89.1 +/- 1.8, $2.28
inkling-2-small: 89.1 +/- 1.8, $2.40
gpt-5.6-luna: 52.9 +/- 2.3, $2.40
gpt-5.4-nano: 20.9 +/- 1.5, $2.48
deepseek-v4.1-flash-1017: 81.6 +/- 2.2, $2.70
deepseek-v4.1-flash-1017@parasail: 81.6 +/- 2.2, $2.70
deepseek-flash: 78.0 +/- 2.3, $2.70
deepseek-flash@alibaba-intl: 78.0 +/- 2.3, $2.70
deepseek-flash@parasail: 78.0 +/- 2.3, $2.70
deepseek-flash@together: 78.0 +/- 2.3, $2.70
gemini-3.1-flash-lite: 8.6 +/- 1.1, $3.00
inkling-small: 55.5 +/- 2.4, $3.15
qwen3.8-27b@deepinfra: 17.8 +/- 1.3, $3.26
deepseek-v4-flash-0731@alibaba-intl: 71.5 +/- 2.4, $3.30
gpt-4.1-mini: 1.8 +/- 0.5, $3.60
gpt-5-mini: 25.2 +/- 1.1, $3.75
qwen3.8-27b@parasail: 17.8 +/- 1.3, $4.02
deepseek-flash-fast@parasail: 78.0 +/- 2.3, $4.05
glm-5.2@deepinfra: 49.9 +/- 2.2, $4.39
gemini-3.5-flash-lite: 31.9 +/- 1.9, $4.65
deepseek-r1-0528: 11.2 +/- 1.1, $4.73
gemini-3-flash-preview: 11.4 +/- 1.1, $6.00
qwen3.8-27b: 17.8 +/- 1.3, $6.00
deepseek-v4-pro-0813@deepinfra: 73.4 +/- 2.4, $7.80
gemini-3.8-flash: 73.3 +/- 1.2, $7.88
gemini-3.7-flash: 90.0 +/- 1.7, $7.88
gemini-3.5-flash: 82.3 +/- 2.2, $7.88
gemini-3.6-flash: 75.8 +/- 2.3, $7.88
gpt-5.4-mini: 35.6 +/- 2.0, $9.00
inkling: 58.0 +/- 2.4, $9.08
deepseek-v4-pro-0813: 73.4 +/- 2.4, $9.90
o4-mini: 27.6 +/- 1.2, $9.90
o3-mini: 18.8 +/- 1.4, $9.90
claude-haiku-4-5: 7.8 +/- 1.0, $10.50
glm-5.2: 49.9 +/- 2.2, $10.80
glm-5.2@alibaba-intl: 49.9 +/- 2.2, $10.80
glm-5.2@alibaba-us: 49.9 +/- 2.2, $10.80
grok-4.6: 76.1 +/- 2.3, $16.50
grok-4.7: 74.3 +/- 2.4, $16.50
o3: 29.2 +/- 1.2, $18.00
gpt-4.1: 3.8 +/- 0.5, $18.00
gpt-5: 32.6 +/- 1.3, $18.75
gpt-5.1: 36.3 +/- 1.4, $18.75
gemini-4-argon: 94.9 +/- 1.3, $21.00
gemini-4-carbon: 94.8 +/- 1.3, $21.00
gpt-6.1-sol: 95.1 +/- 1.3, $21.00
claude-sonnet-5-5: 68.5 +/- 1.6, $21.00
gpt-6-sol: 80.0 +/- 2.2, $21.00
gpt-4o: 2.4 +/- 0.4, $22.50
gemini-3.1-pro-preview: 73.8 +/- 1.5, $24.00
gpt-5.6-terra: 79.5 +/- 2.3, $24.00
gpt-5.2: 47.2 +/- 1.6, $26.25
kimi-k3@deepinfra: 70.8 +/- 2.4, $29.93
gpt-5.4: 65.5 +/- 1.6, $30.00
kimi-k3: 70.8 +/- 2.4, $31.50
kimi-k3@moonshot: 70.8 +/- 2.4, $31.50
claude-sonnet-4-6: 73.5 +/- 2.4, $31.50
claude-sonnet-4-5: 12.3 +/- 1.0, $31.50
claude-opus-5-5: 95.9 +/- 1.2, $42.00
gpt-5.6-sol: 91.2 +/- 1.7, $42.00
claude-opus-5: 92.9 +/- 1.5, $52.50
claude-opus-4-7: 62.3 +/- 1.5, $52.50
claude-opus-4-8: 82.0 +/- 2.2, $52.50
claude-opus-4-6: 51.8 +/- 1.5, $52.50
claude-opus-4-5: 20.7 +/- 1.2, $52.50
gpt-5.5: 88.9 +/- 1.8, $60.00
claude-fable-5-2: 94.0 +/- 1.4, $84.00
claude-fable-5: 91.5 +/- 1.6, $105
claude-fable-5-1: 77.1 +/- 1.2, $105
gpt-6-astra: 78.4 +/- 1.1, $105
o1: 13.0 +/- 1.1, $135
gemini-4-argon Best, Balanced Ranking, Hardest 1. claude-fable-5-2: 94.0 +/- 1.4 1 claude-fable-5-2 94.0 2. gemini-4-argon: 94.9 +/- 1.3 2 gemini-4-argon 94.9 3. gemini-4-carbon: 94.8 +/- 1.3 3 gemini-4-carbon 94.8 4. claude-opus-5-5: 95.9 +/- 1.2 4 claude-opus-5-5 95.9 5. claude-fable-5: 91.5 +/- 1.6 5 claude-fable-5 91.5 6. gemini-3.8-flash: 73.3 +/- 1.2 6 gemini-3.8-flash 73.3 7. claude-fable-5-1: 77.1 +/- 1.2 7 claude-fable-5-1 77.1 8. claude-opus-5: 92.9 +/- 1.5 8 claude-opus-5 92.9 9. gpt-5.6-sol: 91.2 +/- 1.7 9 gpt-5.6-sol 91.2 10. gpt-6.1-sol: 95.1 +/- 1.3 10 gpt-6.1-sol 95.1 11. gpt-6-astra: 78.4 +/- 1.1 11 gpt-6-astra 78.4 12. gemini-3.7-flash: 90.0 +/- 1.7 12 gemini-3.7-flash 90.0 13. claude-opus-4-7: 62.3 +/- 1.5 13 claude-opus-4-7 62.3 14. gemini-3.1-pro-preview: 73.8 +/- 1.5 14 gemini-3.1-pro-preview 73.8 15. gpt-5.5: 88.9 +/- 1.8 15 gpt-5.5 88.9 17. deepseek-v4.1-flash-1017: 81.6 +/- 2.2 17 deepseek-v4.1-flash-1017 81.6 18. deepseek-v4.1-flash-1017@deepinfra: 81.6 +/- 2.2 18 deepseek-v4.1-flash-1017@deepinfra 81.6 19. deepseek-v4.1-flash-1017@parasail: 81.6 +/- 2.2 19 deepseek-v4.1-flash-1017@parasail 81.6 20. claude-opus-4-8: 82.0 +/- 2.2 20 claude-opus-4-8 82.0 21. claude-opus-4-6: 51.8 +/- 1.5 21 claude-opus-4-6 51.8 22. claude-sonnet-5-5: 68.5 +/- 1.6 22 claude-sonnet-5-5 68.5 28. deepseek-flash: 78.0 +/- 2.3 28 deepseek-flash 78.0 29. deepseek-flash-fast@parasail: 78.0 +/- 2.3 29 deepseek-flash-fast@parasail 78.0 30. deepseek-flash@alibaba-intl: 78.0 +/- 2.3 30 deepseek-flash@alibaba-intl 78.0 31. deepseek-flash@deepinfra: 78.0 +/- 2.3 31 deepseek-flash@deepinfra 78.0 32. deepseek-flash@parasail: 78.0 +/- 2.3 32 deepseek-flash@parasail 78.0 33. deepseek-flash@together: 78.0 +/- 2.3 33 deepseek-flash@together 78.0 35. kimi-k3: 70.8 +/- 2.4 35 kimi-k3 70.8 36. kimi-k3@deepinfra: 70.8 +/- 2.4 36 kimi-k3@deepinfra 70.8 37. kimi-k3@moonshot: 70.8 +/- 2.4 37 kimi-k3@moonshot 70.8 38. gpt-5.6-terra: 79.5 +/- 2.3 38 gpt-5.6-terra 79.5 39. inkling-2-small: 89.1 +/- 1.8 39 inkling-2-small 89.1 40. inkling-2-small@deepinfra: 89.1 +/- 1.8 40 inkling-2-small@deepinfra 89.1 46. gpt-6-sol: 80.0 +/- 2.2 46 gpt-6-sol 80.0 47. gemini-3.5-flash: 82.3 +/- 2.2 47 gemini-3.5-flash 82.3 48. gemini-3.6-flash: 75.8 +/- 2.3 48 gemini-3.6-flash 75.8 49. claude-sonnet-4-6: 73.5 +/- 2.4 49 claude-sonnet-4-6 73.5 52. gpt-5.4: 65.5 +/- 1.6 52 gpt-5.4 65.5 54. grok-4.6: 76.1 +/- 2.3 54 grok-4.6 76.1 55. glm-5.3-flash: 61.0 +/- 2.5 55 glm-5.3-flash 61.0 56. glm-5.3-flash@deepinfra: 61.0 +/- 2.5 56 glm-5.3-flash@deepinfra 61.0 61. deepseek-v4-flash-0731: 71.5 +/- 2.4 61 deepseek-v4-flash-0731 71.5 62. deepseek-v4-flash-0731@alibaba-intl: 71.5 +/- 2.4 62 deepseek-v4-flash-0731@alibaba-intl 71.5 63. deepseek-v4-flash-0731@deepinfra: 71.5 +/- 2.4 63 deepseek-v4-flash-0731@deepinfra 71.5 64. deepseek-v4-flash-0731@parasail: 71.5 +/- 2.4 64 deepseek-v4-flash-0731@parasail 71.5 65. grok-4.7: 74.3 +/- 2.4 65 grok-4.7 74.3 66. deepseek-v4-pro-0813: 73.4 +/- 2.4 66 deepseek-v4-pro-0813 73.4 67. deepseek-v4-pro-0813@deepinfra: 73.4 +/- 2.4 67 deepseek-v4-pro-0813@deepinfra 73.4 68. glm-5.2: 49.9 +/- 2.2 68 glm-5.2 49.9 69. glm-5.2@alibaba-intl: 49.9 +/- 2.2 69 glm-5.2@alibaba-intl 49.9 70. glm-5.2@alibaba-us: 49.9 +/- 2.2 70 glm-5.2@alibaba-us 49.9 71. glm-5.2@deepinfra: 49.9 +/- 2.2 71 glm-5.2@deepinfra 49.9 75. gpt-5.6-luna: 52.9 +/- 2.3 75 gpt-5.6-luna 52.9 84. claude-opus-4-5: 20.7 +/- 1.2 84 claude-opus-4-5 20.7 85. gpt-6-luna: 50.9 +/- 2.4 85 gpt-6-luna 50.9 88. inkling: 58.0 +/- 2.4 88 inkling 58.0 93. gemini-3-flash-preview: 11.4 +/- 1.1 93 gemini-3-flash-preview 11.4 94. gpt-5.4-mini: 35.6 +/- 2.0 94 gpt-5.4-mini 35.6 95. claude-haiku-5-5: 45.3 +/- 1.6 95 claude-haiku-5-5 45.3 96. gpt-5.2: 47.2 +/- 1.6 96 gpt-5.2 47.2 100. claude-sonnet-4-5: 12.3 +/- 1.0 100 claude-sonnet-4-5 12.3 101. gemini-3.5-flash-lite: 31.9 +/- 1.9 101 gemini-3.5-flash-lite 31.9 103. gpt-5: 32.6 +/- 1.3 103 gpt-5 32.6 105. qwen3.8-27b: 17.8 +/- 1.3 105 qwen3.8-27b 17.8 106. qwen3.8-27b@deepinfra: 17.8 +/- 1.3 106 qwen3.8-27b@deepinfra 17.8 107. qwen3.8-27b@parasail: 17.8 +/- 1.3 107 qwen3.8-27b@parasail 17.8 111. gpt-5.1: 36.3 +/- 1.4 111 gpt-5.1 36.3 116. inkling-small: 55.5 +/- 2.4 116 inkling-small 55.5 119. o3: 29.2 +/- 1.2 119 o3 29.2 120. deepseek-v3.2: 30.5 +/- 1.5 120 deepseek-v3.2 30.5 126. gemini-3.1-flash-lite: 8.6 +/- 1.1 126 gemini-3.1-flash-lite 8.6 127. claude-haiku-4-5: 7.8 +/- 1.0 127 claude-haiku-4-5 7.8 132. gpt-5.4-nano: 20.9 +/- 1.5 132 gpt-5.4-nano 20.9 134. qwen3-235b-a22b-instruct-2507: 6.1 +/- 0.9 134 qwen3-235b-a22b-instruct-2507 6.1 135. qwen3-235b-a22b-instruct-2507@deepinfra: 6.1 +/- 0.9 135 qwen3-235b-a22b-instruct-2507@deepinfra 6.1 136. qwen3-235b-a22b-instruct-2507@parasail: 6.1 +/- 0.9 136 qwen3-235b-a22b-instruct-2507@parasail 6.1 141. o4-mini: 27.6 +/- 1.2 141 o4-mini 27.6 142. gpt-5-mini: 25.2 +/- 1.1 142 gpt-5-mini 25.2 145. o1: 13.0 +/- 1.1 145 o1 13.0 146. deepseek-r1-0528: 11.2 +/- 1.1 146 deepseek-r1-0528 11.2 154. o3-mini: 18.8 +/- 1.4 154 o3-mini 18.8 158. gpt-4.1: 3.8 +/- 0.5 158 gpt-4.1 3.8 160. gpt-5-nano: 9.6 +/- 1.2 160 gpt-5-nano 9.6 161. gpt-4.1-mini: 1.8 +/- 0.5 161 gpt-4.1-mini 1.8 162. gpt-4o-mini: 0.0 +/- 0.4 162 gpt-4o-mini 0.0 163. gpt-4.1-nano: 0.0 +/- 0.2 163 gpt-4.1-nano 0.0 164. llama-4-scout-17b-16e-instruct: 0.3 +/- 0.3 164 llama-4-scout-17b-16e-instruct 0.3 165. gpt-4o: 2.4 +/- 0.4 165 gpt-4o 2.4 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Rating only, not on the chart (77) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
mistral-small-24b-instruct-2501Rating 1,258.8 $0.27 gpt-oss-20bRating 1,315.9 $0.30 gemma-3-4b-itRating 1,260.7 $0.30 gpt-oss-120b@deepinfraRating 1,357.2 $0.37 gemma-3-12b-itRating 1,274.3 $0.38 phi-4Rating 1,268.5 $0.42 gemma-3-27b-itRating 1,338.8 $0.48 deepseek-v4-flash@deepinfraRating 1,457.5 $0.54 granite-4.2-8bRating 1,369.6 $0.56 qwen3-32bRating 1,397.7 $0.66 llama-3.3-70b-instruct-turbo@deepinfraRating 1,298.2 $0.78 mimo-v2.6-flashRating 1,495.2 $0.84 deepseek-v4-flash@parasailRating 1,457.5 $0.84 qwen3.5-flash-2026-02-23Rating 1,419.5 $0.90 qwen3-30b-a3bRating 1,343.2 $1.11 gemma-4-31b-itRating 1,498.0 $1.20 deepseek-v4-flashRating 1,457.5 $1.20 gpt-oss-120bRating 1,357.2 $1.35 llama-3.3-70b-instruct-turbo@parasailRating 1,298.2 $1.41 qwen3-30b-a3b-instruct-2507Rating 1,395.6 $1.80 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,441.6 $1.92 qwen3.5-35b-a3b@deepinfraRating 1,420.1 $1.92 qwen3-next-80b-a3b-instruct@deepinfraRating 1,400.8 $1.92 qwen3.5-35b-a3b@parasailRating 1,420.1 $1.95 qwen3-next-80b-a3b-instruct@parasailRating 1,400.8 $1.95 qwen3-next-80b-a3b-instructRating 1,400.8 $2.25 qwen3-next-80b-a3b-thinkingRating 1,392.1 $2.25 mimo-v2.6-proRating 1,525.8 $2.34 minimax-m3@deepinfraRating 1,467.5 $2.49 minimax-m3Rating 1,467.5 $2.70 minimax-m3@parasailRating 1,467.5 $2.70 qwen3.7-plusRating 1,484.8 $2.88 qwen3.7-plus@alibaba-usRating 1,484.8 $2.88 beam-1.0-501b-a23bRating 1,481.5 $3.00 qwen-plus-2025-01-25Rating 1,369.8 $3.00 qwen3-vl-235b-a22b-instruct@parasailRating 1,441.6 $3.48 qwen3-vl-235b-a22b-instructRating 1,441.6 $3.60 mistral-large-2512Rating 1,428.3 $3.75 qwen3.5-35b-a3bRating 1,420.1 $3.75 gpt-3.5-turboRating 1,216.1 $3.75 glm-4.7Rating 1,449.2 $3.83 minimax-m3-fast@parasailRating 1,467.5 $4.05 qwen3-235b-a22b-thinking-2507Rating 1,468.1 $4.14 qwen3.5-27bRating 1,440.8 $4.50 glm-4.6Rating 1,440.1 $4.50 qwen3.5-27b@deepinfraRating 1,440.8 $4.68 llama-3.3-70b-instruct-turboRating 1,298.2 $4.68 glm-5.3@deepinfraRating 1,522.3 $5.44 qwen3.5-397b-a17b@deepinfraRating 1,477.8 $5.85 qwen3.6-plusRating 1,475.0 $6.00 qwen3.5-122b-a10bRating 1,441.4 $6.00 qwen3.5-397b-a17b@parasailRating 1,477.8 $6.90 qwen3.5-397b-a17bRating 1,477.8 $7.20 qwen3-vl-235b-a22b-thinkingRating 1,440.7 $7.20 kimi-k2.6Rating 1,503.3 $7.50 deepseek-v4-pro@deepinfraRating 1,479.5 $7.80 deepseek-v4-proRating 1,479.5 $9.90 deepseek-v4-pro@parasailRating 1,479.5 $9.90 deepseek-v4-pro@togetherRating 1,479.5 $9.90 glm-5.3Rating 1,522.3 $10.80 glm-5.3@alibaba-intlRating 1,522.3 $10.80 glm-5.3@parasailRating 1,522.3 $10.80 glm-5.1Rating 1,495.6 $10.80 deepseek-v4-pro-fast@parasailRating 1,479.5 $14.85 qwen3.8-max-0902Rating 1,514.4 $15.00 qwen3.6-max-previewRating 1,497.4 $15.60 gemini-4-hydrogenRating 1,542.1 $15.75 mistral-medium-3-5Rating 1,434.4 $15.75 glm-5.3-fast@parasailRating 1,522.3 $16.20 qwen3.7-maxRating 1,508.6 $18.75 qwen3.7-max@alibaba-usRating 1,508.6 $18.75 kimi-k3.1-pro@deepinfraRating 1,515.3 $19.95 kimi-k3.1-proRating 1,515.3 $21.00 kimi-k3.1-pro@togetherRating 1,515.3 $21.00 claude-sonnet-5Rating 1,508.9 $21.00 gpt-4-turboRating 1,297.3 $75.00 gpt-4Rating 1,265.2 $180 Math Nobody publishes a scored result for this one yet. The models below have a preference rating only.
Ranking, Medium Rating only, not on the chart (150) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
mistral-small-24b-instruct-2501Rating 1,261.5 $0.07 gpt-oss-20bRating 1,335.0 $0.08 gemma-3-4b-itRating 1,254.1 $0.08 gpt-oss-120b@deepinfraRating 1,380.8 $0.10 gemma-3-12b-itRating 1,317.7 $0.10 phi-4Rating 1,264.8 $0.11 gemma-3-27b-itRating 1,322.9 $0.13 deepseek-v4-flash@deepinfraRating 1,424.4 $0.14 qwen3-32bRating 1,398.1 $0.18 gpt-5-nanoRating 1,347.8 $0.20 llama-4-scout-17b-16e-instructRating 1,309.6 $0.20 llama-3.3-70b-instruct-turbo@deepinfraRating 1,295.6 $0.21 mimo-v2.6-flashRating 1,462.0 $0.22 deepseek-v4-flash@parasailRating 1,424.4 $0.22 qwen3.5-flash-2026-02-23Rating 1,401.0 $0.24 gpt-4.1-nanoRating 1,273.0 $0.24 deepseek-flash@deepinfraRating 1,495.0 $0.28 gpt-6-lunaRating 1,449.2 $0.28 claude-haiku-5-5Rating 1,450.6 $0.28 qwen3-235b-a22b-instruct-2507@deepinfraRating 1,416.4 $0.29 qwen3-30b-a3bRating 1,351.1 $0.30 glm-5.3-flashRating 1,504.8 $0.32 glm-5.3-flash@deepinfraRating 1,504.8 $0.32 gemma-4-31b-itRating 1,467.1 $0.32 deepseek-v4-flashRating 1,424.4 $0.32 deepseek-v3.2Rating 1,425.5 $0.36 gpt-oss-120bRating 1,380.8 $0.36 gpt-4o-miniRating 1,274.9 $0.36 llama-3.3-70b-instruct-turbo@parasailRating 1,295.6 $0.38 qwen3-235b-a22b-instruct-2507@parasailRating 1,416.4 $0.43 qwen3-30b-a3b-instruct-2507Rating 1,377.7 $0.48 qwen3-next-80b-a3b-instruct@deepinfraRating 1,413.5 $0.51 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,409.5 $0.51 qwen3.5-35b-a3b@deepinfraRating 1,396.5 $0.51 qwen3-next-80b-a3b-instruct@parasailRating 1,413.5 $0.52 qwen3.5-35b-a3b@parasailRating 1,396.5 $0.52 qwen3-235b-a22b-instruct-2507Rating 1,416.4 $0.55 qwen3-next-80b-a3b-instructRating 1,413.5 $0.60 qwen3-next-80b-a3b-thinkingRating 1,391.3 $0.60 mimo-v2.6-proRating 1,483.5 $0.62 gpt-5.6-lunaRating 1,476.2 $0.64 gpt-5.4-nanoRating 1,423.9 $0.66 minimax-m3@deepinfraRating 1,424.6 $0.66 deepseek-flashRating 1,495.0 $0.72 deepseek-flash@alibaba-intlRating 1,495.0 $0.72 deepseek-flash@parasailRating 1,495.0 $0.72 deepseek-flash@togetherRating 1,495.0 $0.72 minimax-m3Rating 1,424.6 $0.72 minimax-m3@parasailRating 1,424.6 $0.72 qwen3.7-plusRating 1,465.4 $0.77 qwen3.7-plus@alibaba-usRating 1,465.4 $0.77 beam-1.0-501b-a23bRating 1,471.2 $0.80 gemini-3.1-flash-liteRating 1,435.7 $0.80 qwen-plus-2025-01-25Rating 1,321.8 $0.80 inkling-smallRating 1,451.8 $0.84 qwen3.8-27b@deepinfraRating 1,451.3 $0.87 qwen3-vl-235b-a22b-instruct@parasailRating 1,409.5 $0.93 qwen3-vl-235b-a22b-instructRating 1,409.5 $0.96 gpt-4.1-miniRating 1,353.0 $0.96 gpt-5-miniRating 1,407.3 $1.00 mistral-large-2512Rating 1,401.5 $1.00 qwen3.5-35b-a3bRating 1,396.5 $1.00 gpt-3.5-turboRating 1,205.4 $1.00 glm-4.7Rating 1,426.6 $1.02 qwen3.8-27b@parasailRating 1,451.3 $1.07 deepseek-flash-fast@parasailRating 1,495.0 $1.08 minimax-m3-fast@parasailRating 1,424.6 $1.08 qwen3-235b-a22b-thinking-2507Rating 1,397.1 $1.10 glm-5.2@deepinfraRating 1,486.9 $1.17 qwen3.5-27bRating 1,422.8 $1.20 glm-4.6Rating 1,417.8 $1.20 gemini-3.5-flash-liteRating 1,442.1 $1.24 qwen3.5-27b@deepinfraRating 1,422.8 $1.25 llama-3.3-70b-instruct-turboRating 1,295.6 $1.25 deepseek-r1-0528Rating 1,412.1 $1.26 glm-5.3@deepinfraRating 1,501.4 $1.45 qwen3.5-397b-a17b@deepinfraRating 1,449.9 $1.56 gemini-3-flash-previewRating 1,474.4 $1.60 qwen3.6-plusRating 1,451.2 $1.60 qwen3.8-27bRating 1,451.3 $1.60 qwen3.5-122b-a10bRating 1,425.0 $1.60 qwen3.5-397b-a17b@parasailRating 1,449.9 $1.84 qwen3.5-397b-a17bRating 1,449.9 $1.92 qwen3-vl-235b-a22b-thinkingRating 1,398.2 $1.92 kimi-k2.6Rating 1,479.0 $2.00 deepseek-v4-pro-0813@deepinfraRating 1,449.3 $2.08 deepseek-v4-pro@deepinfraRating 1,443.1 $2.08 gemini-3.8-flashRating 1,519.2 $2.10 gemini-3.7-flashRating 1,501.6 $2.10 gemini-3.6-flashRating 1,501.1 $2.10 gemini-3.5-flashRating 1,497.4 $2.10 gpt-5.4-miniRating 1,439.1 $2.40 inklingRating 1,479.0 $2.42 deepseek-v4-pro-0813Rating 1,449.3 $2.64 deepseek-v4-proRating 1,443.1 $2.64 deepseek-v4-pro@parasailRating 1,443.1 $2.64 deepseek-v4-pro@togetherRating 1,443.1 $2.64 o4-miniRating 1,417.6 $2.64 o3-miniRating 1,381.6 $2.64 claude-haiku-4-5Rating 1,401.8 $2.80 glm-5.3Rating 1,501.4 $2.88 glm-5.3@alibaba-intlRating 1,501.4 $2.88 glm-5.3@parasailRating 1,501.4 $2.88 glm-5.2Rating 1,486.9 $2.88 glm-5.2@alibaba-intlRating 1,486.9 $2.88 glm-5.2@alibaba-usRating 1,486.9 $2.88 glm-5.1Rating 1,474.3 $2.88 deepseek-v4-pro-fast@parasailRating 1,443.1 $3.96 qwen3.8-max-0902Rating 1,505.4 $4.00 qwen3.6-max-previewRating 1,471.4 $4.16 gemini-4-hydrogenRating 1,495.3 $4.20 mistral-medium-3-5Rating 1,425.7 $4.20 glm-5.3-fast@parasailRating 1,501.4 $4.32 grok-4.7Rating 1,447.2 $4.40 grok-4.6Rating 1,444.5 $4.40 o3Rating 1,448.2 $4.80 gpt-4.1Rating 1,374.6 $4.80 qwen3.7-maxRating 1,483.4 $5.00 qwen3.7-max@alibaba-usRating 1,483.4 $5.00 gpt-5Rating 1,435.1 $5.00 gpt-5.1Rating 1,423.6 $5.00 gemini-4-argonRating 1,536.3 $5.60 gpt-6.1-solRating 1,499.3 $5.60 claude-sonnet-5-5Rating 1,486.3 $5.60 claude-sonnet-5Rating 1,472.2 $5.60 gpt-6-solRating 1,440.1 $5.60 gpt-4oRating 1,308.6 $6.00 gemini-3.1-pro-previewRating 1,487.8 $6.40 gpt-5.6-terraRating 1,478.4 $6.40 gpt-5.2Rating 1,431.8 $7.00 kimi-k3@deepinfraRating 1,501.9 $7.98 gpt-5.4Rating 1,459.0 $8.00 kimi-k3Rating 1,501.9 $8.40 kimi-k3@moonshotRating 1,501.9 $8.40 claude-sonnet-4-6Rating 1,461.3 $8.40 claude-sonnet-4-5Rating 1,426.8 $8.40 claude-opus-5-5Rating 1,514.2 $11.20 gpt-5.6-solRating 1,494.2 $11.20 claude-opus-5Rating 1,514.7 $14.00 claude-opus-4-6Rating 1,503.0 $14.00 claude-opus-4-7Rating 1,490.0 $14.00 claude-opus-4-8Rating 1,475.8 $14.00 claude-opus-4-5Rating 1,462.1 $14.00 gpt-5.5Rating 1,500.0 $16.00 gpt-4-turboRating 1,303.1 $20.00 claude-fable-5-1Rating 1,526.2 $28.00 claude-fable-5Rating 1,520.4 $28.00 gpt-6-astraRating 1,480.6 $28.00 o1Rating 1,408.9 $36.00 gpt-4Rating 1,284.8 $48.00 Price against score, Hard 0 25 50 75 100 $1 $10 $100
phi-4: 64.9 +/- 1.1, $0.27
deepseek-v4-flash-0731@deepinfra: 86.8 +/- 1.7, $0.28
gemma-3-27b-it: 74.0 +/- 1.1, $0.30
deepseek-v4-flash@deepinfra: 79.7 +/- 2.1, $0.34
gpt-5-nano: 94.9 +/- 0.5, $0.46
llama-4-scout-17b-16e-instruct: 62.3 +/- 1.2, $0.47
llama-3.3-70b-instruct-turbo@deepinfra: 41.6 +/- 1.2, $0.49
deepseek-v4-flash-0731: 86.8 +/- 1.7, $0.53
deepseek-v4-flash-0731@parasail: 86.8 +/- 1.7, $0.53
deepseek-v4-flash@parasail: 79.7 +/- 2.1, $0.53
gpt-4.1-nano: 70.0 +/- 1.0, $0.56
claude-haiku-5-5: 92.8 +/- 1.3, $0.65
gpt-6-luna: 89.1 +/- 1.6, $0.65
deepseek-flash@deepinfra: 93.3 +/- 1.3, $0.66
glm-5.3-flash: 81.2 +/- 2.0, $0.75
glm-5.3-flash@deepinfra: 81.2 +/- 2.0, $0.75
deepseek-v4-flash: 79.7 +/- 2.1, $0.76
gpt-4o-mini: 52.6 +/- 1.1, $0.84
llama-3.3-70b-instruct-turbo@parasail: 41.6 +/- 1.2, $0.89
deepseek-v4-flash-vision-exp: 87.8 +/- 1.7, $1.01
gpt-5.6-luna: 87.2 +/- 1.7, $1.48
gpt-5.4-nano: 91.0 +/- 1.5, $1.53
minimax-m3@deepinfra: 77.0 +/- 2.2, $1.55
deepseek-flash: 93.3 +/- 1.3, $1.68
deepseek-flash@alibaba-intl: 93.3 +/- 1.3, $1.68
deepseek-flash@parasail: 93.3 +/- 1.3, $1.68
deepseek-flash@together: 93.3 +/- 1.3, $1.68
minimax-m3: 77.0 +/- 2.2, $1.68
minimax-m3@parasail: 77.0 +/- 2.2, $1.68
qwen-plus-2025-01-25: 65.3 +/- 1.1, $1.88
qwen3.8-27b@deepinfra: 86.2 +/- 1.8, $1.99
deepseek-v4-flash-0731@alibaba-intl: 86.8 +/- 1.7, $2.07
gpt-4.1-mini: 87.3 +/- 0.7, $2.24
gpt-5-mini: 97.9 +/- 0.3, $2.30
gpt-3.5-turbo: 11.6 +/- 0.6, $2.35
qwen3.8-27b@parasail: 86.2 +/- 1.8, $2.46
deepseek-flash-fast@parasail: 93.3 +/- 1.3, $2.52
minimax-m3-fast@parasail: 77.0 +/- 2.2, $2.52
glm-5.2@deepinfra: 89.8 +/- 1.6, $2.75
gemini-3.5-flash-lite: 73.7 +/- 2.3, $2.85
deepseek-r1-0528: 96.6 +/- 0.4, $2.94
llama-3.3-70b-instruct-turbo: 41.6 +/- 1.2, $3.02
glm-5.3@deepinfra: 87.9 +/- 1.7, $3.38
qwen3.6-27b@deepinfra: 79.9 +/- 2.1, $3.52
qwen3.8-27b: 86.2 +/- 1.8, $3.70
qwen3.6-plus: 83.7 +/- 1.9, $3.70
qwen3.6-27b: 79.9 +/- 2.1, $4.44
kimi-k2.6: 84.3 +/- 1.9, $4.65
gemini-3.7-flash: 93.5 +/- 1.3, $4.88
gemini-3.8-flash: 91.6 +/- 1.4, $4.88
gemini-3.5-flash: 88.2 +/- 1.7, $4.88
gemini-3.6-flash: 86.4 +/- 1.8, $4.88
deepseek-v4-pro-0813@deepinfra: 95.1 +/- 1.1, $4.94
deepseek-v4-pro@deepinfra: 90.7 +/- 1.5, $4.94
kimi-k2.7-code: 79.6 +/- 2.1, $5.50
gpt-5.4-mini: 78.5 +/- 2.1, $5.55
inkling: 88.4 +/- 1.7, $5.65
o4-mini: 97.8 +/- 0.3, $6.16
o3-mini: 96.5 +/- 0.4, $6.16
deepseek-v4-pro-0813: 95.1 +/- 1.1, $6.20
deepseek-v4-pro: 90.7 +/- 1.5, $6.20
deepseek-v4-pro@parasail: 90.7 +/- 1.5, $6.20
deepseek-v4-pro@together: 90.7 +/- 1.5, $6.20
claude-haiku-4-5: 86.9 +/- 0.8, $6.50
glm-5.2: 89.8 +/- 1.6, $6.76
glm-5.2@alibaba-intl: 89.8 +/- 1.6, $6.76
glm-5.2@alibaba-us: 89.8 +/- 1.6, $6.76
glm-5.3: 87.9 +/- 1.7, $6.76
glm-5.3@alibaba-intl: 87.9 +/- 1.7, $6.76
glm-5.3@parasail: 87.9 +/- 1.7, $6.76
deepseek-v4-pro-fast@parasail: 90.7 +/- 1.5, $9.31
qwen3.8-max-0902: 91.3 +/- 1.5, $9.40
glm-5.3-fast@parasail: 87.9 +/- 1.7, $10.14
grok-4.7: 95.7 +/- 1.0, $10.34
grok-4.6: 92.6 +/- 1.4, $10.34
o3: 97.8 +/- 0.3, $11.20
gpt-4.1: 83.0 +/- 0.9, $11.20
gpt-5: 98.1 +/- 0.3, $11.50
qwen3.7-max: 85.3 +/- 1.8, $11.75
qwen3.7-max@alibaba-us: 85.3 +/- 1.8, $11.75
gpt-6.1-sol: 96.8 +/- 0.9, $13.00
gpt-6-sol: 96.4 +/- 1.0, $13.00
claude-sonnet-5-5: 96.1 +/- 1.0, $13.00
claude-sonnet-5: 92.9 +/- 1.3, $13.00
gpt-4o: 49.8 +/- 1.1, $14.00
gpt-5.6-terra: 94.9 +/- 1.1, $14.80
gemini-3.1-pro-preview: 91.0 +/- 1.5, $14.80
gpt-5.2: 93.2 +/- 1.3, $16.10
gpt-5.4: 94.2 +/- 1.2, $18.50
kimi-k3@deepinfra: 84.4 +/- 1.9, $18.53
claude-sonnet-4-5: 97.7 +/- 0.4, $19.50
claude-sonnet-4-6: 87.0 +/- 1.7, $19.50
kimi-k3: 84.4 +/- 1.9, $19.50
kimi-k3@moonshot: 84.4 +/- 1.9, $19.50
claude-opus-5-5: 97.1 +/- 0.9, $26.00
gpt-5.6-sol: 96.2 +/- 1.0, $26.00
claude-opus-5: 95.7 +/- 1.0, $32.50
claude-opus-4-8: 94.3 +/- 1.2, $32.50
claude-opus-4-7: 92.9 +/- 1.3, $32.50
claude-opus-4-5: 90.4 +/- 1.5, $32.50
claude-opus-4-6: 89.3 +/- 1.6, $32.50
gpt-5.5: 95.9 +/- 1.0, $37.00
gpt-4-turbo: 46.7 +/- 1.1, $47.00
claude-fable-5-1: 97.0 +/- 0.9, $65.00
gpt-6-astra: 96.8 +/- 0.9, $65.00
claude-fable-5: 96.0 +/- 1.0, $65.00
o1: 81.7 +/- 1.1, $84.00
gpt-4: 23.0 +/- 0.9, $114
gpt-5-mini Best, Balanced, Cheap Ranking, Hard 1. claude-opus-5-5: 97.1 +/- 0.9 1 claude-opus-5-5 97.1 2. claude-fable-5-1: 97.0 +/- 0.9 2 claude-fable-5-1 97.0 3. gpt-6.1-sol: 96.8 +/- 0.9 3 gpt-6.1-sol 96.8 4. gpt-6-astra: 96.8 +/- 0.9 4 gpt-6-astra 96.8 5. gpt-6-sol: 96.4 +/- 1.0 5 gpt-6-sol 96.4 6. gpt-5.6-sol: 96.2 +/- 1.0 6 gpt-5.6-sol 96.2 7. claude-sonnet-5-5: 96.1 +/- 1.0 7 claude-sonnet-5-5 96.1 8. claude-fable-5: 96.0 +/- 1.0 8 claude-fable-5 96.0 9. gpt-5.5: 95.9 +/- 1.0 9 gpt-5.5 95.9 10. claude-opus-5: 95.7 +/- 1.0 10 claude-opus-5 95.7 11. grok-4.7: 95.7 +/- 1.0 11 grok-4.7 95.7 12. deepseek-v4-pro-0813: 95.1 +/- 1.1 12 deepseek-v4-pro-0813 95.1 13. deepseek-v4-pro-0813@deepinfra: 95.1 +/- 1.1 13 deepseek-v4-pro-0813@deepinfra 95.1 14. gpt-5.6-terra: 94.9 +/- 1.1 14 gpt-5.6-terra 94.9 15. claude-opus-4-8: 94.3 +/- 1.2 15 claude-opus-4-8 94.3 16. gpt-5.4: 94.2 +/- 1.2 16 gpt-5.4 94.2 17. gemini-3.7-flash: 93.5 +/- 1.3 17 gemini-3.7-flash 93.5 18. gpt-5: 98.1 +/- 0.3 18 gpt-5 98.1 19. deepseek-flash: 93.3 +/- 1.3 19 deepseek-flash 93.3 20. deepseek-flash-fast@parasail: 93.3 +/- 1.3 20 deepseek-flash-fast@parasail 93.3 21. deepseek-flash@alibaba-intl: 93.3 +/- 1.3 21 deepseek-flash@alibaba-intl 93.3 22. deepseek-flash@deepinfra: 93.3 +/- 1.3 22 deepseek-flash@deepinfra 93.3 23. deepseek-flash@parasail: 93.3 +/- 1.3 23 deepseek-flash@parasail 93.3 24. deepseek-flash@together: 93.3 +/- 1.3 24 deepseek-flash@together 93.3 25. gpt-5.2: 93.2 +/- 1.3 25 gpt-5.2 93.2 26. gpt-5-mini: 97.9 +/- 0.3 26 gpt-5-mini 97.9 27. o4-mini: 97.8 +/- 0.3 27 o4-mini 97.8 28. o3: 97.8 +/- 0.3 28 o3 97.8 29. claude-sonnet-5: 92.9 +/- 1.3 29 claude-sonnet-5 92.9 30. claude-sonnet-4-5: 97.7 +/- 0.4 30 claude-sonnet-4-5 97.7 31. claude-opus-4-7: 92.9 +/- 1.3 31 claude-opus-4-7 92.9 32. claude-haiku-5-5: 92.8 +/- 1.3 32 claude-haiku-5-5 92.8 33. grok-4.6: 92.6 +/- 1.4 33 grok-4.6 92.6 34. deepseek-r1-0528: 96.6 +/- 0.4 34 deepseek-r1-0528 96.6 35. o3-mini: 96.5 +/- 0.4 35 o3-mini 96.5 36. gemini-3.8-flash: 91.6 +/- 1.4 36 gemini-3.8-flash 91.6 37. qwen3.8-max-0902: 91.3 +/- 1.5 37 qwen3.8-max-0902 91.3 38. gemini-3.1-pro-preview: 91.0 +/- 1.5 38 gemini-3.1-pro-preview 91.0 39. gpt-5.4-nano: 91.0 +/- 1.5 39 gpt-5.4-nano 91.0 40. deepseek-v4-pro: 90.7 +/- 1.5 40 deepseek-v4-pro 90.7 41. deepseek-v4-pro-fast@parasail: 90.7 +/- 1.5 41 deepseek-v4-pro-fast@parasail 90.7 42. deepseek-v4-pro@deepinfra: 90.7 +/- 1.5 42 deepseek-v4-pro@deepinfra 90.7 43. deepseek-v4-pro@parasail: 90.7 +/- 1.5 43 deepseek-v4-pro@parasail 90.7 44. deepseek-v4-pro@together: 90.7 +/- 1.5 44 deepseek-v4-pro@together 90.7 45. gpt-5-nano: 94.9 +/- 0.5 45 gpt-5-nano 94.9 46. claude-opus-4-5: 90.4 +/- 1.5 46 claude-opus-4-5 90.4 47. glm-5.2: 89.8 +/- 1.6 47 glm-5.2 89.8 48. glm-5.2@alibaba-intl: 89.8 +/- 1.6 48 glm-5.2@alibaba-intl 89.8 49. glm-5.2@alibaba-us: 89.8 +/- 1.6 49 glm-5.2@alibaba-us 89.8 50. glm-5.2@deepinfra: 89.8 +/- 1.6 50 glm-5.2@deepinfra 89.8 51. claude-opus-4-6: 89.3 +/- 1.6 51 claude-opus-4-6 89.3 52. gpt-6-luna: 89.1 +/- 1.6 52 gpt-6-luna 89.1 53. inkling: 88.4 +/- 1.7 53 inkling 88.4 54. gemini-3.5-flash: 88.2 +/- 1.7 54 gemini-3.5-flash 88.2 55. glm-5.3: 87.9 +/- 1.7 55 glm-5.3 87.9 56. glm-5.3-fast@parasail: 87.9 +/- 1.7 56 glm-5.3-fast@parasail 87.9 57. glm-5.3@alibaba-intl: 87.9 +/- 1.7 57 glm-5.3@alibaba-intl 87.9 58. glm-5.3@deepinfra: 87.9 +/- 1.7 58 glm-5.3@deepinfra 87.9 59. glm-5.3@parasail: 87.9 +/- 1.7 59 glm-5.3@parasail 87.9 60. deepseek-v4-flash-vision-exp: 87.8 +/- 1.7 60 deepseek-v4-flash-vision-exp 87.8 61. gpt-5.6-luna: 87.2 +/- 1.7 61 gpt-5.6-luna 87.2 62. claude-sonnet-4-6: 87.0 +/- 1.7 62 claude-sonnet-4-6 87.0 63. deepseek-v4-flash-0731: 86.8 +/- 1.7 63 deepseek-v4-flash-0731 86.8 64. deepseek-v4-flash-0731@alibaba-intl: 86.8 +/- 1.7 64 deepseek-v4-flash-0731@alibaba-intl 86.8 65. deepseek-v4-flash-0731@deepinfra: 86.8 +/- 1.7 65 deepseek-v4-flash-0731@deepinfra 86.8 66. deepseek-v4-flash-0731@parasail: 86.8 +/- 1.7 66 deepseek-v4-flash-0731@parasail 86.8 67. gemini-3.6-flash: 86.4 +/- 1.8 67 gemini-3.6-flash 86.4 68. qwen3.8-27b: 86.2 +/- 1.8 68 qwen3.8-27b 86.2 69. qwen3.8-27b@deepinfra: 86.2 +/- 1.8 69 qwen3.8-27b@deepinfra 86.2 70. qwen3.8-27b@parasail: 86.2 +/- 1.8 70 qwen3.8-27b@parasail 86.2 71. qwen3.7-max: 85.3 +/- 1.8 71 qwen3.7-max 85.3 72. qwen3.7-max@alibaba-us: 85.3 +/- 1.8 72 qwen3.7-max@alibaba-us 85.3 73. kimi-k3: 84.4 +/- 1.9 73 kimi-k3 84.4 74. kimi-k3@deepinfra: 84.4 +/- 1.9 74 kimi-k3@deepinfra 84.4 75. kimi-k3@moonshot: 84.4 +/- 1.9 75 kimi-k3@moonshot 84.4 76. gpt-4.1-mini: 87.3 +/- 0.7 76 gpt-4.1-mini 87.3 77. kimi-k2.6: 84.3 +/- 1.9 77 kimi-k2.6 84.3 78. claude-haiku-4-5: 86.9 +/- 0.8 78 claude-haiku-4-5 86.9 79. qwen3.6-plus: 83.7 +/- 1.9 79 qwen3.6-plus 83.7 80. glm-5.3-flash: 81.2 +/- 2.0 80 glm-5.3-flash 81.2 81. glm-5.3-flash@deepinfra: 81.2 +/- 2.0 81 glm-5.3-flash@deepinfra 81.2 82. qwen3.6-27b: 79.9 +/- 2.1 82 qwen3.6-27b 79.9 83. qwen3.6-27b@deepinfra: 79.9 +/- 2.1 83 qwen3.6-27b@deepinfra 79.9 84. deepseek-v4-flash: 79.7 +/- 2.1 84 deepseek-v4-flash 79.7 85. deepseek-v4-flash@deepinfra: 79.7 +/- 2.1 85 deepseek-v4-flash@deepinfra 79.7 86. deepseek-v4-flash@parasail: 79.7 +/- 2.1 86 deepseek-v4-flash@parasail 79.7 87. kimi-k2.7-code: 79.6 +/- 2.1 87 kimi-k2.7-code 79.6 88. gpt-4.1: 83.0 +/- 0.9 88 gpt-4.1 83.0 89. gpt-5.4-mini: 78.5 +/- 2.1 89 gpt-5.4-mini 78.5 90. o1: 81.7 +/- 1.1 90 o1 81.7 91. minimax-m3: 77.0 +/- 2.2 91 minimax-m3 77.0 92. minimax-m3-fast@parasail: 77.0 +/- 2.2 92 minimax-m3-fast@parasail 77.0 93. minimax-m3@deepinfra: 77.0 +/- 2.2 93 minimax-m3@deepinfra 77.0 94. minimax-m3@parasail: 77.0 +/- 2.2 94 minimax-m3@parasail 77.0 95. gemini-3.5-flash-lite: 73.7 +/- 2.3 95 gemini-3.5-flash-lite 73.7 96. gemma-3-27b-it: 74.0 +/- 1.1 96 gemma-3-27b-it 74.0 97. gpt-4.1-nano: 70.0 +/- 1.0 97 gpt-4.1-nano 70.0 98. qwen-plus-2025-01-25: 65.3 +/- 1.1 98 qwen-plus-2025-01-25 65.3 99. phi-4: 64.9 +/- 1.1 99 phi-4 64.9 100. llama-4-scout-17b-16e-instruct: 62.3 +/- 1.2 100 llama-4-scout-17b-16e-instruct 62.3 101. gpt-4o-mini: 52.6 +/- 1.1 101 gpt-4o-mini 52.6 102. gpt-4o: 49.8 +/- 1.1 102 gpt-4o 49.8 103. gpt-4-turbo: 46.7 +/- 1.1 103 gpt-4-turbo 46.7 104. llama-3.3-70b-instruct-turbo: 41.6 +/- 1.2 104 llama-3.3-70b-instruct-turbo 41.6 105. llama-3.3-70b-instruct-turbo@deepinfra: 41.6 +/- 1.2 105 llama-3.3-70b-instruct-turbo@deepinfra 41.6 106. llama-3.3-70b-instruct-turbo@parasail: 41.6 +/- 1.2 106 llama-3.3-70b-instruct-turbo@parasail 41.6 107. gpt-4: 23.0 +/- 0.9 107 gpt-4 23.0 108. gpt-3.5-turbo: 11.6 +/- 0.6 108 gpt-3.5-turbo 11.6 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Price against score, Hardest 0 25 50 75 100 $1 $10 $100
gpt-oss-20b: 40.3 +/- 6.3, $0.30
gemma-3-4b-it: 7.5 +/- 2.8, $0.30
gpt-oss-120b@deepinfra: 88.9 +/- 4.5, $0.37
gemma-3-12b-it: 16.7 +/- 4.2, $0.38
phi-4: 13.8 +/- 3.7, $0.42
deepseek-v4-flash-0731@deepinfra: 76.0 +/- 1.8, $0.45
gemma-3-27b-it: 22.5 +/- 5.3, $0.48
qwen3.5-9b@deepinfra: 42.8 +/- 6.5, $0.53
qwen3-32b: 66.9 +/- 6.0, $0.66
qwen3.5-9b@parasail: 42.8 +/- 6.5, $0.68
gemma-4-26b-a4b-it: 82.2 +/- 5.8, $0.72
qwen3-14b: 66.4 +/- 6.2, $0.72
deepseek-v4.1-flash-1017@deepinfra: 85.3 +/- 1.4, $0.74
gpt-5-nano: 21.1 +/- 2.8, $0.75
llama-4-scout-17b-16e-instruct: 3.9 +/- 2.9, $0.75
llama-3.3-70b-instruct-turbo@deepinfra: 5.1 +/- 2.4, $0.78
deepseek-v4-flash-0731: 76.0 +/- 1.8, $0.84
deepseek-v4-flash-0731@parasail: 76.0 +/- 1.8, $0.84
qwen3.5-9b: 42.8 +/- 6.5, $0.89
qwen3.5-flash-2026-02-23: 33.4 +/- 2.0, $0.90
gpt-4.1-nano: 15.0 +/- 3.0, $0.90
gpt-6-luna: 88.9 +/- 1.3, $1.05
claude-haiku-5-5: 86.2 +/- 1.4, $1.05
deepseek-flash@deepinfra: 82.9 +/- 1.4, $1.05
qwen3-30b-a3b: 62.8 +/- 6.2, $1.11
glm-5.3-flash: 74.8 +/- 1.8, $1.20
glm-5.3-flash@deepinfra: 74.8 +/- 1.8, $1.20
gemma-4-31b-it: 73.3 +/- 6.7, $1.20
gpt-oss-120b: 88.9 +/- 4.5, $1.35
gpt-4o-mini: 3.8 +/- 1.4, $1.35
llama-3.3-70b-instruct-turbo@parasail: 5.1 +/- 2.4, $1.41
qwen3.6-35b-a3b@parasail: 53.5 +/- 2.8, $1.65
qwen3.6-35b-a3b@deepinfra: 53.5 +/- 2.8, $1.73
qwen3-30b-a3b-instruct-2507: 62.2 +/- 6.5, $1.80
qwen3.5-35b-a3b@deepinfra: 54.4 +/- 6.2, $1.92
qwen3.5-35b-a3b@parasail: 54.4 +/- 6.2, $1.95
gpt-5.6-luna: 57.3 +/- 4.4, $2.40
gpt-5.4-nano: 25.7 +/- 2.9, $2.48
minimax-m3@deepinfra: 71.1 +/- 6.8, $2.49
deepseek-v4.1-flash-1017: 85.3 +/- 1.4, $2.70
deepseek-v4.1-flash-1017@parasail: 85.3 +/- 1.4, $2.70
deepseek-flash: 82.9 +/- 1.4, $2.70
deepseek-flash@alibaba-intl: 82.9 +/- 1.4, $2.70
deepseek-flash@parasail: 82.9 +/- 1.4, $2.70
deepseek-flash@together: 82.9 +/- 1.4, $2.70
minimax-m3: 71.1 +/- 6.8, $2.70
minimax-m3@parasail: 71.1 +/- 6.8, $2.70
qwen3.7-plus: 63.9 +/- 2.4, $2.88
qwen3.7-plus@alibaba-us: 63.9 +/- 2.4, $2.88
qwen3.6-flash: 37.3 +/- 2.1, $3.00
qwen3.6-flash@alibaba-us: 37.3 +/- 2.1, $3.00
gemini-3.1-flash-lite: 33.5 +/- 3.9, $3.00
qwen-plus-2025-01-25: 17.8 +/- 4.4, $3.00
qwen-plus: 1.7 +/- 0.8, $3.00
inkling-small: 68.2 +/- 2.5, $3.15
deepseek-v4-flash-0731@alibaba-intl: 76.0 +/- 1.8, $3.30
gpt-4.1-mini: 18.6 +/- 2.2, $3.60
qwen3.5-35b-a3b: 54.4 +/- 6.2, $3.75
gpt-5-mini: 34.1 +/- 2.9, $3.75
gpt-3.5-turbo: 1.1 +/- 2.7, $3.75
glm-4.7: 42.9 +/- 2.4, $3.83
deepseek-flash-fast@parasail: 82.9 +/- 1.4, $4.05
minimax-m3-fast@parasail: 71.1 +/- 6.8, $4.05
qwen3-235b-a22b-thinking-2507: 47.6 +/- 2.7, $4.14
qwen3-30b-a3b-thinking-2507: 70.3 +/- 6.0, $4.20
glm-5.2@deepinfra: 41.4 +/- 4.2, $4.39
qwen3.6-35b-a3b: 53.5 +/- 2.8, $4.50
glm-4.6: 3.8 +/- 1.1, $4.50
gemini-3.5-flash-lite: 44.6 +/- 4.2, $4.65
llama-3.3-70b-instruct-turbo: 5.1 +/- 2.4, $4.68
deepseek-r1-0528: 66.4 +/- 4.9, $4.73
mistral-large-4-0: 78.5 +/- 1.5, $5.25
glm-5.3@deepinfra: 79.9 +/- 2.2, $5.44
qwen3.6-27b@deepinfra: 62.6 +/- 2.6, $5.76
qwen3.5-397b-a17b@deepinfra: 60.1 +/- 2.7, $5.85
gemini-3-flash-preview: 59.9 +/- 1.8, $6.00
qwen3.6-plus: 52.6 +/- 1.8, $6.00
qwen3.5-397b-a17b@parasail: 60.1 +/- 2.7, $6.90
qwen3.6-27b: 62.6 +/- 2.6, $7.20
qwen3.5-397b-a17b: 60.1 +/- 2.7, $7.20
kimi-k2.6: 64.1 +/- 1.6, $7.50
deepseek-v4-pro-0813@deepinfra: 81.6 +/- 1.5, $7.80
deepseek-v4-pro@deepinfra: 46.1 +/- 4.6, $7.80
gemini-3.8-flash: 83.7 +/- 1.5, $7.88
gemini-3.7-flash: 84.4 +/- 1.6, $7.88
gemini-3.5-flash: 65.3 +/- 2.2, $7.88
gemini-3.6-flash: 71.7 +/- 3.4, $7.88
kimi-k2.7-code: 74.8 +/- 2.1, $8.85
gpt-5.4-mini: 23.6 +/- 2.7, $9.00
inkling: 61.1 +/- 2.4, $9.08
deepseek-v4-pro-0813: 81.6 +/- 1.5, $9.90
deepseek-v4-pro: 46.1 +/- 4.6, $9.90
deepseek-v4-pro@parasail: 46.1 +/- 4.6, $9.90
deepseek-v4-pro@together: 46.1 +/- 4.6, $9.90
o4-mini: 28.2 +/- 2.7, $9.90
o3-mini: 20.7 +/- 2.7, $9.90
gpt-6.1-terra: 95.4 +/- 2.6, $10.50
claude-haiku-4-5: 20.0 +/- 3.2, $10.50
glm-5.3: 79.9 +/- 2.2, $10.80
glm-5.3@alibaba-intl: 79.9 +/- 2.2, $10.80
glm-5.3@parasail: 79.9 +/- 2.2, $10.80
glm-5.1: 54.5 +/- 1.8, $10.80
glm-5.2: 41.4 +/- 4.2, $10.80
glm-5.2@alibaba-intl: 41.4 +/- 4.2, $10.80
glm-5.2@alibaba-us: 41.4 +/- 4.2, $10.80
deepseek-v4-pro-fast@parasail: 46.1 +/- 4.6, $14.85
qwen3.8-max-0902: 82.8 +/- 2.9, $15.00
qwen3.6-max-preview: 57.1 +/- 2.5, $15.60
glm-5.3-fast@parasail: 79.9 +/- 2.2, $16.20
grok-4.6: 82.6 +/- 1.4, $16.50
grok-4.7: 75.5 +/- 1.6, $16.50
o3: 29.7 +/- 2.7, $18.00
gpt-4.1: 16.6 +/- 2.2, $18.00
qwen3.7-max: 80.1 +/- 2.1, $18.75
qwen3.7-max@alibaba-us: 80.1 +/- 2.1, $18.75
gpt-5: 39.2 +/- 3.0, $18.75
gpt-5.1: 19.9 +/- 3.7, $18.75
gpt-6.1-sol: 96.8 +/- 2.6, $21.00
gemini-4-carbon: 95.3 +/- 2.7, $21.00
gpt-6-sol: 94.9 +/- 2.7, $21.00
claude-sonnet-5-5: 94.4 +/- 2.7, $21.00
gemini-4-argon: 93.0 +/- 2.7, $21.00
claude-sonnet-5: 72.8 +/- 3.3, $21.00
gpt-4o: 2.3 +/- 0.8, $22.50
gemini-3.1-pro-preview: 64.1 +/- 1.7, $24.00
gpt-5.6-terra: 66.8 +/- 4.5, $24.00
gpt-5.2: 51.8 +/- 3.1, $26.25
kimi-k3@deepinfra: 70.4 +/- 4.0, $29.93
gpt-5.4: 60.9 +/- 3.3, $30.00
kimi-k3: 70.4 +/- 4.0, $31.50
kimi-k3@moonshot: 70.4 +/- 4.0, $31.50
claude-sonnet-4-6: 51.8 +/- 3.7, $31.50
claude-sonnet-4-5: 22.8 +/- 2.8, $31.50
claude-opus-5-5: 95.6 +/- 2.6, $42.00
gpt-5.6-sol: 77.2 +/- 4.2, $42.00
claude-opus-5: 92.8 +/- 1.6, $52.50
claude-opus-4-8: 72.3 +/- 2.5, $52.50
claude-opus-4-7: 66.9 +/- 2.2, $52.50
claude-opus-4-6: 61.8 +/- 1.9, $52.50
claude-opus-4-5: 34.4 +/- 2.6, $52.50
gpt-5.5: 69.1 +/- 4.5, $60.00
gpt-4-turbo: 3.7 +/- 1.2, $75.00
claude-fable-5-2: 97.4 +/- 0.7, $84.00
gpt-6-astra: 96.8 +/- 2.6, $105
claude-fable-5-1: 95.1 +/- 2.7, $105
claude-fable-5: 92.9 +/- 1.5, $105
o1: 17.6 +/- 2.8, $135
gpt-4: 1.1 +/- 0.5, $180
gpt-oss-120b@deepinfra Cheap deepseek-v4.1-flash-1017@deepinfra Balanced gpt-6.1-sol Best Ranking, Hardest 1. claude-fable-5-2: 97.4 +/- 0.7 1 claude-fable-5-2 97.4 2. gpt-6-astra: 96.8 +/- 2.6 2 gpt-6-astra 96.8 3. gpt-6.1-sol: 96.8 +/- 2.6 3 gpt-6.1-sol 96.8 4. claude-opus-5-5: 95.6 +/- 2.6 4 claude-opus-5-5 95.6 5. gpt-6.1-terra: 95.4 +/- 2.6 5 gpt-6.1-terra 95.4 6. gemini-4-carbon: 95.3 +/- 2.7 6 gemini-4-carbon 95.3 7. claude-fable-5-1: 95.1 +/- 2.7 7 claude-fable-5-1 95.1 8. gpt-6-sol: 94.9 +/- 2.7 8 gpt-6-sol 94.9 9. claude-sonnet-5-5: 94.4 +/- 2.7 9 claude-sonnet-5-5 94.4 10. gpt-6-luna: 88.9 +/- 1.3 10 gpt-6-luna 88.9 11. claude-fable-5: 92.9 +/- 1.5 11 claude-fable-5 92.9 12. claude-opus-5: 92.8 +/- 1.6 12 claude-opus-5 92.8 13. gemini-4-argon: 93.0 +/- 2.7 13 gemini-4-argon 93.0 14. deepseek-v4.1-flash-1017: 85.3 +/- 1.4 14 deepseek-v4.1-flash-1017 85.3 15. deepseek-v4.1-flash-1017@deepinfra: 85.3 +/- 1.4 15 deepseek-v4.1-flash-1017@deepinfra 85.3 16. deepseek-v4.1-flash-1017@parasail: 85.3 +/- 1.4 16 deepseek-v4.1-flash-1017@parasail 85.3 17. claude-haiku-5-5: 86.2 +/- 1.4 17 claude-haiku-5-5 86.2 18. gemini-3.8-flash: 83.7 +/- 1.5 18 gemini-3.8-flash 83.7 19. grok-4.6: 82.6 +/- 1.4 19 grok-4.6 82.6 20. gemini-3.7-flash: 84.4 +/- 1.6 20 gemini-3.7-flash 84.4 21. deepseek-flash: 82.9 +/- 1.4 21 deepseek-flash 82.9 22. deepseek-flash-fast@parasail: 82.9 +/- 1.4 22 deepseek-flash-fast@parasail 82.9 23. deepseek-flash@alibaba-intl: 82.9 +/- 1.4 23 deepseek-flash@alibaba-intl 82.9 24. deepseek-flash@deepinfra: 82.9 +/- 1.4 24 deepseek-flash@deepinfra 82.9 25. deepseek-flash@parasail: 82.9 +/- 1.4 25 deepseek-flash@parasail 82.9 26. deepseek-flash@together: 82.9 +/- 1.4 26 deepseek-flash@together 82.9 27. deepseek-v4-pro-0813: 81.6 +/- 1.5 27 deepseek-v4-pro-0813 81.6 28. deepseek-v4-pro-0813@deepinfra: 81.6 +/- 1.5 28 deepseek-v4-pro-0813@deepinfra 81.6 29. qwen3.8-max-0902: 82.8 +/- 2.9 29 qwen3.8-max-0902 82.8 30. qwen3.7-max: 80.1 +/- 2.1 30 qwen3.7-max 80.1 31. qwen3.7-max@alibaba-us: 80.1 +/- 2.1 31 qwen3.7-max@alibaba-us 80.1 32. mistral-large-4-0: 78.5 +/- 1.5 32 mistral-large-4-0 78.5 33. claude-opus-4-8: 72.3 +/- 2.5 33 claude-opus-4-8 72.3 34. glm-5.3: 79.9 +/- 2.2 34 glm-5.3 79.9 35. glm-5.3-fast@parasail: 79.9 +/- 2.2 35 glm-5.3-fast@parasail 79.9 36. glm-5.3@alibaba-intl: 79.9 +/- 2.2 36 glm-5.3@alibaba-intl 79.9 37. glm-5.3@deepinfra: 79.9 +/- 2.2 37 glm-5.3@deepinfra 79.9 38. glm-5.3@parasail: 79.9 +/- 2.2 38 glm-5.3@parasail 79.9 39. grok-4.7: 75.5 +/- 1.6 39 grok-4.7 75.5 40. gemini-3.1-pro-preview: 64.1 +/- 1.7 40 gemini-3.1-pro-preview 64.1 41. claude-opus-4-7: 66.9 +/- 2.2 41 claude-opus-4-7 66.9 42. claude-opus-4-6: 61.8 +/- 1.9 42 claude-opus-4-6 61.8 43. kimi-k2.6: 64.1 +/- 1.6 43 kimi-k2.6 64.1 44. kimi-k2.7-code: 74.8 +/- 2.1 44 kimi-k2.7-code 74.8 45. deepseek-v4-flash-0731: 76.0 +/- 1.8 45 deepseek-v4-flash-0731 76.0 46. deepseek-v4-flash-0731@alibaba-intl: 76.0 +/- 1.8 46 deepseek-v4-flash-0731@alibaba-intl 76.0 47. deepseek-v4-flash-0731@deepinfra: 76.0 +/- 1.8 47 deepseek-v4-flash-0731@deepinfra 76.0 48. deepseek-v4-flash-0731@parasail: 76.0 +/- 1.8 48 deepseek-v4-flash-0731@parasail 76.0 49. gemini-3.5-flash: 65.3 +/- 2.2 49 gemini-3.5-flash 65.3 50. glm-5.3-flash: 74.8 +/- 1.8 50 glm-5.3-flash 74.8 51. glm-5.3-flash@deepinfra: 74.8 +/- 1.8 51 glm-5.3-flash@deepinfra 74.8 52. qwen3.6-max-preview: 57.1 +/- 2.5 52 qwen3.6-max-preview 57.1 53. gemini-3-flash-preview: 59.9 +/- 1.8 53 gemini-3-flash-preview 59.9 54. gpt-5.6-sol: 77.2 +/- 4.2 54 gpt-5.6-sol 77.2 55. gpt-oss-120b: 88.9 +/- 4.5 55 gpt-oss-120b 88.9 56. gpt-oss-120b@deepinfra: 88.9 +/- 4.5 56 gpt-oss-120b@deepinfra 88.9 57. claude-sonnet-5: 72.8 +/- 3.3 57 claude-sonnet-5 72.8 58. glm-5.1: 54.5 +/- 1.8 58 glm-5.1 54.5 59. qwen3.7-plus: 63.9 +/- 2.4 59 qwen3.7-plus 63.9 60. qwen3.7-plus@alibaba-us: 63.9 +/- 2.4 60 qwen3.7-plus@alibaba-us 63.9 61. qwen3.6-plus: 52.6 +/- 1.8 61 qwen3.6-plus 52.6 62. inkling-small: 68.2 +/- 2.5 62 inkling-small 68.2 63. gpt-5.4: 60.9 +/- 3.3 63 gpt-5.4 60.9 64. gemini-3.6-flash: 71.7 +/- 3.4 64 gemini-3.6-flash 71.7 65. kimi-k3: 70.4 +/- 4.0 65 kimi-k3 70.4 66. kimi-k3@deepinfra: 70.4 +/- 4.0 66 kimi-k3@deepinfra 70.4 67. kimi-k3@moonshot: 70.4 +/- 4.0 67 kimi-k3@moonshot 70.4 68. qwen3.6-27b: 62.6 +/- 2.6 68 qwen3.6-27b 62.6 69. qwen3.6-27b@deepinfra: 62.6 +/- 2.6 69 qwen3.6-27b@deepinfra 62.6 70. qwen3-235b-a22b-thinking-2507: 47.6 +/- 2.7 70 qwen3-235b-a22b-thinking-2507 47.6 71. gpt-5.5: 69.1 +/- 4.5 71 gpt-5.5 69.1 72. gemma-4-26b-a4b-it: 82.2 +/- 5.8 72 gemma-4-26b-a4b-it 82.2 73. gpt-5.6-terra: 66.8 +/- 4.5 73 gpt-5.6-terra 66.8 74. qwen3.5-397b-a17b: 60.1 +/- 2.7 74 qwen3.5-397b-a17b 60.1 75. qwen3.5-397b-a17b@deepinfra: 60.1 +/- 2.7 75 qwen3.5-397b-a17b@deepinfra 60.1 76. qwen3.5-397b-a17b@parasail: 60.1 +/- 2.7 76 qwen3.5-397b-a17b@parasail 60.1 77. inkling: 61.1 +/- 2.4 77 inkling 61.1 78. gpt-5.2: 51.8 +/- 3.1 78 gpt-5.2 51.8 79. glm-4.7: 42.9 +/- 2.4 79 glm-4.7 42.9 80. claude-sonnet-4-6: 51.8 +/- 3.7 80 claude-sonnet-4-6 51.8 81. qwen3.6-35b-a3b: 53.5 +/- 2.8 81 qwen3.6-35b-a3b 53.5 82. qwen3.6-35b-a3b@deepinfra: 53.5 +/- 2.8 82 qwen3.6-35b-a3b@deepinfra 53.5 83. qwen3.6-35b-a3b@parasail: 53.5 +/- 2.8 83 qwen3.6-35b-a3b@parasail 53.5 84. gemma-4-31b-it: 73.3 +/- 6.7 84 gemma-4-31b-it 73.3 85. gpt-5.6-luna: 57.3 +/- 4.4 85 gpt-5.6-luna 57.3 86. minimax-m3: 71.1 +/- 6.8 86 minimax-m3 71.1 87. minimax-m3-fast@parasail: 71.1 +/- 6.8 87 minimax-m3-fast@parasail 71.1 88. minimax-m3@deepinfra: 71.1 +/- 6.8 88 minimax-m3@deepinfra 71.1 89. minimax-m3@parasail: 71.1 +/- 6.8 89 minimax-m3@parasail 71.1 90. qwen3-30b-a3b-thinking-2507: 70.3 +/- 6.0 90 qwen3-30b-a3b-thinking-2507 70.3 91. qwen3-32b: 66.9 +/- 6.0 91 qwen3-32b 66.9 92. deepseek-r1-0528: 66.4 +/- 4.9 92 deepseek-r1-0528 66.4 93. qwen3.6-flash: 37.3 +/- 2.1 93 qwen3.6-flash 37.3 94. qwen3.6-flash@alibaba-us: 37.3 +/- 2.1 94 qwen3.6-flash@alibaba-us 37.3 95. qwen3-14b: 66.4 +/- 6.2 95 qwen3-14b 66.4 96. qwen3-30b-a3b: 62.8 +/- 6.2 96 qwen3-30b-a3b 62.8 97. qwen3-30b-a3b-instruct-2507: 62.2 +/- 6.5 97 qwen3-30b-a3b-instruct-2507 62.2 98. qwen3.5-flash-2026-02-23: 33.4 +/- 2.0 98 qwen3.5-flash-2026-02-23 33.4 99. qwen3.5-35b-a3b: 54.4 +/- 6.2 99 qwen3.5-35b-a3b 54.4 100. qwen3.5-35b-a3b@deepinfra: 54.4 +/- 6.2 100 qwen3.5-35b-a3b@deepinfra 54.4 101. qwen3.5-35b-a3b@parasail: 54.4 +/- 6.2 101 qwen3.5-35b-a3b@parasail 54.4 102. gpt-5: 39.2 +/- 3.0 102 gpt-5 39.2 103. gemini-3.5-flash-lite: 44.6 +/- 4.2 103 gemini-3.5-flash-lite 44.6 104. deepseek-v4-pro: 46.1 +/- 4.6 104 deepseek-v4-pro 46.1 105. deepseek-v4-pro-fast@parasail: 46.1 +/- 4.6 105 deepseek-v4-pro-fast@parasail 46.1 106. deepseek-v4-pro@deepinfra: 46.1 +/- 4.6 106 deepseek-v4-pro@deepinfra 46.1 107. deepseek-v4-pro@parasail: 46.1 +/- 4.6 107 deepseek-v4-pro@parasail 46.1 108. deepseek-v4-pro@together: 46.1 +/- 4.6 108 deepseek-v4-pro@together 46.1 109. qwen3.5-9b: 42.8 +/- 6.5 109 qwen3.5-9b 42.8 110. qwen3.5-9b@deepinfra: 42.8 +/- 6.5 110 qwen3.5-9b@deepinfra 42.8 111. qwen3.5-9b@parasail: 42.8 +/- 6.5 111 qwen3.5-9b@parasail 42.8 112. glm-5.2: 41.4 +/- 4.2 112 glm-5.2 41.4 113. glm-5.2@alibaba-intl: 41.4 +/- 4.2 113 glm-5.2@alibaba-intl 41.4 114. glm-5.2@alibaba-us: 41.4 +/- 4.2 114 glm-5.2@alibaba-us 41.4 115. glm-5.2@deepinfra: 41.4 +/- 4.2 115 glm-5.2@deepinfra 41.4 116. claude-opus-4-5: 34.4 +/- 2.6 116 claude-opus-4-5 34.4 117. gpt-5-mini: 34.1 +/- 2.9 117 gpt-5-mini 34.1 118. gpt-oss-20b: 40.3 +/- 6.3 118 gpt-oss-20b 40.3 119. o3: 29.7 +/- 2.7 119 o3 29.7 120. o4-mini: 28.2 +/- 2.7 120 o4-mini 28.2 121. claude-haiku-4-5: 20.0 +/- 3.2 121 claude-haiku-4-5 20.0 122. gemini-3.1-flash-lite: 33.5 +/- 3.9 122 gemini-3.1-flash-lite 33.5 123. gpt-5.1: 19.9 +/- 3.7 123 gpt-5.1 19.9 124. gpt-5.4-nano: 25.7 +/- 2.9 124 gpt-5.4-nano 25.7 125. gemma-3-27b-it: 22.5 +/- 5.3 125 gemma-3-27b-it 22.5 126. claude-sonnet-4-5: 22.8 +/- 2.8 126 claude-sonnet-4-5 22.8 127. qwen-plus-2025-01-25: 17.8 +/- 4.4 127 qwen-plus-2025-01-25 17.8 128. gpt-4.1-nano: 15.0 +/- 3.0 128 gpt-4.1-nano 15.0 129. gpt-5-nano: 21.1 +/- 2.8 129 gpt-5-nano 21.1 130. gemma-3-12b-it: 16.7 +/- 4.2 130 gemma-3-12b-it 16.7 131. o3-mini: 20.7 +/- 2.7 131 o3-mini 20.7 132. gpt-5.4-mini: 23.6 +/- 2.7 132 gpt-5.4-mini 23.6 133. phi-4: 13.8 +/- 3.7 133 phi-4 13.8 134. gpt-4.1-mini: 18.6 +/- 2.2 134 gpt-4.1-mini 18.6 135. gemma-3-4b-it: 7.5 +/- 2.8 135 gemma-3-4b-it 7.5 136. gpt-4.1: 16.6 +/- 2.2 136 gpt-4.1 16.6 137. llama-3.3-70b-instruct-turbo: 5.1 +/- 2.4 137 llama-3.3-70b-instruct-turbo 5.1 138. llama-3.3-70b-instruct-turbo@deepinfra: 5.1 +/- 2.4 138 llama-3.3-70b-instruct-turbo@deepinfra 5.1 139. llama-3.3-70b-instruct-turbo@parasail: 5.1 +/- 2.4 139 llama-3.3-70b-instruct-turbo@parasail 5.1 140. o1: 17.6 +/- 2.8 140 o1 17.6 141. llama-4-scout-17b-16e-instruct: 3.9 +/- 2.9 141 llama-4-scout-17b-16e-instruct 3.9 142. gpt-4: 1.1 +/- 0.5 142 gpt-4 1.1 143. gpt-4o-mini: 3.8 +/- 1.4 143 gpt-4o-mini 3.8 144. gpt-4-turbo: 3.7 +/- 1.2 144 gpt-4-turbo 3.7 145. gpt-3.5-turbo: 1.1 +/- 2.7 145 gpt-3.5-turbo 1.1 146. gpt-4o: 2.3 +/- 0.8 146 gpt-4o 2.3 147. glm-4.6: 3.8 +/- 1.1 147 glm-4.6 3.8 148. qwen-plus: 1.7 +/- 0.8 148 qwen-plus 1.7 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Writing code Price against score, Hard 0 25 50 75 100 $1 $10 $100
deepseek-v4.1-flash-1017@deepinfra: 38.8 +/- 11.5, $0.59
deepseek-flash@deepinfra: 64.4 +/- 11.4, $0.85
claude-haiku-5-5: 63.4 +/- 7.1, $0.88
gpt-6-luna: 25.7 +/- 6.8, $0.88
gemma-4-31b-it: 8.5 +/- 6.6, $0.94
glm-5.3-flash: 20.7 +/- 9.6, $0.98
glm-5.3-flash@deepinfra: 20.7 +/- 9.6, $0.98
minimax-m3.1-flash: 25.7 +/- 10.3, $1.11
minimax-m3.1-flash@parasail: 25.7 +/- 10.3, $1.11
mimo-v2.6-pro: 36.4 +/- 11.3, $1.83
gpt-5.6-luna: 21.1 +/- 9.8, $2.02
deepseek-flash: 64.4 +/- 11.4, $2.22
deepseek-flash@alibaba-intl: 64.4 +/- 11.4, $2.22
deepseek-flash@parasail: 64.4 +/- 11.4, $2.22
deepseek-flash@together: 64.4 +/- 11.4, $2.22
deepseek-v4.1-flash-1017: 38.8 +/- 11.5, $2.22
deepseek-v4.1-flash-1017@parasail: 38.8 +/- 11.5, $2.22
qwen3.8-27b@deepinfra: 15.9 +/- 8.6, $2.83
deepseek-flash-fast@parasail: 64.4 +/- 11.4, $3.33
qwen3.8-27b@parasail: 15.9 +/- 8.6, $3.45
glm-5.2@deepinfra: 36.7 +/- 11.5, $3.56
mistral-large-4-0: 32.4 +/- 7.4, $4.24
qwen3.8-27b: 15.9 +/- 8.6, $5.05
deepseek-v4-pro-0813@deepinfra: 33.3 +/- 10.0, $6.11
gemini-3.8-flash: 27.7 +/- 7.4, $6.56
gemini-3.7-flash: 41.1 +/- 12.2, $6.56
gemini-3.6-flash: 23.3 +/- 9.3, $6.56
inkling: 9.8 +/- 7.0, $7.47
deepseek-v4-pro-0813: 33.3 +/- 10.0, $7.99
glm-5.2: 36.7 +/- 11.5, $8.74
glm-5.2@alibaba-intl: 36.7 +/- 11.5, $8.74
glm-5.2@alibaba-us: 36.7 +/- 11.5, $8.74
gpt-6.1-terra: 66.4 +/- 7.5, $8.75
qwen3.8-max-0902: 37.2 +/- 7.3, $12.10
gemini-4-hydrogen: 61.8 +/- 11.5, $13.13
grok-4.7: 49.9 +/- 8.2, $13.31
grok-4.6: 29.9 +/- 7.2, $13.31
gpt-5: 4.4 +/- 3.4, $16.00
kimi-k3.1-pro@deepinfra: 38.7 +/- 11.5, $16.63
gemini-4-carbon: 72.7 +/- 10.5, $17.50
gpt-6.1-sol: 63.4 +/- 8.0, $17.50
gemini-4-argon: 58.7 +/- 8.3, $17.50
claude-sonnet-5-5: 80.6 +/- 5.7, $17.50
kimi-k3.1-pro: 38.7 +/- 11.5, $17.50
kimi-k3.1-pro@together: 38.7 +/- 11.5, $17.50
gpt-6-sol: 48.9 +/- 11.2, $17.50
claude-sonnet-5: 63.3 +/- 10.7, $17.50
gpt-5.6-terra: 33.5 +/- 8.1, $20.20
gemini-3.1-pro-preview: 8.9 +/- 4.6, $20.20
gpt-5.2: 32.2 +/- 10.0, $22.40
kimi-k3@deepinfra: 45.2 +/- 8.0, $24.94
gpt-5.4: 23.3 +/- 8.1, $25.25
kimi-k3: 45.2 +/- 8.0, $26.25
kimi-k3@moonshot: 45.2 +/- 8.0, $26.25
claude-opus-5-5: 84.2 +/- 5.4, $35.00
gpt-5.6-sol: 20.0 +/- 9.2, $35.00
claude-opus-5: 87.8 +/- 7.2, $43.75
claude-opus-4-7: 43.3 +/- 9.6, $43.75
claude-opus-4-8: 65.6 +/- 10.9, $43.75
claude-opus-4-6: 33.3 +/- 10.0, $43.75
gpt-5.5: 23.3 +/- 10.1, $50.50
claude-fable-5-2: 82.9 +/- 8.9, $70.00
claude-fable-5-1: 83.8 +/- 5.5, $87.50
gpt-6-astra: 67.8 +/- 7.8, $87.50
claude-fable-5: 70.0 +/- 10.4, $87.50
deepseek-flash@deepinfra Cheap gemini-4-carbon Best, Balanced Ranking, Hard 1. claude-opus-5-5: 84.2 +/- 5.4 1 claude-opus-5-5 84.2 2. gemini-4-carbon: 72.7 +/- 10.5 2 gemini-4-carbon 72.7 3. claude-fable-5-1: 83.8 +/- 5.5 3 claude-fable-5-1 83.8 4. gpt-6-astra: 67.8 +/- 7.8 4 gpt-6-astra 67.8 5. gpt-6.1-sol: 63.4 +/- 8.0 5 gpt-6.1-sol 63.4 6. gemini-4-argon: 58.7 +/- 8.3 6 gemini-4-argon 58.7 7. claude-opus-5: 87.8 +/- 7.2 7 claude-opus-5 87.8 8. claude-sonnet-5-5: 80.6 +/- 5.7 8 claude-sonnet-5-5 80.6 9. claude-fable-5-2: 82.9 +/- 8.9 9 claude-fable-5-2 82.9 10. claude-fable-5: 70.0 +/- 10.4 10 claude-fable-5 70.0 11. kimi-k3: 45.2 +/- 8.0 11 kimi-k3 45.2 12. kimi-k3@deepinfra: 45.2 +/- 8.0 12 kimi-k3@deepinfra 45.2 13. kimi-k3@moonshot: 45.2 +/- 8.0 13 kimi-k3@moonshot 45.2 14. kimi-k3.1-pro: 38.7 +/- 11.5 14 kimi-k3.1-pro 38.7 15. kimi-k3.1-pro@deepinfra: 38.7 +/- 11.5 15 kimi-k3.1-pro@deepinfra 38.7 16. kimi-k3.1-pro@together: 38.7 +/- 11.5 16 kimi-k3.1-pro@together 38.7 17. gpt-6.1-terra: 66.4 +/- 7.5 17 gpt-6.1-terra 66.4 18. gemini-4-hydrogen: 61.8 +/- 11.5 18 gemini-4-hydrogen 61.8 19. mimo-v2.6-pro: 36.4 +/- 11.3 19 mimo-v2.6-pro 36.4 20. gpt-6-sol: 48.9 +/- 11.2 20 gpt-6-sol 48.9 21. qwen3.8-max-0902: 37.2 +/- 7.3 21 qwen3.8-max-0902 37.2 22. deepseek-flash: 64.4 +/- 11.4 22 deepseek-flash 64.4 23. deepseek-flash-fast@parasail: 64.4 +/- 11.4 23 deepseek-flash-fast@parasail 64.4 24. deepseek-flash@alibaba-intl: 64.4 +/- 11.4 24 deepseek-flash@alibaba-intl 64.4 25. deepseek-flash@deepinfra: 64.4 +/- 11.4 25 deepseek-flash@deepinfra 64.4 26. deepseek-flash@parasail: 64.4 +/- 11.4 26 deepseek-flash@parasail 64.4 27. deepseek-flash@together: 64.4 +/- 11.4 27 deepseek-flash@together 64.4 33. deepseek-v4.1-flash-1017: 38.8 +/- 11.5 33 deepseek-v4.1-flash-1017 38.8 34. deepseek-v4.1-flash-1017@deepinfra: 38.8 +/- 11.5 34 deepseek-v4.1-flash-1017@deepinfra 38.8 35. deepseek-v4.1-flash-1017@parasail: 38.8 +/- 11.5 35 deepseek-v4.1-flash-1017@parasail 38.8 36. claude-opus-4-7: 43.3 +/- 9.6 36 claude-opus-4-7 43.3 37. glm-5.3-flash: 20.7 +/- 9.6 37 glm-5.3-flash 20.7 38. glm-5.3-flash@deepinfra: 20.7 +/- 9.6 38 glm-5.3-flash@deepinfra 20.7 40. claude-opus-4-8: 65.6 +/- 10.9 40 claude-opus-4-8 65.6 41. claude-opus-4-6: 33.3 +/- 10.0 41 claude-opus-4-6 33.3 42. gpt-5.6-sol: 20.0 +/- 9.2 42 gpt-5.6-sol 20.0 43. gemini-3.8-flash: 27.7 +/- 7.4 43 gemini-3.8-flash 27.7 45. claude-sonnet-5: 63.3 +/- 10.7 45 claude-sonnet-5 63.3 46. gemini-3.7-flash: 41.1 +/- 12.2 46 gemini-3.7-flash 41.1 47. glm-5.2: 36.7 +/- 11.5 47 glm-5.2 36.7 48. glm-5.2@alibaba-intl: 36.7 +/- 11.5 48 glm-5.2@alibaba-intl 36.7 49. glm-5.2@alibaba-us: 36.7 +/- 11.5 49 glm-5.2@alibaba-us 36.7 50. glm-5.2@deepinfra: 36.7 +/- 11.5 50 glm-5.2@deepinfra 36.7 52. grok-4.7: 49.9 +/- 8.2 52 grok-4.7 49.9 53. gpt-5.6-terra: 33.5 +/- 8.1 53 gpt-5.6-terra 33.5 54. grok-4.6: 29.9 +/- 7.2 54 grok-4.6 29.9 57. claude-haiku-5-5: 63.4 +/- 7.1 57 claude-haiku-5-5 63.4 59. gemini-3.6-flash: 23.3 +/- 9.3 59 gemini-3.6-flash 23.3 63. gpt-5.5: 23.3 +/- 10.1 63 gpt-5.5 23.3 65. deepseek-v4-pro-0813: 33.3 +/- 10.0 65 deepseek-v4-pro-0813 33.3 66. deepseek-v4-pro-0813@deepinfra: 33.3 +/- 10.0 66 deepseek-v4-pro-0813@deepinfra 33.3 67. gpt-6-luna: 25.7 +/- 6.8 67 gpt-6-luna 25.7 70. qwen3.8-27b: 15.9 +/- 8.6 70 qwen3.8-27b 15.9 71. qwen3.8-27b@deepinfra: 15.9 +/- 8.6 71 qwen3.8-27b@deepinfra 15.9 72. qwen3.8-27b@parasail: 15.9 +/- 8.6 72 qwen3.8-27b@parasail 15.9 75. gpt-5.6-luna: 21.1 +/- 9.8 75 gpt-5.6-luna 21.1 76. gemini-3.1-pro-preview: 8.9 +/- 4.6 76 gemini-3.1-pro-preview 8.9 77. minimax-m3.1-flash: 25.7 +/- 10.3 77 minimax-m3.1-flash 25.7 78. minimax-m3.1-flash@parasail: 25.7 +/- 10.3 78 minimax-m3.1-flash@parasail 25.7 82. gpt-5.4: 23.3 +/- 8.1 82 gpt-5.4 23.3 97. gemma-4-31b-it: 8.5 +/- 6.6 97 gemma-4-31b-it 8.5 98. mistral-large-4-0: 32.4 +/- 7.4 98 mistral-large-4-0 32.4 99. inkling: 9.8 +/- 7.0 99 inkling 9.8 105. gpt-5.2: 32.2 +/- 10.0 105 gpt-5.2 32.2 120. gpt-5: 4.4 +/- 3.4 120 gpt-5 4.4 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Rating only, not on the chart (102) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
mistral-small-24b-instruct-2501Rating 1,312.0 $0.21 gemma-3-4b-itRating 1,273.5 $0.24 gpt-oss-20bRating 1,368.3 $0.25 gemma-3-12b-itRating 1,315.8 $0.30 gpt-oss-120b@deepinfraRating 1,389.5 $0.30 phi-4Rating 1,305.7 $0.33 gemma-3-27b-itRating 1,357.8 $0.38 deepseek-v4-flash@deepinfraRating 1,581.6 $0.42 granite-4.2-8bRating 1,366.7 $0.46 qwen3-32bRating 1,406.0 $0.54 llama-4-scout-17b-16e-instructRating 1,362.5 $0.61 llama-3.3-70b-instruct-turbo@deepinfraRating 1,345.8 $0.63 gpt-5-nanoRating 1,385.4 $0.64 mimo-v2.6-flashRating 1,638.5 $0.66 deepseek-v4-flash@parasailRating 1,581.6 $0.66 gpt-4.1-nanoRating 1,373.0 $0.74 qwen3.5-flash-2026-02-23Rating 1,435.4 $0.74 qwen3-30b-a3bRating 1,385.0 $0.92 qwen3-235b-a22b-instruct-2507@deepinfraRating 1,471.9 $0.92 deepseek-v4-flashRating 1,581.6 $0.94 deepseek-v3.2Rating 1,469.4 $1.03 gpt-oss-120bRating 1,389.5 $1.11 gpt-4o-miniRating 1,347.6 $1.11 llama-3.3-70b-instruct-turbo@parasailRating 1,345.8 $1.12 qwen3-235b-a22b-instruct-2507@parasailRating 1,471.9 $1.36 qwen3-30b-a3b-instruct-2507Rating 1,437.9 $1.48 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,462.8 $1.59 qwen3.5-35b-a3b@deepinfraRating 1,434.5 $1.63 qwen3.5-35b-a3b@parasailRating 1,434.5 $1.65 qwen3-next-80b-a3b-instruct@deepinfraRating 1,444.1 $1.67 qwen3-next-80b-a3b-instruct@parasailRating 1,444.1 $1.69 qwen3-235b-a22b-instruct-2507Rating 1,471.9 $1.70 inkling-2-small@deepinfraRating 1,644.7 $1.85 qwen3-next-80b-a3b-instructRating 1,444.1 $1.92 qwen3-next-80b-a3b-thinkingRating 1,420.9 $1.92 inkling-2-smallRating 1,644.7 $1.95 minimax-m3@deepinfraRating 1,493.0 $2.05 gpt-5.4-nanoRating 1,459.6 $2.09 minimax-m3Rating 1,493.0 $2.22 minimax-m3@parasailRating 1,493.0 $2.22 qwen3.7-plusRating 1,500.3 $2.37 qwen3.7-plus@alibaba-usRating 1,500.3 $2.37 beam-1.0-501b-a23bRating 1,504.7 $2.42 qwen-plus-2025-01-25Rating 1,378.5 $2.42 inkling-smallRating 1,472.4 $2.52 gemini-3.1-flash-liteRating 1,457.5 $2.53 qwen3-vl-235b-a22b-instructRating 1,462.8 $2.96 gpt-4.1-miniRating 1,433.0 $2.96 qwen3-vl-235b-a22b-instruct@parasailRating 1,462.8 $2.99 mistral-large-2512Rating 1,471.0 $3.03 gpt-3.5-turboRating 1,274.8 $3.03 glm-4.7Rating 1,484.2 $3.16 gpt-5-miniRating 1,432.1 $3.20 qwen3.5-35b-a3bRating 1,434.5 $3.20 minimax-m3-fast@parasailRating 1,493.0 $3.33 llama-3.3-70b-instruct-turboRating 1,345.8 $3.48 qwen3-235b-a22b-thinking-2507Rating 1,441.9 $3.57 glm-4.6Rating 1,458.2 $3.70 qwen3.5-27bRating 1,451.5 $3.84 deepseek-r1-0528Rating 1,464.0 $3.90 gemini-3.5-flash-liteRating 1,502.1 $3.98 qwen3.5-27b@deepinfraRating 1,451.5 $4.03 glm-5.3@deepinfraRating 1,622.3 $4.50 qwen3.5-397b-a17b@deepinfraRating 1,490.2 $4.95 hy4-previewRating 1,631.7 $5.04 gemini-3-flash-previewRating 1,505.3 $5.05 qwen3.6-plusRating 1,496.3 $5.05 qwen3.5-122b-a10bRating 1,458.2 $5.12 qwen3.5-397b-a17b@parasailRating 1,490.2 $5.86 qwen3.5-397b-a17bRating 1,490.2 $6.06 deepseek-v4-pro@deepinfraRating 1,500.5 $6.11 qwen3-vl-235b-a22b-thinkingRating 1,452.0 $6.20 kimi-k2.6Rating 1,516.1 $6.23 gemini-3.5-flashRating 1,508.0 $6.56 kimi-k2.7-codeRating 1,474.5 $7.30 gpt-5.4-miniRating 1,494.9 $7.58 deepseek-v4-proRating 1,500.5 $7.99 deepseek-v4-pro@parasailRating 1,500.5 $7.99 deepseek-v4-pro@togetherRating 1,500.5 $7.99 o4-miniRating 1,433.1 $8.14 o3-miniRating 1,416.6 $8.14 glm-5.3Rating 1,622.3 $8.74 glm-5.3@alibaba-intlRating 1,622.3 $8.74 glm-5.3@parasailRating 1,622.3 $8.74 glm-5.1Rating 1,512.3 $8.74 claude-haiku-4-5Rating 1,481.5 $8.75 deepseek-v4-pro-fast@parasailRating 1,500.5 $11.98 glm-5.3-fast@parasailRating 1,622.3 $13.11 mistral-medium-3-5Rating 1,478.1 $13.13 qwen3.6-max-previewRating 1,510.0 $13.13 o3Rating 1,460.2 $14.80 gpt-4.1Rating 1,455.8 $14.80 qwen3.7-maxRating 1,524.9 $15.13 qwen3.7-max@alibaba-usRating 1,524.9 $15.13 gpt-5.1Rating 1,475.4 $16.00 gpt-4oRating 1,368.9 $18.50 claude-sonnet-4-6Rating 1,528.3 $26.25 claude-sonnet-4-5Rating 1,512.3 $26.25 claude-opus-4-5Rating 1,522.9 $43.75 gpt-4-turboRating 1,346.8 $60.50 o1Rating 1,432.7 $111 gpt-4Rating 1,329.8 $141 Fixing code Price against score, Hard 0 25 50 75 100 $1 $10 $100
gpt-oss-20b: 3.1 +/- 1.5, $0.25
gpt-oss-120b@deepinfra: 25.6 +/- 1.9, $0.30
gemma-3-27b-it: 4.9 +/- 1.4, $0.38
qwen3.5-9b@deepinfra: 9.2 +/- 2.4, $0.40
qwen3.5-9b@parasail: 9.2 +/- 2.4, $0.54
qwen3-32b: 40.0 +/- 3.3, $0.54
deepseek-v4.1-flash-1017@deepinfra: 80.4 +/- 8.1, $0.59
gpt-5-nano: 7.0 +/- 1.9, $0.64
qwen3.5-9b: 9.2 +/- 2.4, $0.68
gpt-4.1-nano: 8.9 +/- 1.9, $0.74
claude-haiku-5-5: 77.6 +/- 8.5, $0.88
gpt-6-luna: 70.8 +/- 9.3, $0.88
qwen3-235b-a22b-instruct-2507@deepinfra: 59.6 +/- 3.3, $0.92
gemma-4-31b-it: 20.3 +/- 8.2, $0.94
glm-5.3-flash: 70.9 +/- 9.3, $0.98
glm-5.3-flash@deepinfra: 70.9 +/- 9.3, $0.98
deepseek-v3.2: 39.6 +/- 2.8, $1.03
minimax-m3.1-flash: 68.8 +/- 9.5, $1.11
minimax-m3.1-flash@parasail: 68.8 +/- 9.5, $1.11
gpt-oss-120b: 25.6 +/- 1.9, $1.11
gpt-4o-mini: 3.6 +/- 1.2, $1.11
qwen3-235b-a22b-instruct-2507@parasail: 59.6 +/- 3.3, $1.36
qwen3.6-35b-a3b@parasail: 24.6 +/- 3.2, $1.42
qwen3.6-35b-a3b@deepinfra: 24.6 +/- 3.2, $1.48
qwen3-235b-a22b-instruct-2507: 59.6 +/- 3.3, $1.70
mimo-v2.6-pro: 64.5 +/- 9.8, $1.83
deepseek-v4.1-flash-1017: 80.4 +/- 8.1, $2.22
deepseek-v4.1-flash-1017@parasail: 80.4 +/- 8.1, $2.22
qwen3.8-27b@deepinfra: 69.3 +/- 9.4, $2.83
gpt-4.1-mini: 32.4 +/- 3.1, $2.96
glm-4.7: 33.3 +/- 2.5, $3.16
gpt-5-mini: 47.0 +/- 1.7, $3.20
qwen3.8-27b@parasail: 69.3 +/- 9.4, $3.45
glm-5.2@deepinfra: 78.7 +/- 1.9, $3.56
glm-4.6: 24.5 +/- 2.4, $3.70
qwen3.6-35b-a3b: 24.6 +/- 3.2, $3.79
deepseek-r1-0528: 71.4 +/- 3.0, $3.90
mistral-large-4-0: 79.4 +/- 8.3, $4.24
gemini-3-flash-preview: 66.1 +/- 1.7, $5.05
qwen3.8-27b: 69.3 +/- 9.4, $5.05
qwen3.6-plus: 57.9 +/- 2.3, $5.05
deepseek-v4-pro@deepinfra: 77.6 +/- 1.9, $6.11
kimi-k2.6: 76.7 +/- 1.9, $6.23
gemini-3.5-flash: 79.3 +/- 1.8, $6.56
gemini-3.8-flash: 73.3 +/- 9.0, $6.56
inkling: 48.7 +/- 10.2, $7.47
deepseek-v4-pro: 77.6 +/- 1.9, $7.99
deepseek-v4-pro@parasail: 77.6 +/- 1.9, $7.99
deepseek-v4-pro@together: 77.6 +/- 1.9, $7.99
o4-mini: 72.0 +/- 3.0, $8.14
o3-mini: 60.4 +/- 3.3, $8.14
glm-5.2: 78.7 +/- 1.9, $8.74
glm-5.2@alibaba-intl: 78.7 +/- 1.9, $8.74
glm-5.2@alibaba-us: 78.7 +/- 1.9, $8.74
glm-5.1: 74.2 +/- 2.0, $8.74
gpt-6.1-terra: 89.2 +/- 6.3, $8.75
claude-haiku-4-5: 13.9 +/- 2.7, $8.75
deepseek-v4-pro-fast@parasail: 77.6 +/- 1.9, $11.98
qwen3.8-max-0902: 60.9 +/- 10.0, $12.10
gemini-4-hydrogen: 78.5 +/- 8.4, $13.13
qwen3.6-max-preview: 76.7 +/- 1.9, $13.13
grok-4.6: 76.7 +/- 8.6, $13.31
grok-4.7: 73.1 +/- 9.1, $13.31
o3: 68.4 +/- 1.7, $14.80
gpt-4.1: 49.8 +/- 1.9, $14.80
qwen3.7-max: 77.3 +/- 1.9, $15.13
qwen3.7-max@alibaba-us: 77.3 +/- 1.9, $15.13
gpt-5: 63.4 +/- 1.4, $16.00
gpt-5.1: 59.5 +/- 1.7, $16.00
kimi-k3.1-pro@deepinfra: 67.8 +/- 9.5, $16.63
gemini-4-carbon: 91.0 +/- 5.8, $17.50
gemini-4-argon: 85.7 +/- 7.2, $17.50
gpt-6.1-sol: 82.1 +/- 7.8, $17.50
claude-sonnet-5-5: 77.2 +/- 8.6, $17.50
kimi-k3.1-pro: 67.8 +/- 9.5, $17.50
kimi-k3.1-pro@together: 67.8 +/- 9.5, $17.50
gpt-4o: 28.4 +/- 1.7, $18.50
gpt-5.6-terra: 84.2 +/- 7.5, $20.20
gemini-3.1-pro-preview: 61.4 +/- 4.1, $20.20
gpt-5.2: 65.5 +/- 1.7, $22.40
kimi-k3@deepinfra: 53.1 +/- 10.2, $24.94
gpt-5.4: 78.9 +/- 1.4, $25.25
claude-sonnet-4-6: 67.9 +/- 1.6, $26.25
claude-sonnet-4-5: 60.9 +/- 1.7, $26.25
kimi-k3: 53.1 +/- 10.2, $26.25
kimi-k3@moonshot: 53.1 +/- 10.2, $26.25
claude-opus-5-5: 87.1 +/- 6.8, $35.00
claude-opus-4-7: 82.1 +/- 1.3, $43.75
claude-opus-4-6: 75.8 +/- 1.5, $43.75
claude-opus-4-5: 72.1 +/- 1.6, $43.75
gpt-5.5: 66.1 +/- 2.5, $50.50
claude-fable-5-2: 79.2 +/- 8.3, $70.00
claude-fable-5-1: 89.7 +/- 6.2, $87.50
gpt-6-astra: 87.5 +/- 6.8, $87.50
o1: 61.7 +/- 3.2, $111
deepseek-v4.1-flash-1017@deepinfra Cheap gpt-6.1-terra Best, Balanced Ranking, Hard 1. gemini-3.5-flash: 79.3 +/- 1.8 1 gemini-3.5-flash 79.3 2. glm-5.2: 78.7 +/- 1.9 2 glm-5.2 78.7 3. glm-5.2@alibaba-intl: 78.7 +/- 1.9 3 glm-5.2@alibaba-intl 78.7 4. glm-5.2@alibaba-us: 78.7 +/- 1.9 4 glm-5.2@alibaba-us 78.7 5. glm-5.2@deepinfra: 78.7 +/- 1.9 5 glm-5.2@deepinfra 78.7 6. claude-opus-4-7: 82.1 +/- 1.3 6 claude-opus-4-7 82.1 7. deepseek-v4-pro: 77.6 +/- 1.9 7 deepseek-v4-pro 77.6 8. deepseek-v4-pro-fast@parasail: 77.6 +/- 1.9 8 deepseek-v4-pro-fast@parasail 77.6 9. deepseek-v4-pro@deepinfra: 77.6 +/- 1.9 9 deepseek-v4-pro@deepinfra 77.6 10. deepseek-v4-pro@parasail: 77.6 +/- 1.9 10 deepseek-v4-pro@parasail 77.6 11. deepseek-v4-pro@together: 77.6 +/- 1.9 11 deepseek-v4-pro@together 77.6 12. gemini-4-carbon: 91.0 +/- 5.8 12 gemini-4-carbon 91.0 13. qwen3.7-max: 77.3 +/- 1.9 13 qwen3.7-max 77.3 14. qwen3.7-max@alibaba-us: 77.3 +/- 1.9 14 qwen3.7-max@alibaba-us 77.3 15. kimi-k2.6: 76.7 +/- 1.9 15 kimi-k2.6 76.7 16. qwen3.6-max-preview: 76.7 +/- 1.9 16 qwen3.6-max-preview 76.7 17. claude-fable-5-1: 89.7 +/- 6.2 17 claude-fable-5-1 89.7 18. gpt-6.1-terra: 89.2 +/- 6.3 18 gpt-6.1-terra 89.2 19. gpt-6-astra: 87.5 +/- 6.8 19 gpt-6-astra 87.5 20. claude-opus-5-5: 87.1 +/- 6.8 20 claude-opus-5-5 87.1 21. glm-5.1: 74.2 +/- 2.0 21 glm-5.1 74.2 22. gpt-5.4: 78.9 +/- 1.4 22 gpt-5.4 78.9 23. gemini-4-argon: 85.7 +/- 7.2 23 gemini-4-argon 85.7 24. claude-opus-4-6: 75.8 +/- 1.5 24 claude-opus-4-6 75.8 25. gpt-5.6-terra: 84.2 +/- 7.5 25 gpt-5.6-terra 84.2 26. gpt-6.1-sol: 82.1 +/- 7.8 26 gpt-6.1-sol 82.1 27. claude-opus-4-5: 72.1 +/- 1.6 27 claude-opus-4-5 72.1 28. deepseek-v4.1-flash-1017: 80.4 +/- 8.1 28 deepseek-v4.1-flash-1017 80.4 29. deepseek-v4.1-flash-1017@deepinfra: 80.4 +/- 8.1 29 deepseek-v4.1-flash-1017@deepinfra 80.4 30. deepseek-v4.1-flash-1017@parasail: 80.4 +/- 8.1 30 deepseek-v4.1-flash-1017@parasail 80.4 31. o4-mini: 72.0 +/- 3.0 31 o4-mini 72.0 32. mistral-large-4-0: 79.4 +/- 8.3 32 mistral-large-4-0 79.4 33. deepseek-r1-0528: 71.4 +/- 3.0 33 deepseek-r1-0528 71.4 34. claude-fable-5-2: 79.2 +/- 8.3 34 claude-fable-5-2 79.2 35. gemini-4-hydrogen: 78.5 +/- 8.4 35 gemini-4-hydrogen 78.5 36. claude-haiku-5-5: 77.6 +/- 8.5 36 claude-haiku-5-5 77.6 37. claude-sonnet-5-5: 77.2 +/- 8.6 37 claude-sonnet-5-5 77.2 38. grok-4.6: 76.7 +/- 8.6 38 grok-4.6 76.7 39. claude-sonnet-4-6: 67.9 +/- 1.6 39 claude-sonnet-4-6 67.9 40. gemini-3-flash-preview: 66.1 +/- 1.7 40 gemini-3-flash-preview 66.1 41. gemini-3.8-flash: 73.3 +/- 9.0 41 gemini-3.8-flash 73.3 42. gpt-5.5: 66.1 +/- 2.5 42 gpt-5.5 66.1 43. o3: 68.4 +/- 1.7 43 o3 68.4 44. grok-4.7: 73.1 +/- 9.1 44 grok-4.7 73.1 45. gpt-5.2: 65.5 +/- 1.7 45 gpt-5.2 65.5 46. glm-5.3-flash: 70.9 +/- 9.3 46 glm-5.3-flash 70.9 47. glm-5.3-flash@deepinfra: 70.9 +/- 9.3 47 glm-5.3-flash@deepinfra 70.9 48. gpt-6-luna: 70.8 +/- 9.3 48 gpt-6-luna 70.8 49. qwen3.8-27b: 69.3 +/- 9.4 49 qwen3.8-27b 69.3 50. qwen3.8-27b@deepinfra: 69.3 +/- 9.4 50 qwen3.8-27b@deepinfra 69.3 51. qwen3.8-27b@parasail: 69.3 +/- 9.4 51 qwen3.8-27b@parasail 69.3 52. minimax-m3.1-flash: 68.8 +/- 9.5 52 minimax-m3.1-flash 68.8 53. minimax-m3.1-flash@parasail: 68.8 +/- 9.5 53 minimax-m3.1-flash@parasail 68.8 54. kimi-k3.1-pro: 67.8 +/- 9.5 54 kimi-k3.1-pro 67.8 55. kimi-k3.1-pro@deepinfra: 67.8 +/- 9.5 55 kimi-k3.1-pro@deepinfra 67.8 56. kimi-k3.1-pro@together: 67.8 +/- 9.5 56 kimi-k3.1-pro@together 67.8 57. gemini-3.1-pro-preview: 61.4 +/- 4.1 57 gemini-3.1-pro-preview 61.4 58. gpt-5: 63.4 +/- 1.4 58 gpt-5 63.4 59. o1: 61.7 +/- 3.2 59 o1 61.7 60. mimo-v2.6-pro: 64.5 +/- 9.8 60 mimo-v2.6-pro 64.5 61. claude-sonnet-4-5: 60.9 +/- 1.7 61 claude-sonnet-4-5 60.9 62. o3-mini: 60.4 +/- 3.3 62 o3-mini 60.4 63. qwen3.6-plus: 57.9 +/- 2.3 63 qwen3.6-plus 57.9 64. qwen3-235b-a22b-instruct-2507: 59.6 +/- 3.3 64 qwen3-235b-a22b-instruct-2507 59.6 65. qwen3-235b-a22b-instruct-2507@deepinfra: 59.6 +/- 3.3 65 qwen3-235b-a22b-instruct-2507@deepinfra 59.6 66. qwen3-235b-a22b-instruct-2507@parasail: 59.6 +/- 3.3 66 qwen3-235b-a22b-instruct-2507@parasail 59.6 67. qwen3.8-max-0902: 60.9 +/- 10.0 67 qwen3.8-max-0902 60.9 68. gpt-5.1: 59.5 +/- 1.7 68 gpt-5.1 59.5 69. kimi-k3: 53.1 +/- 10.2 69 kimi-k3 53.1 70. kimi-k3@deepinfra: 53.1 +/- 10.2 70 kimi-k3@deepinfra 53.1 71. kimi-k3@moonshot: 53.1 +/- 10.2 71 kimi-k3@moonshot 53.1 72. gpt-4.1: 49.8 +/- 1.9 72 gpt-4.1 49.8 73. inkling: 48.7 +/- 10.2 73 inkling 48.7 74. gpt-5-mini: 47.0 +/- 1.7 74 gpt-5-mini 47.0 75. deepseek-v3.2: 39.6 +/- 2.8 75 deepseek-v3.2 39.6 76. qwen3-32b: 40.0 +/- 3.3 76 qwen3-32b 40.0 77. gpt-4o: 28.4 +/- 1.7 77 gpt-4o 28.4 78. glm-4.7: 33.3 +/- 2.5 78 glm-4.7 33.3 79. gemma-4-31b-it: 20.3 +/- 8.2 79 gemma-4-31b-it 20.3 80. gpt-4.1-mini: 32.4 +/- 3.1 80 gpt-4.1-mini 32.4 81. qwen3.6-35b-a3b: 24.6 +/- 3.2 81 qwen3.6-35b-a3b 24.6 82. qwen3.6-35b-a3b@deepinfra: 24.6 +/- 3.2 82 qwen3.6-35b-a3b@deepinfra 24.6 83. qwen3.6-35b-a3b@parasail: 24.6 +/- 3.2 83 qwen3.6-35b-a3b@parasail 24.6 84. glm-4.6: 24.5 +/- 2.4 84 glm-4.6 24.5 85. gpt-oss-120b: 25.6 +/- 1.9 85 gpt-oss-120b 25.6 86. gpt-oss-120b@deepinfra: 25.6 +/- 1.9 86 gpt-oss-120b@deepinfra 25.6 87. claude-haiku-4-5: 13.9 +/- 2.7 87 claude-haiku-4-5 13.9 88. qwen3.5-9b: 9.2 +/- 2.4 88 qwen3.5-9b 9.2 89. qwen3.5-9b@deepinfra: 9.2 +/- 2.4 89 qwen3.5-9b@deepinfra 9.2 90. qwen3.5-9b@parasail: 9.2 +/- 2.4 90 qwen3.5-9b@parasail 9.2 91. gpt-5-nano: 7.0 +/- 1.9 91 gpt-5-nano 7.0 92. gpt-4.1-nano: 8.9 +/- 1.9 92 gpt-4.1-nano 8.9 93. gpt-oss-20b: 3.1 +/- 1.5 93 gpt-oss-20b 3.1 94. gemma-3-27b-it: 4.9 +/- 1.4 94 gemma-3-27b-it 4.9 95. gpt-4o-mini: 3.6 +/- 1.2 95 gpt-4o-mini 3.6 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model SQL and data queries Price against score, Medium 0 25 50 75 100 $0.1 $1 $10
deepseek-v4.1-flash-1017@deepinfra: 48.2 +/- 6.9, $0.14
gpt-6-luna: 49.6 +/- 6.9, $0.18
claude-haiku-5-5: 44.8 +/- 6.9, $0.18
glm-5.3-flash: 48.8 +/- 6.9, $0.22
glm-5.3-flash@deepinfra: 48.8 +/- 6.9, $0.22
minimax-m3.1-flash: 48.0 +/- 6.9, $0.24
minimax-m3.1-flash@parasail: 48.0 +/- 6.9, $0.24
gemma-4-31b-it: 46.3 +/- 6.9, $0.24
mimo-v2.6-pro: 48.3 +/- 6.9, $0.47
deepseek-v4.1-flash-1017: 48.2 +/- 6.9, $0.48
deepseek-v4.1-flash-1017@parasail: 48.2 +/- 6.9, $0.48
qwen3.8-27b@deepinfra: 48.8 +/- 6.9, $0.50
qwen3.8-27b@parasail: 48.8 +/- 6.9, $0.63
mistral-large-4-0: 48.2 +/- 6.9, $0.98
qwen3.8-27b: 48.8 +/- 6.9, $1.00
gemini-3.8-flash: 45.3 +/- 6.9, $1.35
inkling: 48.5 +/- 6.9, $1.61
gpt-6.1-terra: 56.5 +/- 6.9, $1.80
gemini-4-hydrogen: 56.7 +/- 6.9, $2.70
qwen3.8-max-0902: 50.1 +/- 6.9, $2.80
grok-4.7: 44.9 +/- 6.9, $3.08
grok-4.6: 43.1 +/- 6.9, $3.08
kimi-k3.1-pro@deepinfra: 49.1 +/- 6.9, $3.42
gemini-4-argon: 69.9 +/- 6.4, $3.60
gemini-4-carbon: 67.3 +/- 6.5, $3.60
gpt-6.1-sol: 57.4 +/- 6.9, $3.60
kimi-k3.1-pro: 49.1 +/- 6.9, $3.60
kimi-k3.1-pro@together: 49.1 +/- 6.9, $3.60
claude-sonnet-5-5: 39.8 +/- 6.8, $3.60
gpt-5.6-terra: 50.2 +/- 6.9, $4.00
kimi-k3@deepinfra: 48.4 +/- 6.9, $5.13
kimi-k3: 48.4 +/- 6.9, $5.40
kimi-k3@moonshot: 48.4 +/- 6.9, $5.40
claude-opus-5-5: 52.3 +/- 6.9, $7.20
claude-fable-5-2: 55.8 +/- 6.9, $14.40
gpt-6-astra: 58.4 +/- 6.8, $18.00
claude-fable-5-1: 56.2 +/- 6.9, $18.00
gpt-6.1-terra Cheap gemini-4-hydrogen Balanced gemini-4-argon Best Ranking, Medium 1. gemini-4-argon: 69.9 +/- 6.4 1 gemini-4-argon 69.9 2. gemini-4-carbon: 67.3 +/- 6.5 2 gemini-4-carbon 67.3 3. gpt-6-astra: 58.4 +/- 6.8 3 gpt-6-astra 58.4 4. gpt-6.1-sol: 57.4 +/- 6.9 4 gpt-6.1-sol 57.4 5. gemini-4-hydrogen: 56.7 +/- 6.9 5 gemini-4-hydrogen 56.7 6. gpt-6.1-terra: 56.5 +/- 6.9 6 gpt-6.1-terra 56.5 7. claude-fable-5-1: 56.2 +/- 6.9 7 claude-fable-5-1 56.2 8. claude-fable-5-2: 55.8 +/- 6.9 8 claude-fable-5-2 55.8 9. claude-opus-5-5: 52.3 +/- 6.9 9 claude-opus-5-5 52.3 10. gpt-5.6-terra: 50.2 +/- 6.9 10 gpt-5.6-terra 50.2 11. qwen3.8-max-0902: 50.1 +/- 6.9 11 qwen3.8-max-0902 50.1 12. gpt-6-luna: 49.6 +/- 6.9 12 gpt-6-luna 49.6 13. kimi-k3.1-pro: 49.1 +/- 6.9 13 kimi-k3.1-pro 49.1 14. kimi-k3.1-pro@deepinfra: 49.1 +/- 6.9 14 kimi-k3.1-pro@deepinfra 49.1 15. kimi-k3.1-pro@together: 49.1 +/- 6.9 15 kimi-k3.1-pro@together 49.1 16. glm-5.3-flash: 48.8 +/- 6.9 16 glm-5.3-flash 48.8 17. glm-5.3-flash@deepinfra: 48.8 +/- 6.9 17 glm-5.3-flash@deepinfra 48.8 18. qwen3.8-27b: 48.8 +/- 6.9 18 qwen3.8-27b 48.8 19. qwen3.8-27b@deepinfra: 48.8 +/- 6.9 19 qwen3.8-27b@deepinfra 48.8 20. qwen3.8-27b@parasail: 48.8 +/- 6.9 20 qwen3.8-27b@parasail 48.8 21. inkling: 48.5 +/- 6.9 21 inkling 48.5 22. kimi-k3: 48.4 +/- 6.9 22 kimi-k3 48.4 23. kimi-k3@deepinfra: 48.4 +/- 6.9 23 kimi-k3@deepinfra 48.4 24. kimi-k3@moonshot: 48.4 +/- 6.9 24 kimi-k3@moonshot 48.4 25. mimo-v2.6-pro: 48.3 +/- 6.9 25 mimo-v2.6-pro 48.3 26. deepseek-v4.1-flash-1017: 48.2 +/- 6.9 26 deepseek-v4.1-flash-1017 48.2 27. deepseek-v4.1-flash-1017@deepinfra: 48.2 +/- 6.9 27 deepseek-v4.1-flash-1017@deepinfra 48.2 28. deepseek-v4.1-flash-1017@parasail: 48.2 +/- 6.9 28 deepseek-v4.1-flash-1017@parasail 48.2 29. mistral-large-4-0: 48.2 +/- 6.9 29 mistral-large-4-0 48.2 30. minimax-m3.1-flash: 48.0 +/- 6.9 30 minimax-m3.1-flash 48.0 31. minimax-m3.1-flash@parasail: 48.0 +/- 6.9 31 minimax-m3.1-flash@parasail 48.0 32. gemma-4-31b-it: 46.3 +/- 6.9 32 gemma-4-31b-it 46.3 33. gemini-3.8-flash: 45.3 +/- 6.9 33 gemini-3.8-flash 45.3 34. grok-4.7: 44.9 +/- 6.9 34 grok-4.7 44.9 35. claude-haiku-5-5: 44.8 +/- 6.9 35 claude-haiku-5-5 44.8 36. grok-4.6: 43.1 +/- 6.9 36 grok-4.6 43.1 37. claude-sonnet-5-5: 39.8 +/- 6.8 37 claude-sonnet-5-5 39.8 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Using tools Price against score, Hard 0 25 50 75 100 $1 $10 $100
gpt-oss-20b: 3.1 +/- 1.5, $0.37
gpt-oss-120b@deepinfra: 14.2 +/- 2.3, $0.45
qwen3.5-9b@deepinfra: 9.2 +/- 2.4, $0.67
qwen3.5-9b@parasail: 9.2 +/- 2.4, $0.85
gpt-5-nano: 7.0 +/- 1.9, $0.92
deepseek-v4.1-flash-1017@deepinfra: 38.8 +/- 11.5, $0.92
qwen3.5-9b: 9.2 +/- 2.4, $1.13
claude-haiku-5-5: 47.9 +/- 11.8, $1.30
gpt-6-luna: 24.7 +/- 10.2, $1.30
glm-5.3-flash: 20.7 +/- 9.6, $1.50
glm-5.3-flash@deepinfra: 20.7 +/- 9.6, $1.50
gemma-4-31b-it: 8.5 +/- 6.6, $1.52
minimax-m3.1-flash: 25.7 +/- 10.3, $1.68
minimax-m3.1-flash@parasail: 25.7 +/- 10.3, $1.68
gpt-oss-120b: 14.2 +/- 2.3, $1.68
deepseek-v3.2: 39.6 +/- 2.8, $1.72
qwen3.6-35b-a3b@parasail: 24.6 +/- 3.2, $2.02
qwen3.6-35b-a3b@deepinfra: 24.6 +/- 3.2, $2.11
mimo-v2.6-pro: 36.4 +/- 11.3, $2.96
deepseek-v4.1-flash-1017: 38.8 +/- 11.5, $3.36
deepseek-v4.1-flash-1017@parasail: 38.8 +/- 11.5, $3.36
qwen3.8-27b@deepinfra: 15.9 +/- 8.6, $3.98
gpt-5-mini: 22.3 +/- 2.6, $4.60
glm-4.7: 33.3 +/- 2.5, $4.75
qwen3.8-27b@parasail: 15.9 +/- 8.6, $4.92
qwen3.6-35b-a3b: 24.6 +/- 3.2, $5.55
glm-4.6: 24.5 +/- 2.4, $5.60
mistral-large-4-0: 24.7 +/- 10.2, $6.58
gemini-3-flash-preview: 47.4 +/- 3.0, $7.40
qwen3.8-27b: 15.9 +/- 8.6, $7.40
gemini-3.8-flash: 16.5 +/- 8.8, $9.75
inkling: 9.8 +/- 7.0, $11.29
gpt-6.1-terra: 56.2 +/- 11.7, $13.00
claude-haiku-4-5: 13.9 +/- 2.7, $13.00
qwen3.8-max-0902: 26.6 +/- 10.4, $18.80
gemini-4-hydrogen: 61.8 +/- 11.5, $19.50
grok-4.7: 45.3 +/- 11.7, $20.68
grok-4.6: 18.7 +/- 9.2, $20.68
o3: 9.1 +/- 1.5, $22.40
gpt-5.1: 47.6 +/- 2.8, $23.00
gpt-5: 33.9 +/- 2.9, $23.00
kimi-k3.1-pro@deepinfra: 38.7 +/- 11.5, $24.70
gpt-6.1-sol: 65.6 +/- 11.2, $26.00
claude-sonnet-5-5: 63.4 +/- 11.4, $26.00
gemini-4-carbon: 72.7 +/- 10.5, $26.00
gemini-4-argon: 58.5 +/- 11.6, $26.00
kimi-k3.1-pro: 38.7 +/- 11.5, $26.00
kimi-k3.1-pro@together: 38.7 +/- 11.5, $26.00
gpt-4o: 8.6 +/- 2.1, $28.00
gemini-3.1-pro-preview: 61.4 +/- 4.1, $29.60
gpt-5.6-terra: 25.9 +/- 10.3, $29.60
gpt-5.2: 54.0 +/- 2.9, $32.20
gpt-5.4: 81.8 +/- 2.0, $37.00
kimi-k3@deepinfra: 36.0 +/- 11.3, $37.05
claude-sonnet-4-6: 62.8 +/- 1.8, $39.00
claude-sonnet-4-5: 41.5 +/- 1.9, $39.00
kimi-k3: 36.0 +/- 11.3, $39.00
kimi-k3@moonshot: 36.0 +/- 11.3, $39.00
claude-opus-5-5: 72.8 +/- 10.5, $52.00
claude-opus-4-7: 80.2 +/- 2.1, $65.00
claude-opus-4-6: 69.9 +/- 2.5, $65.00
claude-opus-4-5: 64.7 +/- 1.8, $65.00
gpt-5.5: 66.1 +/- 2.5, $74.00
claude-fable-5-2: 82.9 +/- 8.9, $104
gpt-6-astra: 75.6 +/- 10.1, $130
claude-fable-5-1: 74.2 +/- 10.3, $130
gpt-6.1-terra Cheap claude-fable-5-2 Best, Balanced Ranking, Hard 1. claude-fable-5-2: 82.9 +/- 8.9 1 claude-fable-5-2 82.9 2. gpt-5.4: 81.8 +/- 2.0 2 gpt-5.4 81.8 3. claude-opus-4-7: 80.2 +/- 2.1 3 claude-opus-4-7 80.2 4. gpt-6-astra: 75.6 +/- 10.1 4 gpt-6-astra 75.6 5. claude-opus-5-5: 72.8 +/- 10.5 5 claude-opus-5-5 72.8 6. claude-fable-5-1: 74.2 +/- 10.3 6 claude-fable-5-1 74.2 7. claude-opus-4-6: 69.9 +/- 2.5 7 claude-opus-4-6 69.9 8. gpt-6.1-sol: 65.6 +/- 11.2 8 gpt-6.1-sol 65.6 9. claude-sonnet-5-5: 63.4 +/- 11.4 9 claude-sonnet-5-5 63.4 10. gpt-5.5: 66.1 +/- 2.5 10 gpt-5.5 66.1 11. gemini-4-carbon: 72.7 +/- 10.5 11 gemini-4-carbon 72.7 12. gemini-4-hydrogen: 61.8 +/- 11.5 12 gemini-4-hydrogen 61.8 13. claude-opus-4-5: 64.7 +/- 1.8 13 claude-opus-4-5 64.7 14. gpt-6.1-terra: 56.2 +/- 11.7 14 gpt-6.1-terra 56.2 15. claude-sonnet-4-6: 62.8 +/- 1.8 15 claude-sonnet-4-6 62.8 20. gpt-5.2: 54.0 +/- 2.9 20 gpt-5.2 54.0 21. gemini-4-argon: 58.5 +/- 11.6 21 gemini-4-argon 58.5 22. gpt-5.1: 47.6 +/- 2.8 22 gpt-5.1 47.6 23. gemini-3-flash-preview: 47.4 +/- 3.0 23 gemini-3-flash-preview 47.4 25. gemini-3.1-pro-preview: 61.4 +/- 4.1 25 gemini-3.1-pro-preview 61.4 26. grok-4.7: 45.3 +/- 11.7 26 grok-4.7 45.3 29. claude-haiku-5-5: 47.9 +/- 11.8 29 claude-haiku-5-5 47.9 30. kimi-k3.1-pro: 38.7 +/- 11.5 30 kimi-k3.1-pro 38.7 31. kimi-k3.1-pro@deepinfra: 38.7 +/- 11.5 31 kimi-k3.1-pro@deepinfra 38.7 32. kimi-k3.1-pro@together: 38.7 +/- 11.5 32 kimi-k3.1-pro@together 38.7 33. claude-sonnet-4-5: 41.5 +/- 1.9 33 claude-sonnet-4-5 41.5 34. deepseek-v3.2: 39.6 +/- 2.8 34 deepseek-v3.2 39.6 35. deepseek-v4.1-flash-1017: 38.8 +/- 11.5 35 deepseek-v4.1-flash-1017 38.8 36. deepseek-v4.1-flash-1017@deepinfra: 38.8 +/- 11.5 36 deepseek-v4.1-flash-1017@deepinfra 38.8 37. deepseek-v4.1-flash-1017@parasail: 38.8 +/- 11.5 37 deepseek-v4.1-flash-1017@parasail 38.8 38. gpt-5: 33.9 +/- 2.9 38 gpt-5 33.9 39. glm-4.7: 33.3 +/- 2.5 39 glm-4.7 33.3 51. mimo-v2.6-pro: 36.4 +/- 11.3 51 mimo-v2.6-pro 36.4 52. kimi-k3: 36.0 +/- 11.3 52 kimi-k3 36.0 53. kimi-k3@deepinfra: 36.0 +/- 11.3 53 kimi-k3@deepinfra 36.0 54. kimi-k3@moonshot: 36.0 +/- 11.3 54 kimi-k3@moonshot 36.0 56. glm-4.6: 24.5 +/- 2.4 56 glm-4.6 24.5 57. qwen3.6-35b-a3b: 24.6 +/- 3.2 57 qwen3.6-35b-a3b 24.6 58. qwen3.6-35b-a3b@deepinfra: 24.6 +/- 3.2 58 qwen3.6-35b-a3b@deepinfra 24.6 59. qwen3.6-35b-a3b@parasail: 24.6 +/- 3.2 59 qwen3.6-35b-a3b@parasail 24.6 60. minimax-m3.1-flash: 25.7 +/- 10.3 60 minimax-m3.1-flash 25.7 61. minimax-m3.1-flash@parasail: 25.7 +/- 10.3 61 minimax-m3.1-flash@parasail 25.7 62. qwen3.8-max-0902: 26.6 +/- 10.4 62 qwen3.8-max-0902 26.6 63. gpt-5.6-terra: 25.9 +/- 10.3 63 gpt-5.6-terra 25.9 64. gpt-5-mini: 22.3 +/- 2.6 64 gpt-5-mini 22.3 65. gpt-6-luna: 24.7 +/- 10.2 65 gpt-6-luna 24.7 66. mistral-large-4-0: 24.7 +/- 10.2 66 mistral-large-4-0 24.7 67. grok-4.6: 18.7 +/- 9.2 67 grok-4.6 18.7 68. glm-5.3-flash: 20.7 +/- 9.6 68 glm-5.3-flash 20.7 69. glm-5.3-flash@deepinfra: 20.7 +/- 9.6 69 glm-5.3-flash@deepinfra 20.7 70. gemini-3.8-flash: 16.5 +/- 8.8 70 gemini-3.8-flash 16.5 71. qwen3.8-27b: 15.9 +/- 8.6 71 qwen3.8-27b 15.9 72. qwen3.8-27b@deepinfra: 15.9 +/- 8.6 72 qwen3.8-27b@deepinfra 15.9 73. qwen3.8-27b@parasail: 15.9 +/- 8.6 73 qwen3.8-27b@parasail 15.9 74. gpt-oss-120b: 14.2 +/- 2.3 74 gpt-oss-120b 14.2 75. gpt-oss-120b@deepinfra: 14.2 +/- 2.3 75 gpt-oss-120b@deepinfra 14.2 76. claude-haiku-4-5: 13.9 +/- 2.7 76 claude-haiku-4-5 13.9 80. qwen3.5-9b: 9.2 +/- 2.4 80 qwen3.5-9b 9.2 81. qwen3.5-9b@deepinfra: 9.2 +/- 2.4 81 qwen3.5-9b@deepinfra 9.2 82. qwen3.5-9b@parasail: 9.2 +/- 2.4 82 qwen3.5-9b@parasail 9.2 85. gpt-5-nano: 7.0 +/- 1.9 85 gpt-5-nano 7.0 86. gemma-4-31b-it: 8.5 +/- 6.6 86 gemma-4-31b-it 8.5 87. gpt-oss-20b: 3.1 +/- 1.5 87 gpt-oss-20b 3.1 92. inkling: 9.8 +/- 7.0 92 inkling 9.8 93. o3: 9.1 +/- 1.5 93 o3 9.1 94. gpt-4o: 8.6 +/- 2.1 94 gpt-4o 8.6 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Rating only, not on the chart (30) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
mimo-v2.6-flashRating -0.0 $1.06 inkling-2-small@deepinfraRating 0.0 $2.85 inkling-2-smallRating 0.0 $3.00 minimax-m3@deepinfraRating -0.1 $3.10 minimax-m3Rating -0.1 $3.36 minimax-m3@parasailRating -0.1 $3.36 qwen3.7-plusRating -0.1 $3.58 qwen3.7-plus@alibaba-usRating -0.1 $3.58 beam-1.0-501b-a23bRating -0.1 $3.76 inkling-smallRating -0.2 $3.96 minimax-m3-fast@parasailRating -0.1 $5.04 glm-5.2@deepinfraRating 0.0 $5.49 glm-5.3@deepinfraRating 0.0 $6.75 hy4-previewRating 0.1 $7.84 gemini-3.7-flashRating -0.0 $9.75 glm-5.2Rating 0.0 $13.52 glm-5.2@alibaba-intlRating 0.0 $13.52 glm-5.2@alibaba-usRating 0.0 $13.52 glm-5.3Rating 0.0 $13.52 glm-5.3@alibaba-intlRating 0.0 $13.52 glm-5.3@parasailRating 0.0 $13.52 mistral-medium-3-5Rating -0.1 $19.50 glm-5.3-fast@parasailRating 0.0 $20.28 qwen3.7-maxRating -0.1 $23.50 qwen3.7-max@alibaba-usRating -0.1 $23.50 gpt-6-solRating 0.1 $26.00 claude-sonnet-5Rating 0.0 $26.00 gpt-5.6-solRating 0.0 $52.00 claude-opus-5Rating 0.1 $65.00 claude-fable-5Rating 0.1 $130 Reading images Nobody publishes a scored result for this one yet. The models below have a preference rating only.
Ranking, Medium Rating only, not on the chart (82) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
gemma-3-27b-itRating 1,164.4 $0.13 gpt-5-nanoRating 1,159.1 $0.20 llama-4-scout-17b-16e-instructRating 1,118.4 $0.20 mimo-v2.6-flashRating 1,259.6 $0.22 gpt-4.1-nanoRating 1,063.1 $0.24 deepseek-flash@deepinfraRating 1,279.2 $0.28 gpt-6-lunaRating 1,211.2 $0.28 glm-5.3-flashRating 1,294.7 $0.32 glm-5.3-flash@deepinfraRating 1,294.7 $0.32 gemma-4-31b-itRating 1,279.4 $0.32 gpt-4o-miniRating 1,065.5 $0.36 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,246.6 $0.51 mimo-v2.6-proRating 1,265.9 $0.62 gpt-5.6-lunaRating 1,255.8 $0.64 gpt-5.4-nanoRating 1,195.5 $0.66 minimax-m3@deepinfraRating 1,249.2 $0.66 deepseek-flashRating 1,279.2 $0.72 deepseek-flash@alibaba-intlRating 1,279.2 $0.72 deepseek-flash@parasailRating 1,279.2 $0.72 deepseek-flash@togetherRating 1,279.2 $0.72 minimax-m3Rating 1,249.2 $0.72 minimax-m3@parasailRating 1,249.2 $0.72 qwen3.7-plusRating 1,277.3 $0.77 qwen3.7-plus@alibaba-usRating 1,277.3 $0.77 gemini-3.1-flash-liteRating 1,240.3 $0.80 inkling-smallRating 1,235.9 $0.84 qwen3.8-27b@deepinfraRating 1,269.6 $0.87 qwen3-vl-235b-a22b-instruct@parasailRating 1,246.6 $0.93 qwen3-vl-235b-a22b-instructRating 1,246.6 $0.96 gpt-4.1-miniRating 1,181.3 $0.96 mistral-large-2512Rating 1,221.8 $1.00 gpt-5-miniRating 1,202.5 $1.00 qwen3.8-27b@parasailRating 1,269.6 $1.07 deepseek-flash-fast@parasailRating 1,279.2 $1.08 minimax-m3-fast@parasailRating 1,249.2 $1.08 qwen3.5-27bRating 1,239.6 $1.20 gemini-3.5-flash-liteRating 1,268.0 $1.24 qwen3.5-27b@deepinfraRating 1,239.6 $1.25 qwen3.5-397b-a17b@deepinfraRating 1,262.4 $1.56 gemini-3-flash-previewRating 1,284.6 $1.60 qwen3.8-27bRating 1,269.6 $1.60 qwen3.5-122b-a10bRating 1,244.2 $1.60 qwen3.5-397b-a17b@parasailRating 1,262.4 $1.84 qwen3.5-397b-a17bRating 1,262.4 $1.92 qwen3-vl-235b-a22b-thinkingRating 1,206.3 $1.92 kimi-k2.6Rating 1,282.4 $2.00 gemini-3.8-flashRating 1,316.8 $2.10 gemini-3.7-flashRating 1,313.1 $2.10 gemini-3.5-flashRating 1,307.6 $2.10 gemini-3.6-flashRating 1,296.7 $2.10 gpt-5.4-miniRating 1,244.4 $2.40 o4-miniRating 1,194.0 $2.64 qwen3.8-max-0902Rating 1,312.0 $4.00 mistral-medium-3-5Rating 1,222.4 $4.20 grok-4.6Rating 1,264.0 $4.40 grok-4.7Rating 1,230.0 $4.40 o3Rating 1,213.7 $4.80 gpt-4.1Rating 1,210.2 $4.80 gpt-5.1Rating 1,235.0 $5.00 gpt-5Rating 1,208.0 $5.00 gpt-6.1-solRating 1,285.9 $5.60 claude-sonnet-5-5Rating 1,284.3 $5.60 claude-sonnet-5Rating 1,271.6 $5.60 gpt-6-solRating 1,232.7 $5.60 gpt-4oRating 1,136.5 $6.00 gemini-3.1-pro-previewRating 1,296.0 $6.40 gpt-5.6-terraRating 1,269.1 $6.40 gpt-5.2Rating 1,228.8 $7.00 gpt-5.4Rating 1,291.7 $8.00 claude-sonnet-4-6Rating 1,282.7 $8.40 claude-opus-5-5Rating 1,323.8 $11.20 gpt-5.6-solRating 1,282.2 $11.20 claude-opus-4-7Rating 1,318.7 $14.00 claude-opus-5Rating 1,317.5 $14.00 claude-opus-4-6Rating 1,312.2 $14.00 claude-opus-4-8Rating 1,288.6 $14.00 gpt-5.5Rating 1,295.8 $16.00 gpt-4-turboRating 1,089.8 $20.00 claude-fable-5Rating 1,325.6 $28.00 claude-fable-5-1Rating 1,321.6 $28.00 gpt-6-astraRating 1,278.5 $28.00 o1Rating 1,168.3 $36.00 Questions about a document Nobody publishes a scored result for this one yet. The models below have a preference rating only.
Ranking, Medium Rating only, not on the chart (157) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
gpt-oss-20bRating 1,303.2 $0.15 gpt-oss-120b@deepinfraRating 1,324.4 $0.19 mistral-small-24b-instruct-2501Rating 1,279.7 $0.19 gemma-3-4b-itRating 1,312.0 $0.20 gemma-3-12b-itRating 1,346.6 $0.22 phi-4Rating 1,267.1 $0.28 granite-4.2-8bRating 1,305.5 $0.29 gemma-3-27b-itRating 1,365.6 $0.32 gpt-5-nanoRating 1,331.6 $0.32 deepseek-v4-flash@deepinfraRating 1,448.0 $0.36 qwen3-32bRating 1,355.1 $0.37 llama-4-scout-17b-16e-instructRating 1,325.5 $0.44 llama-3.3-70b-instruct-turbo@deepinfraRating 1,313.6 $0.45 qwen3.5-flash-2026-02-23Rating 1,399.9 $0.48 gpt-4.1-nanoRating 1,325.3 $0.48 qwen3-235b-a22b-instruct-2507@deepinfraRating 1,433.0 $0.51 gpt-6-lunaRating 1,448.5 $0.52 claude-haiku-5-5Rating 1,419.0 $0.52 mimo-v2.6-flashRating 1,457.5 $0.56 deepseek-v4-flash@parasailRating 1,448.0 $0.56 qwen3-30b-a3bRating 1,338.0 $0.58 deepseek-flash@deepinfraRating 1,480.8 $0.62 glm-5.3-flashRating 1,479.7 $0.68 glm-5.3-flash@deepinfraRating 1,479.7 $0.68 gpt-4o-miniRating 1,326.0 $0.72 gpt-oss-120bRating 1,324.4 $0.72 qwen3-next-80b-a3b-instruct@deepinfraRating 1,389.5 $0.73 qwen3-next-80b-a3b-instruct@parasailRating 1,389.5 $0.76 qwen3-235b-a22b-instruct-2507@parasailRating 1,433.0 $0.77 gemma-4-31b-itRating 1,468.8 $0.80 deepseek-v4-flashRating 1,448.0 $0.80 qwen3.5-35b-a3b@deepinfraRating 1,399.1 $0.85 qwen3.5-35b-a3b@parasailRating 1,399.1 $0.88 llama-3.3-70b-instruct-turbo@parasailRating 1,313.6 $0.90 qwen3-next-80b-a3b-instructRating 1,389.5 $0.96 qwen3-30b-a3b-instruct-2507Rating 1,381.2 $0.96 qwen3-next-80b-a3b-thinkingRating 1,370.3 $0.96 deepseek-v3.2Rating 1,440.8 $0.98 qwen3-vl-235b-a22b-instruct@deepinfraRating 1,420.0 $0.99 qwen3-235b-a22b-instruct-2507Rating 1,433.0 $1.10 gpt-5.6-lunaRating 1,451.9 $1.12 gpt-5.4-nanoRating 1,402.5 $1.14 qwen3.8-27b@deepinfraRating 1,442.6 $1.23 inkling-2-small@deepinfraRating 1,409.6 $1.29 minimax-m3@deepinfraRating 1,445.3 $1.34 inkling-2-smallRating 1,409.6 $1.36 gemini-3.1-flash-liteRating 1,429.5 $1.40 qwen3-vl-235b-a22b-instruct@parasailRating 1,420.0 $1.43 deepseek-flashRating 1,480.8 $1.44 deepseek-flash@alibaba-intlRating 1,480.8 $1.44 deepseek-flash@parasailRating 1,480.8 $1.44 deepseek-flash@togetherRating 1,480.8 $1.44 minimax-m3Rating 1,445.3 $1.44 minimax-m3@parasailRating 1,445.3 $1.44 qwen3.7-plusRating 1,464.9 $1.54 qwen3.7-plus@alibaba-usRating 1,464.9 $1.54 mimo-v2.6-proRating 1,495.9 $1.55 qwen3.5-35b-a3bRating 1,399.1 $1.60 gpt-5-miniRating 1,370.5 $1.60 qwen3.8-27b@parasailRating 1,442.6 $1.65 qwen3-235b-a22b-thinking-2507Rating 1,402.0 $1.66 beam-1.0-501b-a23bRating 1,438.1 $1.76 qwen-plus-2025-01-25Rating 1,362.6 $1.76 qwen3.5-27b@deepinfraRating 1,422.5 $1.87 qwen3.5-27bRating 1,422.5 $1.92 qwen3-vl-235b-a22b-instructRating 1,420.0 $1.92 inkling-smallRating 1,400.5 $1.92 gpt-4.1-miniRating 1,389.7 $1.92 gemini-3.5-flash-liteRating 1,462.5 $1.96 glm-4.7Rating 1,451.2 $1.98 deepseek-flash-fast@parasailRating 1,480.8 $2.16 minimax-m3-fast@parasailRating 1,445.3 $2.16 mistral-large-2512Rating 1,418.5 $2.20 gpt-3.5-turboRating 1,222.9 $2.20 glm-4.6Rating 1,431.6 $2.40 deepseek-r1-0528Rating 1,407.7 $2.46 glm-5.2@deepinfraRating 1,485.7 $2.52 qwen3.5-122b-a10bRating 1,417.8 $2.56 qwen3.5-397b-a17b@deepinfraRating 1,453.5 $2.64 glm-5.3@deepinfraRating 1,488.4 $2.80 gemini-3-flash-previewRating 1,476.0 $2.80 qwen3.6-plusRating 1,454.6 $2.80 qwen3.8-27bRating 1,442.6 $2.80 qwen3-vl-235b-a22b-thinkingRating 1,403.5 $2.88 qwen3.5-397b-a17b@parasailRating 1,453.5 $3.04 qwen3.5-397b-a17bRating 1,453.5 $3.36 llama-3.3-70b-instruct-turboRating 1,313.6 $3.74 kimi-k2.6Rating 1,477.6 $3.80 gemini-3.8-flashRating 1,510.8 $3.90 gemini-3.7-flashRating 1,501.0 $3.90 gemini-3.6-flashRating 1,485.1 $3.90 gemini-3.5-flashRating 1,483.1 $3.90 gpt-5.4-miniRating 1,447.6 $4.20 inklingRating 1,442.3 $4.82 deepseek-v4-pro-0813@deepinfraRating 1,474.3 $5.20 deepseek-v4-pro@deepinfraRating 1,471.8 $5.20 claude-haiku-4-5Rating 1,435.8 $5.20 o4-miniRating 1,369.3 $5.28 o3-miniRating 1,359.5 $5.28 deepseek-v4-pro-0813Rating 1,474.3 $5.81 deepseek-v4-proRating 1,471.8 $5.81 deepseek-v4-pro@parasailRating 1,471.8 $5.81 deepseek-v4-pro@togetherRating 1,471.8 $5.81 glm-5.3Rating 1,488.4 $6.24 glm-5.3@alibaba-intlRating 1,488.4 $6.24 glm-5.3@parasailRating 1,488.4 $6.24 glm-5.2Rating 1,485.7 $6.24 glm-5.2@alibaba-intlRating 1,485.7 $6.24 glm-5.2@alibaba-usRating 1,485.7 $6.24 glm-5.1Rating 1,479.7 $6.24 qwen3.6-max-previewRating 1,473.4 $7.28 gemini-4-hydrogenRating 1,514.1 $7.80 mistral-medium-3-5Rating 1,429.3 $7.80 gpt-5.1Rating 1,442.3 $8.00 gpt-5Rating 1,412.8 $8.00 deepseek-v4-pro-fast@parasailRating 1,471.8 $8.71 qwen3.8-max-0902Rating 1,496.0 $8.80 glm-5.3-fast@parasailRating 1,488.4 $9.36 gpt-4.1Rating 1,426.2 $9.60 o3Rating 1,411.2 $9.60 grok-4.6Rating 1,476.0 $9.68 grok-4.7Rating 1,453.9 $9.68 kimi-k3.1-pro@deepinfraRating 1,490.7 $9.88 gemini-4-argonRating 1,543.8 $10.40 gemini-4-carbonRating 1,538.1 $10.40 claude-sonnet-5-5Rating 1,490.5 $10.40 kimi-k3.1-proRating 1,490.7 $10.40 kimi-k3.1-pro@togetherRating 1,490.7 $10.40 gpt-6.1-solRating 1,487.8 $10.40 claude-sonnet-5Rating 1,480.7 $10.40 gpt-6-solRating 1,465.8 $10.40 qwen3.7-maxRating 1,491.0 $11.00 qwen3.7-max@alibaba-usRating 1,491.0 $11.00 gemini-3.1-pro-previewRating 1,498.8 $11.20 gpt-5.6-terraRating 1,468.9 $11.20 gpt-5.2Rating 1,441.2 $11.20 gpt-4oRating 1,339.0 $12.00 gpt-5.4Rating 1,477.2 $14.00 kimi-k3@deepinfraRating 1,500.1 $14.82 kimi-k3Rating 1,500.1 $15.60 kimi-k3@moonshotRating 1,500.1 $15.60 claude-sonnet-4-6Rating 1,492.2 $15.60 claude-sonnet-4-5Rating 1,481.8 $15.60 claude-opus-5-5Rating 1,520.0 $20.80 gpt-5.6-solRating 1,497.8 $20.80 claude-opus-4-6Rating 1,516.5 $26.00 claude-opus-4-7Rating 1,507.5 $26.00 claude-opus-5Rating 1,501.6 $26.00 claude-opus-4-8Rating 1,499.0 $26.00 claude-opus-4-5Rating 1,492.7 $26.00 gpt-5.5Rating 1,486.4 $28.00 gpt-4-turboRating 1,317.9 $44.00 claude-fable-5Rating 1,522.1 $52.00 claude-fable-5-1Rating 1,509.5 $52.00 gpt-6-astraRating 1,485.3 $52.00 o1Rating 1,412.5 $72.00 gpt-4Rating 1,280.9 $120 Price against score, Hard 0 25 50 75 100 $1 $10 $100
gpt-oss-120b@deepinfra: 36.1 +/- 5.0, $0.45
gpt-5-nano: 44.4 +/- 5.0, $0.76
gemma-3-27b-it: 33.3 +/- 5.0, $0.78
qwen3-32b: 74.2 +/- 5.0, $0.89
llama-4-scout-17b-16e-instruct: 36.0 +/- 5.0, $1.07
llama-3.3-70b-instruct-turbo@deepinfra: 33.3 +/- 5.0, $1.09
gpt-4.1-nano: 25.0 +/- 5.0, $1.16
qwen3-14b: 62.5 +/- 5.0, $1.18
qwen3-30b-a3b: 40.6 +/- 5.0, $1.41
qwen3-next-80b-a3b-instruct@deepinfra: 41.7 +/- 5.0, $1.71
gpt-oss-120b: 36.1 +/- 5.0, $1.74
qwen3-next-80b-a3b-instruct@parasail: 41.7 +/- 5.0, $1.79
llama-3.3-70b-instruct-turbo@parasail: 33.3 +/- 5.0, $2.21
qwen3-next-80b-a3b-instruct: 41.7 +/- 5.0, $2.28
gpt-5-mini: 69.4 +/- 5.0, $3.80
qwen3-235b-a22b-thinking-2507: 75.0 +/- 5.0, $3.91
gpt-4.1-mini: 44.4 +/- 5.0, $4.64
deepseek-r1-0528: 75.0 +/- 5.0, $5.94
llama-3.3-70b-instruct-turbo: 33.3 +/- 5.0, $9.26
o4-mini: 77.8 +/- 5.0, $12.76
o3-mini: 50.0 +/- 5.0, $12.76
gpt-5: 97.2 +/- 5.0, $19.00
o3: 88.9 +/- 5.0, $23.20
gpt-4.1: 63.9 +/- 5.0, $23.20
o1: 83.3 +/- 5.0, $174
Ranking, Hard 1. gpt-5: 97.2 +/- 5.0 1 gpt-5 97.2 5. o3: 88.9 +/- 5.0 5 o3 88.9 8. o1: 83.3 +/- 5.0 8 o1 83.3 9. o4-mini: 77.8 +/- 5.0 9 o4-mini 77.8 10. deepseek-r1-0528: 75.0 +/- 5.0 10 deepseek-r1-0528 75.0 11. qwen3-235b-a22b-thinking-2507: 75.0 +/- 5.0 11 qwen3-235b-a22b-thinking-2507 75.0 13. qwen3-32b: 74.2 +/- 5.0 13 qwen3-32b 74.2 17. gpt-5-mini: 69.4 +/- 5.0 17 gpt-5-mini 69.4 19. gpt-4.1: 63.9 +/- 5.0 19 gpt-4.1 63.9 21. qwen3-14b: 62.5 +/- 5.0 21 qwen3-14b 62.5 29. o3-mini: 50.0 +/- 5.0 29 o3-mini 50.0 33. gpt-4.1-mini: 44.4 +/- 5.0 33 gpt-4.1-mini 44.4 34. gpt-5-nano: 44.4 +/- 5.0 34 gpt-5-nano 44.4 35. qwen3-next-80b-a3b-instruct: 41.7 +/- 5.0 35 qwen3-next-80b-a3b-instruct 41.7 36. qwen3-next-80b-a3b-instruct@deepinfra: 41.7 +/- 5.0 36 qwen3-next-80b-a3b-instruct@deepinfra 41.7 37. qwen3-next-80b-a3b-instruct@parasail: 41.7 +/- 5.0 37 qwen3-next-80b-a3b-instruct@parasail 41.7 38. qwen3-30b-a3b: 40.6 +/- 5.0 38 qwen3-30b-a3b 40.6 40. gpt-oss-120b: 36.1 +/- 5.0 40 gpt-oss-120b 36.1 41. gpt-oss-120b@deepinfra: 36.1 +/- 5.0 41 gpt-oss-120b@deepinfra 36.1 42. llama-4-scout-17b-16e-instruct: 36.0 +/- 5.0 42 llama-4-scout-17b-16e-instruct 36.0 50. gemma-3-27b-it: 33.3 +/- 5.0 50 gemma-3-27b-it 33.3 51. llama-3.3-70b-instruct-turbo: 33.3 +/- 5.0 51 llama-3.3-70b-instruct-turbo 33.3 52. llama-3.3-70b-instruct-turbo@deepinfra: 33.3 +/- 5.0 52 llama-3.3-70b-instruct-turbo@deepinfra 33.3 53. llama-3.3-70b-instruct-turbo@parasail: 33.3 +/- 5.0 53 llama-3.3-70b-instruct-turbo@parasail 33.3 55. gpt-4.1-nano: 25.0 +/- 5.0 55 gpt-4.1-nano 25.0 Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Tied with the next model Rating only, not on the chart (34) A preference rating is not a share of right answers, so these models cannot share the axis. They are listed with their rating instead.
gemma-4-31b-itRating 1,434.3 $1.96 gpt-5.6-lunaRating 1,453.8 $2.68 minimax-m3@deepinfraRating 1,436.9 $3.23 minimax-m3Rating 1,436.9 $3.48 minimax-m3@parasailRating 1,436.9 $3.48 qwen3.7-plusRating 1,437.4 $3.71 qwen3.7-plus@alibaba-usRating 1,437.4 $3.71 minimax-m3-fast@parasailRating 1,436.9 $5.22 gemini-3-flash-previewRating 1,414.3 $6.70 kimi-k2.6Rating 1,450.5 $9.15 gemini-3.8-flashRating 1,481.9 $9.38 gemini-3.7-flashRating 1,476.5 $9.38 gemini-3.5-flashRating 1,461.8 $9.38 gemini-3.6-flashRating 1,456.1 $9.38 claude-haiku-4-5Rating 1,420.4 $12.50 gpt-5.1Rating 1,403.0 $19.00 grok-4.6Rating 1,438.1 $23.54 claude-sonnet-5Rating 1,459.5 $25.00 gpt-5.2Rating 1,402.1 $26.60 gpt-5.6-terraRating 1,467.7 $26.80 gemini-3.1-pro-previewRating 1,443.3 $26.80 gpt-5.4Rating 1,473.0 $33.50 claude-sonnet-4-6Rating 1,481.9 $37.50 claude-sonnet-4-5Rating 1,449.3 $37.50 gpt-5.6-solRating 1,480.0 $50.00 claude-opus-5Rating 1,508.7 $62.50 claude-opus-4-6Rating 1,506.2 $62.50 claude-opus-4-7Rating 1,494.8 $62.50 claude-opus-4-5Rating 1,465.7 $62.50 claude-opus-4-8Rating 1,465.1 $62.50 gpt-5.5Rating 1,480.9 $67.00 claude-fable-5-1Rating 1,514.8 $125 claude-fable-5Rating 1,496.6 $125 gpt-6-astraRating 1,467.3 $125 Price per 1,000 tasks, USD. Score, %. Greyed out: a cheaper model already scores as well. Ringed: the best, balanced and cheap choice.