The chart that I pay attention to is Intelligence Index vs. Output Tokens per Task[0]--it penalizes models running at 'max' thinking with lower intelligence and more churn to get to the answer.
The proprietary models 'cheat' on this since they generate but don't send output thinking tokens to be counted. So to grade them we have to use cost or time.
US dominance of the paraeto frontier is really striking. All over hackernews you hear about how open source models are "good enough but a lot cheaper" - it seems to me that they would be better served by using a frontier model on low thinking.
The chart that I pay attention to is Intelligence Index vs. Output Tokens per Task[0]--it penalizes models running at 'max' thinking with lower intelligence and more churn to get to the answer.
The proprietary models 'cheat' on this since they generate but don't send output thinking tokens to be counted. So to grade them we have to use cost or time.
[0] https://artificialanalysis.ai/?intelligence-efficiency=outpu...
US dominance of the paraeto frontier is really striking. All over hackernews you hear about how open source models are "good enough but a lot cheaper" - it seems to me that they would be better served by using a frontier model on low thinking.