I cannot see cost per task for any other thinking levels for Qwen 4 Max.
If it's actually there, what is the cost at medium thinking? It would show Qwen 4 Max (medium) if it's there.
Everywhere that I can see on the page it just shows ”Qwen 4 Max" while for 6.1 Sol, Opus 5.5 there's always a thinking level in brackets after: (low), (medium), (max) etc.
Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
In case it translates to Qwen 4 27B, I've been running Qwen 3.8 Flash Next (which while MoE (and bigger) is supposed to be a preview of Qwen 4's architecture) and it runs pretty solidly regardless of context window size and completes random deepswe challenges I throw at it locally (at Q2-Q3 quantization! so it fits on my shitty setup).
The model I’m referring to that would be faster and similar size, but doesn’t handle images is Qwen 3.8 2.4T: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b. It’s weird that this is pretty much the same as Max but doesn’t support images. No idea why.
This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.
I'm also hoping someone puts out a competitive 30-70B model soon. There's this massive dead zone between 24GB and <100GB of VRAM where it seems your best option is Qwen 3.8 27B.
No idea, but this is a rapidly changing field. Opus 4.5 was released a little less than a year ago and it feels shockingly limited compared to even current open flash models. (I literally just tested it the other day.)
So, with new models coming out on a weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest expectations. I honestly don't know.
That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.
On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.
I can't imagine them keeping up long term given the information asymmetry plus compute restrictions on top of it.
I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.
So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.
I think so but it may be a very short term effect, I think their pace has roughly matched the pre-speed up pace of the frontier labs (the 1.7-2x speed up noted on by METR), and I’d guess we see a similar speed up from the Chinese labs once their internal models hit an equivalent level enabling the same speed up.
Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?
If the speed up is the reason, then they will never catch up (with traditional means). Because it means RSI predictions are real, and we will see an explosive growth in capabilities (far beyond exponential).
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
I feel like the xhigh result is one of the first times I've seen a model go for a non-red colored bicycle (teal). Still haven't seen the leftward facing bike though.
First thing I do when a new model drop is scroll down to find the pelican comment, and then the "the pelican benchmark is no longer useful" debate under it.
The past few days I’ve been finding myself asking “seriously, where is it”… feeling that the world has somehow changed because Qwen 4 27B was later than I expected.
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.
Compared to OpenAI and Anthropic, yes. There's this export ban on Nvidia chips, after all. And all the CPUs they have won't help much with model training.
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
It's great fun asking it questions like "who is the Dalai Lama?", "where is TSMC headquartered?", "why do people post Winnie the Pooh in reference to Xi?", "who controls the South China Sea?", "why was Chiang Kai Shek fighting Mao?", "why were students in Tiananmen Square?"
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
I've done some work on this and a lot of the refusals are in the API. The weights are more permissive at these types of things. Try it again when we get some of the smaller Qwen 4 weights.
You can watch it happen in DeepSeek's chat. You'll see it spit out the real response in the browser the first time, then the text gets cleared and it says it can't help you with that.
I suspect this happens when you run the inference in China or first party from the model maker. Out of all the open weight models I've used on US located hardware, they're fine answering these.
Qwen3.8 flash next handles these questions just fine, what should I expect to see? It correctly reports who the Dalai Lama is, that TSLC is in Taiwan (Hsinchu), explains why people post Winnie the Pooh memes, etc.
Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
From a far off perspective, can we really assume any model to be free of bias (political or otherwise)? I don't know - but would like to see - how the political opinions of reddit users and blog authors are reflected in that of AI, as AI is trained on human (biased) sources.
It scores a 43 on Artificial Analysis with an average cost of $3.27 [1]. For reference, GPT-6.1 Sol (medium) gets a 46 with an average of $0.36.
[1]: https://artificialanalysis.ai/models/qwen4-max
The table on that page shows gpt 6.1 sol (max) at $1.27 per task and it's the only model cheaper than Qwen 4 max.
By comparison opus 5.5 (max) is $10.01.
If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
The Artificial Analysis charts show max effort by default, which isn't how most people use them.
Scroll down to the cost per task pareto front chart which includes more difficult levels by default: https://artificialanalysis.ai/models/qwen4-max#intelligence-...
Qwen4-Max is not in a good spot on that chart. Astra xhigh and Fable 5.2 medium had a similar cost per task with higher intelligence ratings.
I cannot see cost per task for any other thinking levels for Qwen 4 Max.
If it's actually there, what is the cost at medium thinking? It would show Qwen 4 Max (medium) if it's there.
Everywhere that I can see on the page it just shows ”Qwen 4 Max" while for 6.1 Sol, Opus 5.5 there's always a thinking level in brackets after: (low), (medium), (max) etc.
How “safe” is Qwen4? Will it immediately shut down on security research like the US frontier models?
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
They compare different reasoning levels.
https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...
Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
The cost per task metric is accurate for those benchmark tasks. If your tasks don’t look like benchmark tasks..you should use your own benchmarks :)
Lol, I did that with https://aibenchy.com
I mean not believing intelligence score is reasonable but cost is just reporting what they spent?
I believe low cost but not high cost, since a model will work really hard when a question looks benchmark-like.
Looks like a slightly weaker, cheaper Kimi.
Artificial Analysis is a VC-backed firm and it's unclear what the incentives are there.
They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that v5.0.0 set on the first link Opus 5.5 has a lower score than in v5.0.1, and v5.0.1 was released just before Fable dropped.
Some part of me believes that the benchmark mix was put together to favor Anthropic's models.
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
It looks pretty bad on those benchmarks. Is it possible that this is due to the recent crackdown on distillation?
I think it's the most plausible explanation.
I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
[1]: https://agentduel.com/leaderboard
In case it translates to Qwen 4 27B, I've been running Qwen 3.8 Flash Next (which while MoE (and bigger) is supposed to be a preview of Qwen 4's architecture) and it runs pretty solidly regardless of context window size and completes random deepswe challenges I throw at it locally (at Q2-Q3 quantization! so it fits on my shitty setup).
> but we couldn’t include it because it doesn’t handle images
Maybe I'm mixing up model variants, but Qwen 3.8 27B that I run locally totally does handle images. Surely the bigger one would as well?
The model I’m referring to that would be faster and similar size, but doesn’t handle images is Qwen 3.8 2.4T: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b. It’s weird that this is pretty much the same as Max but doesn’t support images. No idea why.
Qwen3.8 doesn't handle images?
Sorry that was confusing. Qwen 3.8 2.4T is faster than what we have on leaderboard but it doesn’t handle image input: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b
This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.
Looking forward to the "Flash" and 27B versions.
I'm also hoping someone puts out a competitive 30-70B model soon. There's this massive dead zone between 24GB and <100GB of VRAM where it seems your best option is Qwen 3.8 27B.
There are many engines that alleviate this issue in particular. The current mood is running qwen flash next using strata, basalt, ect.
is the open weights lag widening? these open models don't seem to be improving as quickly as ant/oai
No idea, but this is a rapidly changing field. Opus 4.5 was released a little less than a year ago and it feels shockingly limited compared to even current open flash models. (I literally just tested it the other day.)
So, with new models coming out on a weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest expectations. I honestly don't know.
That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.
On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.
I can't imagine them keeping up long term given the information asymmetry plus compute restrictions on top of it.
I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.
So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.
open always wins
What advantage do they have over closed? You may want open to win, but logically I don’t see how it would
Right now open is heavily subsidized by China and I fail to see what would happen if that stopped.
Show us your stock holdings
I think so but it may be a very short term effect, I think their pace has roughly matched the pre-speed up pace of the frontier labs (the 1.7-2x speed up noted on by METR), and I’d guess we see a similar speed up from the Chinese labs once their internal models hit an equivalent level enabling the same speed up.
Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?
If the speed up is the reason, then they will never catch up (with traditional means). Because it means RSI predictions are real, and we will see an explosive growth in capabilities (far beyond exponential).
This is Qwen4-Max, which is not open weights.
Their open weights models are the the smaller ones, which aren't released yet.
Yeah, it turns out that the distillation theory has legs.
Might just be the commercial offerings scaling up parameters so much? Like for like the open models don’t seem to be lagging.
either way, ant/oai models dominating the intelligence-cost frontier is not a great outcome for those of us who prefer open weights
ant/oai have stalled in the past 6 mos,
Qwen4-Max pelicans at four reasoning efforts (none, low, high, xhigh): https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
For comparison, here's one I got from Qwen3.8 Max (at default reasoning effort): https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
Here are those 3.8 Max 0902 pelicans: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - they're mostly not as good as 4, but I prefer the 3.8 xhigh one.
I feel like the xhigh result is one of the first times I've seen a model go for a non-red colored bicycle (teal). Still haven't seen the leftward facing bike though.
Astra and the OpenAI models do this color as well
Evidence from where they distilling from /s
Feels like Pelicanbench is getting saturated now. Maybe time to switch to a different animal and vehicle combo?
The 3d one still produces interesting variations - https://www.reddit.com/r/ClaudeCode/comments/1xs4ll7/fable_5...
First thing I do when a new model drop is scroll down to find the pelican comment, and then the "the pelican benchmark is no longer useful" debate under it.
I kinda like the no-effort result more than low, but high and xhigh return to what one would expect for reasoning->quality.
Any info on when the 27B or FN Qwen 4 models are coming out?
The past few days I’ve been finding myself asking “seriously, where is it”… feeling that the world has somehow changed because Qwen 4 27B was later than I expected.
Hoping it’s a great local model though!
Artificial Analysis score doesn't look too hot. Doesn't even surpass Kimi 3.1
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.
alibaba has "limited compute"?
Compared to OpenAI and Anthropic, yes. There's this export ban on Nvidia chips, after all. And all the CPUs they have won't help much with model training.
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
Yeah, I don't know why people bother with it anymore. It no longer reflects what you're likely to see in real-world usage, at least IME.
To be fair, a lot of the popular benchmarks have become pretty meaningless.
Models often get great scores despite being a lot worse than the frontier in practice.
This is true, but if you look at the change in weighting to specific benchmarks from one version of AA to the next, it looks really arbitrary.
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Same for me coding tasks taking $20-50 are easily done by deepseek in $5-10
Anthropic finally figured out how to block them from distilling Opus.
how so ?
> Artificial Analysis
Ah yes, the reverse-benchmaxers of OpenAI models.
Ah yes, another new account without any backup to the claim.
I bet most of us waiting for 27b model and want to see how good it will be.
I’m a QFN dude myself, but yes can’t wait to get it loaded on a spark.
M5U locked and loaded.
I wish there were more ~60b dense options like ripple too!
My 3060 yearns for the next generation MoE!
Spooling up the V100s right now...
It's great fun asking it questions like "who is the Dalai Lama?", "where is TSMC headquartered?", "why do people post Winnie the Pooh in reference to Xi?", "who controls the South China Sea?", "why was Chiang Kai Shek fighting Mao?", "why were students in Tiananmen Square?"
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
I've done some work on this and a lot of the refusals are in the API. The weights are more permissive at these types of things. Try it again when we get some of the smaller Qwen 4 weights.
You can watch it happen in DeepSeek's chat. You'll see it spit out the real response in the browser the first time, then the text gets cleared and it says it can't help you with that.
I suspect this happens when you run the inference in China or first party from the model maker. Out of all the open weight models I've used on US located hardware, they're fine answering these.
Anecdotal though, N = 1.
Qwen3.8 flash next handles these questions just fine, what should I expect to see? It correctly reports who the Dalai Lama is, that TSLC is in Taiwan (Hsinchu), explains why people post Winnie the Pooh memes, etc.
Chinese history is closed-weight. Chinese trade is open-weight.
American history is open-weight. American trade is closed-weight.
Every time a major open weight is released, artificial analysis updates to a new one (v5) now. Exaggeration but some of this feels very apt (its also hilarious) https://www.reddit.com/r/LocalLLM/comments/1xkhha2/mr_incred...
Disappointing performance for a 2.4T model in the end of 2026.
Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
It is on openrouter: https://openrouter.ai/qwen/qwen4-max
I would assume it will not be on Bedrock for some time, AWS has not kept up with the latest non-American models.
Anyone else worried these models have poison pills embedded in the weights? (Past examples: https://cohere.com/blog/concerning-behaviour-popular-open-we...)
No technically serious person is buying the Cohere FUD you posted. I'd implore you to dig deeper and come to your own conclusion.
Does it provide an objective response when asked, "What happened in China on June 4, 1989"?
From a far off perspective, can we really assume any model to be free of bias (political or otherwise)? I don't know - but would like to see - how the political opinions of reddit users and blog authors are reflected in that of AI, as AI is trained on human (biased) sources.