More benchmark noise. They’re already problematic per se, and the error bars only get bigger by aggregating and adding costs on top of it.
Create your own, private evaluation system that reflects your use and your constraints. Focus on hard cases you encountered, find cases that break on one tier and not on another.
Exactly, according to these benchmarks Gemini 3.8 Flash is the best option at that budget level for coding, but in real world testing, I found DeepSeek flash to perform far better at a much lower price.
I've been running a quant/tune of Qwen3.8 27B on my M1 Max 32gb MacBook. That plus a good pi setup is having great results. I've used a full q8 of the model before and I dont see a real difference other than how slow it is. But leaving it running overnight on tasks is working great. It is currently debugging some issues in a native Mac Swift application and getting through the list of issues just fine.
If you have a 24-64GB mac, consider running Qwen3.8 27B locally at night. It's a bit slower to run locally, but if you're sleeping it's less of a problem.
Depending on your memory, you'll need to use the weaker Q4 versions but they still perform well.
It ranks higher than GPT-5.3 Codex (xhigh) or Claude Opus 4.6 (max) so is great for pairing with https://github.com/kunchenguid/gnhf for nightly experimentation, cleanup, or recommendation lists for in the morning.
You're better off going directly to artificial analysis, this is a feature-poor/misleading/outdated repackaging
Ex. this type of price estimation is quite naive - some models can require 2-3x the number of tokens to achieve the same level of intelligence. Artificial Analysis' own cost per task is a more fair estimation of cost.
I'd really like something that's more oriented around subscription fees.
If I want to spend $100 on LLMs next month, what should I do? Get Claude because Opus 5.5/Fable 5.1 are scoring well? Get Grok because 4.7 is supposedly a good mix of competence and cost? Try out a Chinese model? Don't do a subscription at all like this site is saying?
What's impeding a lab from releasing its own new model, pricing it really low for the beautiful Pareto plot, accompanied by phrases VC love like "establishing a new frontier in cost", to just then raise prices back up?
It compares Artificial Analysis scores against usage limits on Opencode Go, so I can see where to waste my quota most efficiently. Updated whenever I feel like it.
Does anyone actually pay API costs out of their own pocket? It's about 10x cheaper to just get a codex or chat gpt subscription, it's so heavily subsidized compared to the API that I'm sure it would be cheaper to use frontier models on a subscription plan rather than paying API prices for deepseek flash.
I do. For local dev work, I'm mostly using jetbrains' Junie, I can swap between a collection of models from google, openai, anthrophic.
I've had more than a few people tell me "oh, it's so much cheaper to use a $20 claude account" or "i've never hit a limit ever using my openai". Inevitably.. I end up reading/hearing "oh, I need to give it another couple hours to start using it again"... I've never hit that with my approach, even if it's costing me a bit more. Being able to work when I want when I have time has some value.
I also have openai and anthropic direct API billing set up for hosted and client projects that need to call out to an LLM service.
Why not use a codex or claude subscription? If you use the entire usage allotment on the $200 plan it's about $2,000 in equivalent API costs. Switching providers may be valuable but it's quite literally an order of magnitude cheaper.
I do. Sharing training data with OpenAI gives me a lot of complementary tokens. I go above that but it's still quite economical and I pick the right model for the task (Luna for most).
I thought Air was JB's multi-model interface? What is Junie? (I see the buttons, but am very confused by JB's AI offerings in general)
Is it worth the ~10x extra cost over the subscriptions? (This is obviously a leading question). Also, I think you can use OpenAI's subcription login with Air, but not Claude's.
I pay $100 for Codex and it last about a day in the weekly limit - mostly Astra and Sol.
Then i got $20 into DeepSeek and i've been using those $20 for two weeks every day now. Use case is automating computer/browser use - Astra is really good at it, but very expensive, Sol and Luna haven't been that great at it, Deepseek as at about 80% of Astra but lasts forever.
I do, but via OpenRouter. Outside of work my use cases are small and cheaper models do great job at those. I noticed even if I "burn tokens like crazy" I still pay less than any subscription available (a few $ a month).
But I guess if I had an agent vibecoding on it's own, I'd go with subscription instantly.
You can't use an API key on subscriptions, but I've gotten around it using the `codex exec` command to run requests outside the CLI or GUI if you're already authenticated on that machine. Won't work for all cases, but I've never ran into a limitation in my use case of not having an API key.
As the time goes on it only becomes harder to differentiate between model capabilities with just one or two numbers. I would love to see some kind of multi-axis placement of all the models on less objective attributes, like wordiness, willingness to give up, an ability to "think ahead" and pre-solve possible problems in code, for example, that I didn't think of or didn't think of talking about, etc etc etc.
For example I've been really enjoying Deepseek v4.1 Flash, it's very "straightforward" to the point of being almost dumb sometimes, but it's absolutely relentless and would solve almost any problem no matter how inefficient the solution is.
No idea how to measure all that, just average CoT length per task is probably a good approximation for some things, but not others.
1. In real life, most of us use token packages like OpenCode Go etc.
It would be handy to have a site like this one that takes into account the various deals and attempts to calculate the number of tokens per monthly fee for a chosen model. I realize this makes the task a lot more difficult.
2. It would be handy to have a chart like that for the AI hardware that people own. It helps you decide which model to run (resulting in different levels of intelligence and speed). Also difficult to please everyone (preprocessing vs token generation for example) and to keep updated!
I found https://llm-list.com/ yesterday and when I had a detailed look, I quickly found outdated entries, for example looking at GLM 5.3 flash it listed several providers as "free" that weren't free any longer.
Fun, but this seems to assume all LLM run on SAAS subscriptions. I would like to compare this to local LLM costs by converting my usage load x hardware costs into a token price. In addition if it games the LLM so often, these eventually are optimized and basically cheat on the score.
Is there a cheaper model than Gemini 3.8 Flash (High) that maybe/kind-of is on-par with it? For me it works really good but hit the limit in two hours tops... last week was the first time I hit the weekly limit and had to wait 4 days... Claude patches OK, but that is also getting drained really fast these days...
yeah I enjoy the speed of Gemini, but I also just have the low tier one (I use the 20x Claude and Codex subs for most of my work). For iteration, Gemini is so much fun, but Opus 5.5 is quite fast, as is sol. If you're on a budget, deepseek does look great
Yeah, I mean, I think in theory I could push it to the next tier on Gemini, but at the same time I wouldn't mind trying something else, since maybe my workloads are not really that smart and I am wasting a lot of computational power on something a cheaper model with similar capabilities can do.
Coding and math graphs are very interesting. Extremely cheap models make it into the upper echelon, delivering 90% of the performance for 1% of the price compared to the #1.
I find it interesting that with the given metric comparison, for coding at min 50 strength, every frontier model brand is from a distinct vendor: Ling, Qwen, Gemini, Muse, Grok, GPT, and Claude in increasing value.
If you maintain this over time, maybe include other sources than just AA and update the design to look less like zero-shot claude styling (I know that font! I know that color! Lol) it's genuinely useful :)
all I ever wanted is an updated website where I can see the best models I can run on my different devices locally, I don't get why people are throwing money at these companies
Depends on what level of intelligence you're wanting to use. A vanishingly small number of people can or would want to go to the hardware expense of running something like GLM 5.3 Flash, much less something like K3.
And if you want Astra/Fable/Opus frontier level, then there's no option at all.
But if you don't need that, or you don't need speed... That opens up the discussion. I've been impressed even with how Siri's been doing with the Apple Foundation Models in MacOS/iOS 27 given how small they are.
Edit: I can't even fully spec the M5 Ultra Mac Studio you'd need for GLM5.3 Flash since 512GB isn't available yet, but it's already at $9500 for 256GB RAM.
> A vanishingly small number of people can or would want to go to the hardware expense of running something like GLM 5.3 Flash, much less something like K3.
It's probably worth letting the user specify their actual costs in such a tool. I run a Framework Desktop 128GB that I bought before memory prices got crazy; the current retail price is almost double what I actually paid a year ago.
Best comparison that has occurred to me is the cost a loaf of bread's ingredients might be slightly cheaper than a baked loaf, depending on how you source it. At home you get total control and know what's going in to it. Yet bake at home is still a niche, perhaps a hobby. So I say as someone who's spent hundreds of hours tinkering with local inference, go for it for anyone reading. But most people just want ... some slices of bread, you know?
More benchmark noise. They’re already problematic per se, and the error bars only get bigger by aggregating and adding costs on top of it.
Create your own, private evaluation system that reflects your use and your constraints. Focus on hard cases you encountered, find cases that break on one tier and not on another.
Exactly, according to these benchmarks Gemini 3.8 Flash is the best option at that budget level for coding, but in real world testing, I found DeepSeek flash to perform far better at a much lower price.
I've been running a quant/tune of Qwen3.8 27B on my M1 Max 32gb MacBook. That plus a good pi setup is having great results. I've used a full q8 of the model before and I dont see a real difference other than how slow it is. But leaving it running overnight on tasks is working great. It is currently debugging some issues in a native Mac Swift application and getting through the list of issues just fine.
This is the one that works good for me on 32gb:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
Specifically this one: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
If you have a 24-64GB mac, consider running Qwen3.8 27B locally at night. It's a bit slower to run locally, but if you're sleeping it's less of a problem.
Depending on your memory, you'll need to use the weaker Q4 versions but they still perform well.
It ranks higher than GPT-5.3 Codex (xhigh) or Claude Opus 4.6 (max) so is great for pairing with https://github.com/kunchenguid/gnhf for nightly experimentation, cleanup, or recommendation lists for in the morning.
On my 64gb m3 max qwen3.8 27b has been great for planning and then letting qwen3.6 35ba3b actually implement the planned changes.
You might be interested in https://prismml.com/news/bonsai-2-27b
You're better off going directly to artificial analysis, this is a feature-poor/misleading/outdated repackaging
Ex. this type of price estimation is quite naive - some models can require 2-3x the number of tokens to achieve the same level of intelligence. Artificial Analysis' own cost per task is a more fair estimation of cost.
Agree :) direct link for those curious:
https://artificialanalysis.ai/models#price-cost
https://artificialanalysis.ai/models?models=gpt-6-luna%2Cglm...
I'd really like something that's more oriented around subscription fees.
If I want to spend $100 on LLMs next month, what should I do? Get Claude because Opus 5.5/Fable 5.1 are scoring well? Get Grok because 4.7 is supposedly a good mix of competence and cost? Try out a Chinese model? Don't do a subscription at all like this site is saying?
Or get a 20$ subscription from each major provider and have 40$ spare for openrouter credits. That’s what I’m doing.
The problem is that subscriptions with AI models get a bit 'vague' on what you get and how (or when) you might be rate-limited.
What's impeding a lab from releasing its own new model, pricing it really low for the beautiful Pareto plot, accompanied by phrases VC love like "establishing a new frontier in cost", to just then raise prices back up?
I wrote (i.e. let AI write) something like this for Opencode Go: https://6bj.de/aaogo/go_value_report.html
It compares Artificial Analysis scores against usage limits on Opencode Go, so I can see where to waste my quota most efficiently. Updated whenever I feel like it.
Does anyone actually pay API costs out of their own pocket? It's about 10x cheaper to just get a codex or chat gpt subscription, it's so heavily subsidized compared to the API that I'm sure it would be cheaper to use frontier models on a subscription plan rather than paying API prices for deepseek flash.
>heavily subsidized
Do you think this will be a problem in the future?
I do. For local dev work, I'm mostly using jetbrains' Junie, I can swap between a collection of models from google, openai, anthrophic.
I've had more than a few people tell me "oh, it's so much cheaper to use a $20 claude account" or "i've never hit a limit ever using my openai". Inevitably.. I end up reading/hearing "oh, I need to give it another couple hours to start using it again"... I've never hit that with my approach, even if it's costing me a bit more. Being able to work when I want when I have time has some value.
I also have openai and anthropic direct API billing set up for hosted and client projects that need to call out to an LLM service.
Why not use a codex or claude subscription? If you use the entire usage allotment on the $200 plan it's about $2,000 in equivalent API costs. Switching providers may be valuable but it's quite literally an order of magnitude cheaper.
I do. Sharing training data with OpenAI gives me a lot of complementary tokens. I go above that but it's still quite economical and I pick the right model for the task (Luna for most).
I thought Air was JB's multi-model interface? What is Junie? (I see the buttons, but am very confused by JB's AI offerings in general)
Is it worth the ~10x extra cost over the subscriptions? (This is obviously a leading question). Also, I think you can use OpenAI's subcription login with Air, but not Claude's.
I pay $100 for Codex and it last about a day in the weekly limit - mostly Astra and Sol.
Then i got $20 into DeepSeek and i've been using those $20 for two weeks every day now. Use case is automating computer/browser use - Astra is really good at it, but very expensive, Sol and Luna haven't been that great at it, Deepseek as at about 80% of Astra but lasts forever.
I do, 3-4$ a month of deep seek is enough for my usage
I do, but via OpenRouter. Outside of work my use cases are small and cheaper models do great job at those. I noticed even if I "burn tokens like crazy" I still pay less than any subscription available (a few $ a month).
But I guess if I had an agent vibecoding on it's own, I'd go with subscription instantly.
I use local LLMs on my Mac Mini.
Otherwise DeepSeek Flash 4.1 is dirt cheap (other "Flash" models are not that expensive either). I pay (very few dollars) out of my own pocket.
There are many things where having an API Key is necessary.
Maybe I’ve missed the boat though: is there now a method to use an api key to access a subscription?
You can't use an API key on subscriptions, but I've gotten around it using the `codex exec` command to run requests outside the CLI or GUI if you're already authenticated on that machine. Won't work for all cases, but I've never ran into a limitation in my use case of not having an API key.
> I use local LLMs on my Mac Mini.
which ones do you use?
Large enterprises pay per token even through a ChatGPT “membership”, for example.
Which is why I said "out of their own pocket"
[delayed]
Cost per token is an extremely naive way to index cost, renders this chart essentially meaningless.
As the time goes on it only becomes harder to differentiate between model capabilities with just one or two numbers. I would love to see some kind of multi-axis placement of all the models on less objective attributes, like wordiness, willingness to give up, an ability to "think ahead" and pre-solve possible problems in code, for example, that I didn't think of or didn't think of talking about, etc etc etc.
For example I've been really enjoying Deepseek v4.1 Flash, it's very "straightforward" to the point of being almost dumb sometimes, but it's absolutely relentless and would solve almost any problem no matter how inefficient the solution is.
No idea how to measure all that, just average CoT length per task is probably a good approximation for some things, but not others.
1. In real life, most of us use token packages like OpenCode Go etc.
It would be handy to have a site like this one that takes into account the various deals and attempts to calculate the number of tokens per monthly fee for a chosen model. I realize this makes the task a lot more difficult.
2. It would be handy to have a chart like that for the AI hardware that people own. It helps you decide which model to run (resulting in different levels of intelligence and speed). Also difficult to please everyone (preprocessing vs token generation for example) and to keep updated!
I found https://llm-list.com/ yesterday and when I had a detailed look, I quickly found outdated entries, for example looking at GLM 5.3 flash it listed several providers as "free" that weren't free any longer.
Updated daily and yet the frontier for math is listed as gpt-5.2
I won’t touch Meta models. I simply don’t trust or like Zuck and especially Wang
Interested to see the Coding and Math scores for Grok 4.7 when they’re available. Grok 4.6 is apparently already on the frontier for Coding.
Fun, but this seems to assume all LLM run on SAAS subscriptions. I would like to compare this to local LLM costs by converting my usage load x hardware costs into a token price. In addition if it games the LLM so often, these eventually are optimized and basically cheat on the score.
This definition of cost is not particularly useful. You want cost per task, not cost per 1m tokens. Artificial Analysis does a good job of this.
Is there a cheaper model than Gemini 3.8 Flash (High) that maybe/kind-of is on-par with it? For me it works really good but hit the limit in two hours tops... last week was the first time I hit the weekly limit and had to wait 4 days... Claude patches OK, but that is also getting drained really fast these days...
DeepSeek 4.1 Flash is your answer.
Ah OK, yeah, I was actually going to go for that earlier, I'll check it out once I get home, thanks!
is this on the $20-ish sub? I hear ultra lasts for a very long time
Yeah, 20... I was going to try Deepseek later on, just chuck 20 in there and see how good it does, and how far I can go as well...
yeah I enjoy the speed of Gemini, but I also just have the low tier one (I use the 20x Claude and Codex subs for most of my work). For iteration, Gemini is so much fun, but Opus 5.5 is quite fast, as is sol. If you're on a budget, deepseek does look great
Yeah, I mean, I think in theory I could push it to the next tier on Gemini, but at the same time I wouldn't mind trying something else, since maybe my workloads are not really that smart and I am wasting a lot of computational power on something a cheaper model with similar capabilities can do.
Coding and math graphs are very interesting. Extremely cheap models make it into the upper echelon, delivering 90% of the performance for 1% of the price compared to the #1.
I find it interesting that with the given metric comparison, for coding at min 50 strength, every frontier model brand is from a distinct vendor: Ling, Qwen, Gemini, Muse, Grok, GPT, and Claude in increasing value.
The page looks Claude generated and ranks Opus at the top.
If you maintain this over time, maybe include other sources than just AA and update the design to look less like zero-shot claude styling (I know that font! I know that color! Lol) it's genuinely useful :)
all I ever wanted is an updated website where I can see the best models I can run on my different devices locally, I don't get why people are throwing money at these companies
Guessing that most people don't have machines powerful enough to run good-enough models locally.
The coding and math tabs seem to be missing the latest models...
Math in particular is quite far behind. GPT 5.2 is recommended as the best for highest cost. Really?
Astra worse than 5.6 Sol is crazy work
It's missing Opus 5.5 which was released over a day ago (and also is clearly on the pareto frontier).
Don't know if they updated it since your comment, but for me it's on the graph – and on the Pareto frontier indeed.
It was expert timing on the commenter's and the developer's part. Opus 5.5 still doesn't show under coding though.
Opus 5.5 is indeed on the Intelligence tab, but I don't see it on the Coding or Math tabs.
For intelligence only as of 8:51 MT
I'm sure that local LLM will be far cheaper
Depends on what level of intelligence you're wanting to use. A vanishingly small number of people can or would want to go to the hardware expense of running something like GLM 5.3 Flash, much less something like K3.
And if you want Astra/Fable/Opus frontier level, then there's no option at all.
But if you don't need that, or you don't need speed... That opens up the discussion. I've been impressed even with how Siri's been doing with the Apple Foundation Models in MacOS/iOS 27 given how small they are.
Edit: I can't even fully spec the M5 Ultra Mac Studio you'd need for GLM5.3 Flash since 512GB isn't available yet, but it's already at $9500 for 256GB RAM.
> A vanishingly small number of people can or would want to go to the hardware expense of running something like GLM 5.3 Flash, much less something like K3.
It's probably worth letting the user specify their actual costs in such a tool. I run a Framework Desktop 128GB that I bought before memory prices got crazy; the current retail price is almost double what I actually paid a year ago.
Best comparison that has occurred to me is the cost a loaf of bread's ingredients might be slightly cheaper than a baked loaf, depending on how you source it. At home you get total control and know what's going in to it. Yet bake at home is still a niche, perhaps a hobby. So I say as someone who's spent hundreds of hours tinkering with local inference, go for it for anyone reading. But most people just want ... some slices of bread, you know?
Luna at the bottom of intelligence is laughable...