Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather good (to the point that I use it mostly nowadays)
On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude.
These token numbers look off for a 5090. For comparison, an rtx 5000 Blackwell SFF with just 470gb/s bandwidth gets me 30 tok/s on gemma4-31B-QAT. Almost 60 tok/s with MTP enabled. A 5090 should get you much more than that!
That Gemma release is what opened my eyes. Even downloading the models on my iPhone was shocking. It reminded me when folks would put Wikipedia on a CD or something similar.
Obviously does not compare to a leading model but it’s impressive for something that was running on my phone. I could see thinking token output and it’s directionally interesting thought.
gemma 31b is my goto day-to-day model. It's adequately capable for coding and natural phenomena, and it follows instructions and skills reasonably well. My main negative trait is all the LLM-isms, but I can get past that for the value it brings
I know putting actual prices regionally localizes your article and temporally, with how prices are so unstable. But analysis of “what to buy” without actual prices is borderline meaningless.
Overall, good article, very interesting to see a real deployment that’s actually attainable and not just a subscription to a big 3 token plan.
This seems like a temporary situation. Utilization maximization is a matter of switching to the software factory unattended flow and intelligently routing tasks that will not require assistance to the unused time.
A decade and a half ago we used to run massive map reduce jobs overnight. Code will be handled like this.
I would love to see such comparisons but with quantized versions, because quantization allows running models on smaller hardware with some quality loss. I am running Qwen3.6-35B-A3B quantized to int4 on an A6000 card that was otherwise just sitting around idle. It works up to a degree, but I would love to see benchmarks comparing different quantizations of several models, especially in quality. This is an important dimension in deciding whether to buy a GPU is worth it, and it is missing from this (otherwise comprehensive) article.
I could not focus on the article with all the noise in the background. It was annoying in the header but then it continued down the page. If you want to do this on your marketing pages have at it but for a blog/news style page? Reader mode was the only way to restore sanity.
Author here(I'm on the team). We updated this post after Monday's Kimi K3 weights release: fitting the 2.8T model means going from 8×B200 to 8×B300, ~20% more hardware cost, and concurrency drops from 24 to 16 users vs GLM-5.2. Caveat we're upfront about in the post: our 64-task SWEBench Pro subset may be in Kimi's training set, so the 86% resolve rate is an upper bound.
Let us know your thoughts, we really value feedback!
Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.
I second this. Anyone considering running their own hardware would also consider quantization since they would have full control over what quant to run.
The value proposition changes a lot if you can get 90% of the quality for 50% the price with a quant due to halving your hardware requirement.
Thank you for the article, though! Very informative.
Recently I started playing with LM Studio and local models and found out that `gemma-4-26b-a4b` is suprisingly capable. I don't need elaborate akin to "create complete app to do X" or "refactor the whole codebase of bazzilion of LOC" but rather "how to go about doing thing X" or for language study (explaining nuances of phrasal verbs or subtelties of vocabulary in other languagues) and darn -- the results are rather good (to the point that I use it mostly nowadays)
On my RTX 5090, Qwen3.6-35B-A3B is my go to for coding, but Gemma-4-26b-a4b (Q4_0) is blazing fast and super high quality. I sometimes use the dense cousins of these models (Qwen3.6-27B QB_0 and Gemma4-31B-QAT Q4_0) but they can be very slow. Integrating web search and code calling via Open-WebUI makes it so that I usually don't need to reach for Claude.
Here are the tok/s I get:
- Gemma-4-26B-A4B (Q4_0) = 208 tok/s
- Qwen3.6-35B-A3B (QB_0) = 30 tok/s
- Qwen3.6-27B (QB_0) = 8 tok/s
- Gemma4-31B-QAT (Q4_0) = 8 tok/s
These token numbers look off for a 5090. For comparison, an rtx 5000 Blackwell SFF with just 470gb/s bandwidth gets me 30 tok/s on gemma4-31B-QAT. Almost 60 tok/s with MTP enabled. A 5090 should get you much more than that!
That Gemma release is what opened my eyes. Even downloading the models on my iPhone was shocking. It reminded me when folks would put Wikipedia on a CD or something similar.
Obviously does not compare to a leading model but it’s impressive for something that was running on my phone. I could see thinking token output and it’s directionally interesting thought.
gemma 31b is my goto day-to-day model. It's adequately capable for coding and natural phenomena, and it follows instructions and skills reasonably well. My main negative trait is all the LLM-isms, but I can get past that for the value it brings
>which box to buy
…
> spark, costs less than a conference trip.
I know putting actual prices regionally localizes your article and temporally, with how prices are so unstable. But analysis of “what to buy” without actual prices is borderline meaningless.
Overall, good article, very interesting to see a real deployment that’s actually attainable and not just a subscription to a big 3 token plan.
This seems like a temporary situation. Utilization maximization is a matter of switching to the software factory unattended flow and intelligently routing tasks that will not require assistance to the unused time.
A decade and a half ago we used to run massive map reduce jobs overnight. Code will be handled like this.
I would love to see such comparisons but with quantized versions, because quantization allows running models on smaller hardware with some quality loss. I am running Qwen3.6-35B-A3B quantized to int4 on an A6000 card that was otherwise just sitting around idle. It works up to a degree, but I would love to see benchmarks comparing different quantizations of several models, especially in quality. This is an important dimension in deciding whether to buy a GPU is worth it, and it is missing from this (otherwise comprehensive) article.
I could not focus on the article with all the noise in the background. It was annoying in the header but then it continued down the page. If you want to do this on your marketing pages have at it but for a blog/news style page? Reader mode was the only way to restore sanity.
I found it trivial to consume. It should support reduced motion preference though for the more feeble among us.
> It should support reduced motion preference though for the more feeble among us.
What a strong start to a sentence before veering into a pretty gross equivalency.
Same here. Stopped reading.
Author here(I'm on the team). We updated this post after Monday's Kimi K3 weights release: fitting the 2.8T model means going from 8×B200 to 8×B300, ~20% more hardware cost, and concurrency drops from 24 to 16 users vs GLM-5.2. Caveat we're upfront about in the post: our 64-task SWEBench Pro subset may be in Kimi's training set, so the 86% resolve rate is an upper bound.
Let us know your thoughts, we really value feedback!
Like I said above, you should benchmark quantized versions. With quantization, the same models can be ran at much cheaper hardware, but quality loss is real and this kind of benchmark is an ideal place to put a finger on it.
I second this. Anyone considering running their own hardware would also consider quantization since they would have full control over what quant to run.
The value proposition changes a lot if you can get 90% of the quality for 50% the price with a quant due to halving your hardware requirement.
Thank you for the article, though! Very informative.