This is good timing. My setup at home involves a two servers with 5080s each and an Rtx spark laptop with 128gb. I plan I have qwen flash next on the latter and the 27b variant on the former. I would’ve preferred to have purchased a dgx spark but everything is just ridiculously expensive now. My own setup would be 3x the cost to buy again.
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that. My secret is that I have this relatively unknown technique for solving problems where I use this quantum engine that developed from just atoms over 4.54 billion years into the most sophisticated machine in the known universe. It's a frontier model that every human has access too, actually.
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
Yep, fully satisfied with DS 4.1 Flash or GLM 5.3 Flash. I prefer using one chat at a time, actually thinking through things with it, as opposed to the YOLO run all night multi-agent thing. I mean I’m still okay with subagents if my main thread needs to explore a few options and then report back, but all on the same idea.
Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.
I was able to remove much of the CCP censorship from Qwen 3.8 27B for around $90 using a LoRA. I multiplied the LoRA in permanently and listed it on HuggingFace.
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
Yes, it'll likely just result in some people standing up a more dedicated purpose community hosting site of abliterated/uncensored open weight GGUF files. In a theoretical scenario where HF bans abliterated/uncensored models I could very well see the same people who are producing those now simply moving the location they upload stuff to.
The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I guess it depends if we're talking about decent new car at $40,000 or $95,000? I've seen lots of people claiming they're running v4.1 flash 1017 on local hardware but then it turns out it's something like the unsloth Q2 XXS or similar size. The Q8 is 508GB so assuming for a moment we're not running it on a Mac, we're looking at a cost of five or six 96GB GPUs to load the whole thing with a good sized full context and KV cache.
4x 96 GB runs the NVFP4 quants, which are indistinguishable from Q8 in real-world performance for this particular model. That's not always true but DeepSeek is unusually quantization-friendly.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
Dumber yes, but not by all that much. If the multiple agent thing really improves the quality by as much as is described here, the quality loss should be nearly imperceptible.
Not sure if I agree. Remember the error rates stack up every turn of the conversation. If a 2bit model is 10% more wrong than the fp16 version, over 10 turns that adds up
Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.
Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet...
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.
I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated
So far, the comments have focused on hopes for the effectiveness of open-weight models versus proprietary frontier models. That's fine, but I'm thinking about the approach and the results, as opposed to the model.
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.
Fable 5 is not anymore frontier and even if, I hardly believe five very small 27b models can do like Fable 5. I think this is whishful thinking more than a fact
I'd like to see this done with qwen 3.8-flash-next (Q8 or full precision) which beats 3.8-27B on just about every coding/command line use benchmark. In particular comparisons like terminalbench 4.0.
will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx
This is good timing. My setup at home involves a two servers with 5080s each and an Rtx spark laptop with 128gb. I plan I have qwen flash next on the latter and the 27b variant on the former. I would’ve preferred to have purchased a dgx spark but everything is just ridiculously expensive now. My own setup would be 3x the cost to buy again.
I’m mainly trying to replicate Gemini flash and muse agent for my family of 4 without breaking the bank (already had a 5080 for gaming and the other for image gen).
It is unclear if this memdoor works for agentic tasks, e.g.: “check daily if dune 3 tickets are for sale and if they are buy tickets for 4 where all 4 seats are next to each other in the same row and as center as possible.”
There is a huge cliff once you go past 128gb. Basically you need either Rtx 6000 pros (20k+) or a cluster of dgx sparks (15k+)
We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
>The nice thing is that current local modes don't go away.
Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.
...so we really do need open models to keep pace, not just be statically competitive at a given point in time
Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that. My secret is that I have this relatively unknown technique for solving problems where I use this quantum engine that developed from just atoms over 4.54 billion years into the most sophisticated machine in the known universe. It's a frontier model that every human has access too, actually.
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
Yep, fully satisfied with DS 4.1 Flash or GLM 5.3 Flash. I prefer using one chat at a time, actually thinking through things with it, as opposed to the YOLO run all night multi-agent thing. I mean I’m still okay with subagents if my main thread needs to explore a few options and then report back, but all on the same idea.
How do you run DS 4.1 Flash if your computer isnt powerful and everything?
The fact that you point to the latest model as „I’d be fine with that“ proves my point.
If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.
It’s a moving target thus we need open weights to move
Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017 or mimo 2.6 or equivalent in size locally.
>everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future.
Unlikely given how trivial it is to DIY this (I think...haven't myself but from afar it seems within avg hn crowd reach)
I was able to remove much of the CCP censorship from Qwen 3.8 27B for around $90 using a LoRA. I multiplied the LoRA in permanently and listed it on HuggingFace.
The cost was approximately $80 for generating synthetic data and $10 to create and integrate the LoRA.
Yes, it'll likely just result in some people standing up a more dedicated purpose community hosting site of abliterated/uncensored open weight GGUF files. In a theoretical scenario where HF bans abliterated/uncensored models I could very well see the same people who are producing those now simply moving the location they upload stuff to.
To use an old analogy, it's not rocket science:
https://github.com/p-e-w/heretic
The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash 1017
Not as bad as you might think. This costs about as much as a decent new car nowadays.
The abliterated 1017 release I've seen is crazy good.
I guess it depends if we're talking about decent new car at $40,000 or $95,000? I've seen lots of people claiming they're running v4.1 flash 1017 on local hardware but then it turns out it's something like the unsloth Q2 XXS or similar size. The Q8 is 508GB so assuming for a moment we're not running it on a Mac, we're looking at a cost of five or six 96GB GPUs to load the whole thing with a good sized full context and KV cache.
4x 96 GB runs the NVFP4 quants, which are indistinguishable from Q8 in real-world performance for this particular model. That's not always true but DeepSeek is unusually quantization-friendly.
I imagine the various GGUFs are still pretty impressive, but haven't tried them myself.
> new car
...which is still tooo much for most people to spend on "AI hardware"....
More like, the ability to chain a demon in your basement that does what you tell it and can't be rug-pulled by his bosses in San Francisco.
That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.
Although the same harnessing should also improve Fable's performance. The important thing here is the cap for local performance looks like it could be quite high.
[0] https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... suggests 16GB VRAM, 32 GB system.
The bonsai quants are not that awesome. Yeah they're fast and they retain the "27B" but in practice the perform similarly
5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
Dumber yes, but not by all that much. If the multiple agent thing really improves the quality by as much as is described here, the quality loss should be nearly imperceptible.
Not sure if I agree. Remember the error rates stack up every turn of the conversation. If a 2bit model is 10% more wrong than the fp16 version, over 10 turns that adds up
I think that requires testing, rather than assuming. There's only so far a harness can go.
Bonsai 2 has a bit different architecture as far I understand and speculative decoding is not so great yet. Before that point, original model is faster if you have the memory. Would be nice to see the speed with proper speculative decoder.
Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in every single computer in this planet...
Happy hallowen Tax Ramltman and Amowhateverwayyournameiswritten...
Happy hallowen! ::::
Actually we tried Cerebras. It caps output at 32K tokens so the 128K reasoning (the author budget) didn't fit. Cerebras speed is real though
GVS5H, which this is based on, was discussed here a week ago: https://news.ycombinator.com/item?id=50228167
Here the intent is to replicate GVS5H with memdoor a headless coding agent I created months ago. Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.
I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated
So far, the comments have focused on hopes for the effectiveness of open-weight models versus proprietary frontier models. That's fine, but I'm thinking about the approach and the results, as opposed to the model.
Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the approach should work regardless of the model, open-weight or otherwise.
My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members. Also, since the division limits the scope of each team member's focus, the context a team member has is used more effectively.
Fable 5 is not anymore frontier and even if, I hardly believe five very small 27b models can do like Fable 5. I think this is whishful thinking more than a fact
"think", "believe" - maybe, but this is why we need better measurability.
This also appeared on HN last week: https://news.ycombinator.com/item?id=50228167, it's the other method they mention, GVS5H
I'd like to see this done with qwen 3.8-flash-next (Q8 or full precision) which beats 3.8-27B on just about every coding/command line use benchmark. In particular comparisons like terminalbench 4.0.
https://artificialanalysis.ai/models/comparisons/qwen3-8-fla...
will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx
I can't wait for qwen4 flash next or something like a 150b moe from qwen.
can it bet a team of 5 fable 5's?