README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all fluff, leave only important details new users must know before using the project".
"Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all fluff, leave only important details new users must know before using the project"
and then above that the mention of hugging face is the bottleneck, not your link...
if someone completely new comes and read the current page... isn't that piece of information something they want to know?
and then the comment below about "extremely irritating" that whatever I read didn't read my mind to provide only and exactly only what I would consider great... it should be a twit that I can repost and be famous... instead I am so "extremely irritated".
what does it say about that group that gets "extremely irritated"?
Hey I spend 20 days working on this that covers something new and maybe grEat, check it out!
"AAARRGHHH I'M SO IRRITATED it has one em-dash ARRRRRRGHHHH"
I hate this AI style writing because since it doesn’t really understand flow, it’s being inserted in irrelevant places and it is extremely irritating to read.
As someone who is just looking at the theoretical benchmarks of each of these models I'm curious if anyone could share what are the problems (maybe around code) that flash-next was able to solve which 27b was not able to
yes! I'm bullish on this. there is a lot of work to do. I've been experimenting with pruning, distillation, and retraining too. I'm sure your 32gb m6 will run a badass local model!
It's hard to believe 16GB unified memory will give you 5 tok/sec unless you are ignoring the thermal warnings. I am running Qwen3.6-35B-A3B on my 16GB M3 and get 7-8 tokens/sec with all the optimizations while keeping the peak memory and thermal warnings at check. https://github.com/deepanwadhwa/samosa-chat
There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon.
At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-LM instead of producing yet another near-identical repo.
The local-LLM ecosystem really does not need every implementation idea rediscovered five times and wrapped in a new README. AI-assisted coding makes producing a new repo cheap; maintaining, benchmarking, and integrating one is the actually valuable part.
I see your point. As an oss defender myself, I agree, however, the spirit of this is to see how fast I can make it. I'm sharing this with the community, which I think is aligned with the original oss spirit.
It's an experiment for myself but I am committing to maintain it. I've been an oss person for a loooong time, way before AI was a thing. Think about it as a new, from-scratch take at it, not as a re-reproduction.
Hey carloslfu, kudos from the other side of the internet, don't get down on people nitpicking everything here, experimenting and discovering is part of learning so keep going!, remember this is the place that said dropbox was dumb and could be replaced by a script.
I'm only joking anyway — it's more a comment on the whole AI-accelerated trend of everyone having their own version of a thing.
I do agree that, ultimately, combining your efforts with others working in this whole area is probably really worth it, but I can see how there's an ease of pushing forward on your own these days.
I do not have fast internet so I am not sure when I'll really be able to download the weights but I do have an M1 Max to try this on, so I will at some point!
I see! yes, downloading the weights part is painful. I tried a couple fixes and it is as fast as it can get downloading from HuggingFace. I think the field is heading toward smaller, more capable models soon, so you won't have to wait that long!
I'm sorry this makes it seem like I didn't do my research. I did a TON. To fix it I'll add a benchmark/comparison table. Also, I wouldn't call it market research since this is not commercial AT ALL.
> Hugging Face is the bottleneck, not your link.
README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all fluff, leave only important details new users must know before using the project".
"Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all fluff, leave only important details new users must know before using the project"
and then above that the mention of hugging face is the bottleneck, not your link...
if someone completely new comes and read the current page... isn't that piece of information something they want to know?
and then the comment below about "extremely irritating" that whatever I read didn't read my mind to provide only and exactly only what I would consider great... it should be a twit that I can repost and be famous... instead I am so "extremely irritated".
what does it say about that group that gets "extremely irritated"?
Hey I spend 20 days working on this that covers something new and maybe grEat, check it out! "AAARRGHHH I'M SO IRRITATED it has one em-dash ARRRRRRGHHHH"
I hate this AI style writing because since it doesn’t really understand flow, it’s being inserted in irrelevant places and it is extremely irritating to read.
I feel you! fix incomming
thanks! I'll do!
As someone who is just looking at the theoretical benchmarks of each of these models I'm curious if anyone could share what are the problems (maybe around code) that flash-next was able to solve which 27b was not able to
Is this going to destroy my SSD?
I'm hoping to see progress in this space.
Folks talking about how 32G is not enough for local use, but then there's been work like this to empower it.
My hope is that the new 32G M6 will be "useful" locally, possibly because of work like this.
yes! I'm bullish on this. there is a lot of work to do. I've been experimenting with pruning, distillation, and retraining too. I'm sure your 32gb m6 will run a badass local model!
It's hard to believe 16GB unified memory will give you 5 tok/sec unless you are ignoring the thermal warnings. I am running Qwen3.6-35B-A3B on my 16GB M3 and get 7-8 tokens/sec with all the optimizations while keeping the peak memory and thermal warnings at check. https://github.com/deepanwadhwa/samosa-chat
how much energy does it consume?
Good one! I haven't measured this. I'll include it!
It seems we could use a new kind of memory that streams the weight data in, like GDDR in reverse.
High Bandwidth Flash? https://www.sandisk.com/company/newsroom/blogs/2025/scaling-...
interesting!
yes! I guess future hardware designs will have something like that!
"Disk is the gate that bites first"
AI;DR
The never ending gate bites.
How I have come to detest certain phrases.
Load bearing gate bites.
There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon.
At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-LM instead of producing yet another near-identical repo.
The local-LLM ecosystem really does not need every implementation idea rediscovered five times and wrapped in a new README. AI-assisted coding makes producing a new repo cheap; maintaining, benchmarking, and integrating one is the actually valuable part.
I see your point. As an oss defender myself, I agree, however, the spirit of this is to see how fast I can make it. I'm sharing this with the community, which I think is aligned with the original oss spirit.
It's an experiment for myself but I am committing to maintain it. I've been an oss person for a loooong time, way before AI was a thing. Think about it as a new, from-scratch take at it, not as a re-reproduction.
Hey carloslfu, kudos from the other side of the internet, don't get down on people nitpicking everything here, experimenting and discovering is part of learning so keep going!, remember this is the place that said dropbox was dumb and could be replaced by a script.
Why should they do that? For you? You could merge those projects and see if they get traction.
Vouched especially since OP might have a perspective on this. And readers may want to look up those other repos and compare for themselves.
Thanks for the feedback! I'll create a section with a benchmark and comparisons. This will hold the project accountable and speed things up imo
And there are `Mference` and `SwiftLM` too, I think they are doing the same use case.
AI NIH
Sorry, I don't get "NIH". what's that?
Not Invented Here.
Ah! Yeah, I didn't invent anything (yet!). The goal is to see how far I can take it in terms of speed without consuming that much RAM.
I'm only joking anyway — it's more a comment on the whole AI-accelerated trend of everyone having their own version of a thing.
I do agree that, ultimately, combining your efforts with others working in this whole area is probably really worth it, but I can see how there's an ease of pushing forward on your own these days.
I do not have fast internet so I am not sure when I'll really be able to download the weights but I do have an M1 Max to try this on, so I will at some point!
I see! yes, downloading the weights part is painful. I tried a couple fixes and it is as fast as it can get downloading from HuggingFace. I think the field is heading toward smaller, more capable models soon, so you won't have to wait that long!
> every implementation idea rediscovered five times and wrapped in a new README
That's open source since forever, unfortunately.
I agree with the sentiment, but have you seen those videos in which all men say other men are gay? This feels like the same, so much AI paranoia!
I genuinely want to contribute. And hey! I was doing oss this since 2014 so waay before AI was cool.
It's what happens when you don't do market research.
I'm sorry this makes it seem like I didn't do my research. I did a TON. To fix it I'll add a benchmark/comparison table. Also, I wouldn't call it market research since this is not commercial AT ALL.
Hey Carlos, thanks for sharing with the community! Appreciated
thanks!
Half the people on here are using Ollama. No one is doing market research.
Does a painter check to make sure that a portrait hasn't been painted? What a dismissive comment.