For 4bit, it takes you 5-10 minutes to compress a model. With this, you can do that in seconds. You do not have to worry about tokenizer configuration or anything. Just one-click compress and run.
This is our first version and yes, it will have some errors that will be fixed in the upcoming versions.
Concretely, if you start with, say, a 1.5 or 4 bit quant, then run this, you get something the size of a 0.5 - 1.5 bits without further degradation in model behavior?
This reads like it. Given that consumer hardware is essentially capped at a certain amount (lets say 24GB for coupled mid range GPUs), this would at best triple either the ‚intelligence‘ (meaning non-degradation of heavy quantization) or the speed of locally running LLMs.
There has to be a catch, this sounds too good to be true.
No, you require BF16 model weights for this quantization to work. If you do run it on 4bit or 1.5bit, it will degrade the model and it will hallucinate a lot.
What is the point of using a outright ancient model like DeepSeek-R1-Distill-1.5B from 2025 based on another ancient model like Qwen2.5-Math-1.5B from 2024, when a much better modern model like LFM3-1.7B exists?
This is a well known tell for using (bad) models to do "research". They always use qwen2.5 and R1, because of the amount of "news" that the two models received. So the models "suggest" those lineages a lot. The local llama sub used to get a few of these a day. That and a zenodo "paper" link are clear tells of badly written stuff.
Although with Anthropic purposefully sandbagging their models for AI R&D, how much of it comes down to model failure, and how much of it comes to deliberate misalignment of the model maker and the customers they sell to?
We have tested it on all kinds of small models including LFM3-1.7B. Our goal is to make a a system that can compress any AI. This is our first version and with your feedback we can improve.
omg, this is the perfect name for this. Much better than "colored left-border."
It's very curious that the models are so drawn to a fingernail (left border callout) when they must be sparse in the training data (human designers don't do this a ton).
Maybe it will be better if you focused more at the research rather than doing such things. Our website it not optimised for mobile users as it has more explanation to do rather than just make visitors feel good. Good luck with the things you are going.
> You’re right, I didn’t actually implement any compression here! My response was misleading, I shouldn’t have claimed this was possible.
Sadly that’s just got to be assumption that there’s no human in the loop. Flashy UI, developed, a readme are no longer indicators you should believe in a project.
I have not read the whole thing, but I'm not sure I understand the benefit of this vs. using an existing e.g. 4 bit quant of the model in question
For 4bit, it takes you 5-10 minutes to compress a model. With this, you can do that in seconds. You do not have to worry about tokenizer configuration or anything. Just one-click compress and run.
This is our first version and yes, it will have some errors that will be fixed in the upcoming versions.
Does this compose well with quantization?
Concretely, if you start with, say, a 1.5 or 4 bit quant, then run this, you get something the size of a 0.5 - 1.5 bits without further degradation in model behavior?
This reads like it. Given that consumer hardware is essentially capped at a certain amount (lets say 24GB for coupled mid range GPUs), this would at best triple either the ‚intelligence‘ (meaning non-degradation of heavy quantization) or the speed of locally running LLMs. There has to be a catch, this sounds too good to be true.
No, you require BF16 model weights for this quantization to work. If you do run it on 4bit or 1.5bit, it will degrade the model and it will hallucinate a lot.
So this is very good for making local models more accessible, right?
Correct, you will be able to compress, load and talk with any AI models. This algorithm compressed all model weights by upto 3x.
Is it possible to be used on bigger models with the same gains?
Yes, you can use it on trillion parameter class AI models.
Validated how?
If you couldn’t tell, this is LLM generated quackery.
Any claims about throwing away 2/3rds of the model weights (not just quantizing them) and maintaining “near lossless” quality are simply false.
I clicked on the benchmarks page expecting some quality benchmarks but only saw speed benchmarks before I gave up and stopped scrolling.
The claims don’t even make sense. It’s showing “100% GPU resident” for a model of a size that cannot fit into the GPU.
Yeah sure I could also quantize Qwen 3.8 27B to 0.15 bits/param. Not sure what the point of this exercise is.
What is the point of using a outright ancient model like DeepSeek-R1-Distill-1.5B from 2025 based on another ancient model like Qwen2.5-Math-1.5B from 2024, when a much better modern model like LFM3-1.7B exists?
This is a well known tell for using (bad) models to do "research". They always use qwen2.5 and R1, because of the amount of "news" that the two models received. So the models "suggest" those lineages a lot. The local llama sub used to get a few of these a day. That and a zenodo "paper" link are clear tells of badly written stuff.
Although with Anthropic purposefully sandbagging their models for AI R&D, how much of it comes down to model failure, and how much of it comes to deliberate misalignment of the model maker and the customers they sell to?
We have tested it on all kinds of small models including LFM3-1.7B. Our goal is to make a a system that can compress any AI. This is our first version and with your feedback we can improve.
I compress everything using the Sloot Digital Coding System, only use 50k for all my data!
No idea if the paper has merit but the UI is slop:
- gradient headline
- random vertical scrollbar on mobile
- fingernail cards
> - fingernail cards
I did not know that this was the name for these but I am glad I now do :-)
omg, this is the perfect name for this. Much better than "colored left-border."
It's very curious that the models are so drawn to a fingernail (left border callout) when they must be sparse in the training data (human designers don't do this a ton).
Really, started looking at the page and the slop trigger hit quickly enough for me to just skim the structure of the page and close it.
Maybe it will be better if you focused more at the research rather than doing such things. Our website it not optimised for mobile users as it has more explanation to do rather than just make visitors feel good. Good luck with the things you are going.
It's a bad smell. As a reader, if the first thing you see is slop, you risk that the rest is also slop and you're wasting your attention.
If UI is not your area or priority, that's ok. Why not just post an Arxiv link?
I know a gen AI UI when I see one and this has all of the red flags. Hate to say it but I feel like that compromises the legitimacy of this
> You’re right, I didn’t actually implement any compression here! My response was misleading, I shouldn’t have claimed this was possible.
Sadly that’s just got to be assumption that there’s no human in the loop. Flashy UI, developed, a readme are no longer indicators you should believe in a project.
The website was made with Gemini. Not a big html coder. I was a bit busy on doing research on things that matter rather than focusing on UI
If you click through to the home page of the site you are greeting with the perfect genAI tagline:
“We didn't just build an AI voice bot. We engineered a full-stack autonomous AI call center platform with quick call center setup in 3 minutes.”