Perf goes from 80% to 47% on Wikitext-2. Also no comparisons to FP4 solutions that are able to maintain or exceed perf on the same dataset 80% perf with a 4.25-4.5 big budget.
I think more meaningful thing here would be a hybrid solution that went down to sub-bit representations when the informational representation does not need it (for example later layers) that still maintains task performance
Perf goes from 80% to 47% on Wikitext-2. Also no comparisons to FP4 solutions that are able to maintain or exceed perf on the same dataset 80% perf with a 4.25-4.5 big budget.
I think more meaningful thing here would be a hybrid solution that went down to sub-bit representations when the informational representation does not need it (for example later layers) that still maintains task performance
Can this produce a useful model? So far 1 bit quants have been less useful than smaller models that use the same memory
I recently found this 1.58-bit model for ASR and it's surprising good (and very fast), that being said it's not a LLM.
https://huggingface.co/moondream/parakeet-redux
Thought this was going to be on the original Little Bit paper, always nice to find out about a surprise sequel!
Their paper shows this comes with huge quality loss, but that doesn't make it a negative result by any means
Has anyone tried this on apple silicon M1-5? Any benchmarks/comps?