44 points | by xenova an hour ago ago
8 comments
The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.
The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models
That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?
TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
There's two variants of this (or, as the joke goes, for very big values of bit):
Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.
1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.
This must be some sort of unpublished app?
I can just see their image tool on the app store
The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.
The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models
That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?
TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
There's two variants of this (or, as the joke goes, for very big values of bit):
Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.
1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.
This must be some sort of unpublished app?
I can just see their image tool on the app store