To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175
Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.
This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.
To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175 Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
At which quantization? That's the only thing I want to know.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
Everything lower than 4/5 is really bad
Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?
probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.
haven't tried it yet though.
This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.
It is 2 bit? anybody tested this? is any good?