Impressive debugging skills, and thank you for the benchmarks. Now I'm wondering if mlx-engine / mlx-lm have these bugs too.
One minor thing: as you are concerned with honest numbers, the graphs should be logarithmic on the y-axis too (like they are on the x-axis). Otherwise it's hard to see whether the curve is sublinear or linear.
I appreciate the amount of detail in the post, I think it's a useful addition to the space.
That said, I have to read LLM output all day all the time, and I would implore you to take the time to explore your own voice a bit more.
> Two separate things then happened, and it is worth keeping them apart.
Is one of those phrases claude spits out nonstop.
Impressive debugging skills, and thank you for the benchmarks. Now I'm wondering if mlx-engine / mlx-lm have these bugs too.
One minor thing: as you are concerned with honest numbers, the graphs should be logarithmic on the y-axis too (like they are on the x-axis). Otherwise it's hard to see whether the curve is sublinear or linear.