Please how to select the model?
I downloaded tinyLlama, put it in ./models, changed reame.conf but I get:
(No such file or directory)
Otherwise putting the model in /opt does not please me much, I fear to forget a model is there, if it is in reame folder its much easier to notice and manage.
Thanks for sharing your work!
Please how to select the model? I downloaded tinyLlama, put it in ./models, changed reame.conf but I get:
(No such file or directory)
Otherwise putting the model in /opt does not please me much, I fear to forget a model is there, if it is in reame folder its much easier to notice and manage.
If a prefix cache is good, why not keep it in primary mem?
The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.
Why qwen 2.5 everywhere? Why not 3.5?