Could you say more about how caching works? One major advantage of sticking with a single model is saving money on cached input tokens. I'd imagine if you swap between a bunch of models, you may improve performance but cost would would balloon out of control
The trick is to rarely switch, or switch at task boundaries. Often the conclusion of routing is actually "this one model is actually at the pareto front for this task, just use it always".
I have not tried it yet. Is it similar to LiteLLM? If so, what sets it apart?
Router and model optimization from traffic is the main differentiator
Could you say more about how caching works? One major advantage of sticking with a single model is saving money on cached input tokens. I'd imagine if you swap between a bunch of models, you may improve performance but cost would would balloon out of control
The trick is to rarely switch, or switch at task boundaries. Often the conclusion of routing is actually "this one model is actually at the pareto front for this task, just use it always".
and caching is related to performance too ofc
Super interesting and congrats on the release. Curious if you initially had this in Python and then rewrote in Rust?
Yep! If you look at the commit history that's exactly what happened.
Finally an open source tool doing this!