I love local AI, but it just can't compete with frontier models (unless your "local" is something similar to a home datacenter).
"Let's assume the bulk of your one-hour session was with an average of 32 background agents doing different tasks, at an average of 20k tok/s input"
Then you do need 32 parallel sequences for your llamacpp (or similar), 20k context (which can be a bit small for a coding agent) * 32 parallel = 640,000. That is a big KV cache, even if quantized.
And what about the model itself? Local ones are impressive, but the frontier ones are a different game. (Again, unless you can run a very big model in your local setup).
Local AI is great for vertical tasks, NLP, and a lot of "simple" cases.
Swarming it consumes the memory (parallel config).
Swarm it with coding agents... I just don't see it working well.
I love local AI, but it just can't compete with frontier models (unless your "local" is something similar to a home datacenter).
"Let's assume the bulk of your one-hour session was with an average of 32 background agents doing different tasks, at an average of 20k tok/s input"
Then you do need 32 parallel sequences for your llamacpp (or similar), 20k context (which can be a bit small for a coding agent) * 32 parallel = 640,000. That is a big KV cache, even if quantized.
And what about the model itself? Local ones are impressive, but the frontier ones are a different game. (Again, unless you can run a very big model in your local setup).
Local AI is great for vertical tasks, NLP, and a lot of "simple" cases. Swarming it consumes the memory (parallel config). Swarm it with coding agents... I just don't see it working well.