I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks.
Are we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
This was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN.
The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others.
I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.
That sounds cool, but it's still pretty meaningless without information about what your home lab looks like. A few DGX Sparks wired up with their fancy super fast network is much different than a few laptops on wifi.
Perf should be fairly straightforward to ballpark. You'll need to transfer roughly 2 . hidden_size . num_shards bytes over the network per token during autoregressive decoding. And divide that number by chunk size during prefill.
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
Is it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).
Good point. I know cocompute is working on splitting, but it’s not there yet; I was referring to the round-robin delegation within a trusted pool. Mesh LLM looks great too!
I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks.
Are we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
This was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN.
The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others.
I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.
That sounds cool, but it's still pretty meaningless without information about what your home lab looks like. A few DGX Sparks wired up with their fancy super fast network is much different than a few laptops on wifi.
What hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?
Perf should be fairly straightforward to ballpark. You'll need to transfer roughly 2 . hidden_size . num_shards bytes over the network per token during autoregressive decoding. And divide that number by chunk size during prefill.
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
Does Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?
Just wondering, why do you care about encryption in this context?
I'm not affiliated, but yes – the main 'point' of iroh is that it's 'dial-a-key', QUIC with encryption based on the keys of the endpoints.
I thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.
That isn’t true. llama RPC is incredibly slow but staged splits in skippy are orders of magnitude faster.
I just wish I had the hardware to try it out!
It sounds like iroh enables distributed compute without having to finangle custom hardware.
cocompute.ai is already doing this really well.
Is it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).
Good point. I know cocompute is working on splitting, but it’s not there yet; I was referring to the round-robin delegation within a trusted pool. Mesh LLM looks great too!
Cool, always good to have more in the ecosystem. I love Iroh and hope this continues to succeed.