Ah, this prompted a few questions from me. Perhaps someone with more hardware background could chime in.
They don't really show it here, but supposedly a Cerebras' chip is sub-divided into a 2D mesh of what they call PEs (cores). So data flow generally flows in two axis, and maintains consistent timings to its 4 neighboring cores. They show the layers flowing from left-to-right being streamed by PEs, but I wonder how true that ends up being: maybe the communications are easier to stream via diagonals, or some other shape that has better topology for data flow.
Then, even from die shots of CPUs and GPUs we often see that cores are tiled as squares, and those don't have the same cross-core connection and streaming requirement. Now, I assume that some parts of what we call a core can be moved around more freely while maintaining whatever power/heating/timing requirements we have, so perhaps it's easier to 'force' a square by moving those loose parts around. But why do we confine those to squares? Is super-core layout just something not worth investing time into, like how interconnects are timed, non-convex tilings, and distances to I/O?
And for streaming architectures like Cerebras, why aren't other types of tilings considered, something like a hexagon grid? This could have 3 inputs / 3 outputs, which could allow more complex data flow (?). Similarly, we could have 8 connections with longer diagonals in a square tiling, etc.
I guess, what are the tradeoffs in place such that so many chip designs seem to reuse square tilings both locally and globally?
> The total number of concurrent requests represents the number of simultaneous requests moving through the pipeline at any point in time. Unlike a GPU batch, those requests don’t wait for one another to finish, they stream through the pipeline continuously.
What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right
Extreme speeds are actually interesting if you think about it. We are apparently getting some interesting results from LLMs nowadays. If they would put 100x effort in 10% the time it takes, or imagine 1% where how long tool calls take become the bottleneck, that would actually probably feel like a really special kind of "super intelligence".
This is why I think there will be selective pressure for fast tools. It’s like an evolutionary process and fast tools will be preferred and created. It’s already happening and also why I think computer use is a dead end.
At 20 kW average consumption per WSE-3T wafer this gives roughly 1.18 MW for the whole system. Assuming 1500 tokens per second and 190 concurrent requests this gives 4.1 J/token. Which is good I suppose for the size of the model?
For a fairly weak comparison, modern Bitcoin mining chips are around 13J/TH, so around 0.32TH per token, or 320 billion double-sha256 mining operations per token served by this system. Peanuts are also about 4200 Joules, or almost exactly 1000 tokens per nut assuming you could efficiently turn it into electricity to run this chip. Just as a frame of reference, I don't think this actually serves to make any point.
Ah, this prompted a few questions from me. Perhaps someone with more hardware background could chime in.
They don't really show it here, but supposedly a Cerebras' chip is sub-divided into a 2D mesh of what they call PEs (cores). So data flow generally flows in two axis, and maintains consistent timings to its 4 neighboring cores. They show the layers flowing from left-to-right being streamed by PEs, but I wonder how true that ends up being: maybe the communications are easier to stream via diagonals, or some other shape that has better topology for data flow.
Then, even from die shots of CPUs and GPUs we often see that cores are tiled as squares, and those don't have the same cross-core connection and streaming requirement. Now, I assume that some parts of what we call a core can be moved around more freely while maintaining whatever power/heating/timing requirements we have, so perhaps it's easier to 'force' a square by moving those loose parts around. But why do we confine those to squares? Is super-core layout just something not worth investing time into, like how interconnects are timed, non-convex tilings, and distances to I/O?
And for streaming architectures like Cerebras, why aren't other types of tilings considered, something like a hexagon grid? This could have 3 inputs / 3 outputs, which could allow more complex data flow (?). Similarly, we could have 8 connections with longer diagonals in a square tiling, etc.
I guess, what are the tradeoffs in place such that so many chip designs seem to reuse square tilings both locally and globally?
A few years old now, but their co-founder whiteboards how their chips work https://www.youtube.com/watch?v=7GV_OdqzmIU
> The total number of concurrent requests represents the number of simultaneous requests moving through the pipeline at any point in time. Unlike a GPU batch, those requests don’t wait for one another to finish, they stream through the pipeline continuously.
What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right
Ok I'm sold, where to buy? :D
If you made the mistake of buying their stock the past few weeks, you might wish you hadn't (or maybe it's an opportunity to buy low yada yada)
Extreme speeds are actually interesting if you think about it. We are apparently getting some interesting results from LLMs nowadays. If they would put 100x effort in 10% the time it takes, or imagine 1% where how long tool calls take become the bottleneck, that would actually probably feel like a really special kind of "super intelligence".
This is why I think there will be selective pressure for fast tools. It’s like an evolutionary process and fast tools will be preferred and created. It’s already happening and also why I think computer use is a dead end.
At 20 kW average consumption per WSE-3T wafer this gives roughly 1.18 MW for the whole system. Assuming 1500 tokens per second and 190 concurrent requests this gives 4.1 J/token. Which is good I suppose for the size of the model?
For a fairly weak comparison, modern Bitcoin mining chips are around 13J/TH, so around 0.32TH per token, or 320 billion double-sha256 mining operations per token served by this system. Peanuts are also about 4200 Joules, or almost exactly 1000 tokens per nut assuming you could efficiently turn it into electricity to run this chip. Just as a frame of reference, I don't think this actually serves to make any point.