>"Features of matrix multipliers differ across vendors and architectures of the same vendor [...] As a result, reproducibility of small matrix multiplier results [differences] across devices is not possible and cannot be achieved by software control. Implementation details of matrix multipliers are not documented, making it difficult to interpret discrepancies in the computed results."
I'm guessing (but not knowing) that small subtle differences in matrix multiply across different vendor's architectures (and product generations of an individual vendor's architecture) is responsible for a good portion of software crashes when trying to run a local LLM on a different architecture or with a different stack (ROCm vs. CUDA, for example) than the ones it has been explicitly tested on.
As such, this marks a rather significant problem for the future, which can basically be stated as:
There needs to be a standard matrix multiply specification (much like IEEE-754 is/was for floating point operations) that all future vendors of AI accelerators (any GPU, CPU, NPU or IC manufacturer whose circuits implement matmul) adhere to, such that the matmul of one vendor is exactly and precisely compatible with the matmul of another.
Hardware vendors of course, are free to compete in terms of speed, power efficiency, number of matmul engines on a given piece of silicon, parallelization optimizations, etc., but the basic matmul operation should be exactly and precisely compatible across vendors and across future product versions.
Step 1: We need a spec for this... (Maybe IEEE is already working on one? If so, that's a good step forward!)
Step 2: Hardware vendors need to implement it, to be universally compatible in all of their IC's that use matmul, in the future...
For instance, when I created script to train my model it worked fine on RTX 5080, but when I rented H100 to hopefully wait less for completion of training, the training would collapse just in a few epochs, suggesting they compute things differently (RTX 5080 would run thousands of epochs without collapsing. The same script and the same data).
>"Features of matrix multipliers differ across vendors and architectures of the same vendor [...] As a result, reproducibility of small matrix multiplier results [differences] across devices is not possible and cannot be achieved by software control. Implementation details of matrix multipliers are not documented, making it difficult to interpret discrepancies in the computed results."
I'm guessing (but not knowing) that small subtle differences in matrix multiply across different vendor's architectures (and product generations of an individual vendor's architecture) is responsible for a good portion of software crashes when trying to run a local LLM on a different architecture or with a different stack (ROCm vs. CUDA, for example) than the ones it has been explicitly tested on.
As such, this marks a rather significant problem for the future, which can basically be stated as:
There needs to be a standard matrix multiply specification (much like IEEE-754 is/was for floating point operations) that all future vendors of AI accelerators (any GPU, CPU, NPU or IC manufacturer whose circuits implement matmul) adhere to, such that the matmul of one vendor is exactly and precisely compatible with the matmul of another.
Hardware vendors of course, are free to compete in terms of speed, power efficiency, number of matmul engines on a given piece of silicon, parallelization optimizations, etc., but the basic matmul operation should be exactly and precisely compatible across vendors and across future product versions.
Step 1: We need a spec for this... (Maybe IEEE is already working on one? If so, that's a good step forward!)
Step 2: Hardware vendors need to implement it, to be universally compatible in all of their IC's that use matmul, in the future...
> is responsible for a good portion of software crashes
I doubt it's the case, it should result in slightly different logits and generated tokens, but it shouldn't lead to crashes.
I suspect what you're facing are simply driver bugs…
Isn't the same amongst same architectures?
For instance, when I created script to train my model it worked fine on RTX 5080, but when I rented H100 to hopefully wait less for completion of training, the training would collapse just in a few epochs, suggesting they compute things differently (RTX 5080 would run thousands of epochs without collapsing. The same script and the same data).