Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
1–10 of 60 posts
Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
#2People have been noticing the effects of this in local LLM inference. Power limiting seems to improve overall performance!
Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
#3People have been noticing the effects of this in local LLM inference. Power limiting seems to improve overall performance!
In general, constraints require optimizations and rearchitectures. I'd also expect the ram shortage for instance to have a big impact on the software industry as a whole, specially in games. They will need to make do with what people have, a ps5/pro or similar in PC power.
Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
#4People have been noticing the effects of this in local LLM inference. Power limiting seems to improve overall performance!
In general, constraints require optimizations and rearchitectures. I'd also expect the ram shortage for instance to have a big impact on the software industry as a whole, specially in games. They will need to make do with what people have, a ps5/pro or similar in PC power.
I actually think it is a good thing to introduce constraints to AI and the overall tech industry. Hopefully everyone will have to look at improving performance without having to add RAM or increase CPU/GPU performance.
Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
#5[dead]
Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
#6I went in expecting to find 'branch prediction'[0] as the answer, but apparently things are even more complex nowadays.
[0] - https://stackoverflow.com/questions/11227809/why-is-conditio...
Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
#7People have been noticing the effects of this in local LLM inference. Power limiting seems to improve overall performance!
This is not observable from LLM inference, where you would not encounter uniform matrices.
Power limiting does not improve performance but it does improve efficiency. You might be able to get 90% of the performance for only 70% of the power usage, for example. It does not make the card go faster though.
Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
#8Designing for predictable execution flow is one of the advantages of Tenstorrent hardware.
https://clehaxze.tw/gemlog/2025/04-21-programming-tensotrren...
https://clehaxze.tw/gemlog/2026/01-22-the-real-tenstorrent-t...
Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
#9I went in expecting to find 'branch prediction'[0] as the answer, but apparently things are even more complex nowadays. [0] - https://stackoverflow.com/questions/11227809/why-is-conditio...
To be fair, the culprit in the article is _less complex_ than branch prediction: "with random data, bits are flipped often, and bit flips in transistors inherently draw power" is less mental gymnastics than "with random data, the cpu fails to predict the future, causing redundant speculative execution"
Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)
#10[deleted]