Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

thonking.ai

Post reply on HN
Print view View original

21–30 of 60 posts

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#21

Post by Nevermark »

bitwize wrote: post #14

It wouldn't surprise me to see some ML algorithm in silico somewhere to select faster matmul paths on favorable data. Yo dawg, I heard you like AI, so we put some AI in your AI so you can infer while you're inferring.

Here is one: An adjustment to weight updates, that makes it more likely for weights to stay uniformly distributed.

~257.5 teraflops for normal distribution, versus ~268 teraflops uniform, reported on the first graph.

I would have liked to see a straight graph of performance vs. clock speed, for each type of data. Pick your data statistics, then pick the peak performance clock speed accordingly.

And for actual runs, from a pre-run sampled curve.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#22

Post by cmovq »

Aurornis wrote: post #15
dan_sbl wrote: post #12

> For example, when the GPU is fully idle, nvidia-smi tells me that it’s only pulling 88W of power. I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?

Server cards are not optimized for idle power usage. They’re expected to be fully utilized. For server gear it’s more common to have less dynamic power and voltage switching because it produces more predictable performance and latency.

For GeForce cards you can get similar behavior by setting “Prefer maximum performance” which disables some of the low power states.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#23

Post by jayd16 »

gruez wrote: post #19
nzach wrote: post #6

I went in expecting to find 'branch prediction'[0] as the answer, but apparently things are even more complex nowadays. [0] - https://stackoverflow.com/questions/11227809/why-is-conditio...

>I went in expecting to find 'branch prediction'[0] GPUs do branch prediction? I thought they didn't bother and try to minimize wasted effort by using high amounts of concurrent threads?

They do texture prefetching, which is sorta similar.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#24

Post by wildzzz »

dan_sbl wrote: post #12

> For example, when the GPU is fully idle, nvidia-smi tells me that it’s only pulling 88W of power. I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?

If my gpu is sitting idle, and I mean idle with nothing loaded into its memory, it's sitting at about 18W. If I load in model that uses nearly all of the memory but that model is idle, it's at 36W. If that model is actively thinking, it's like 118W. I think this is likely due to the GPU being aware that there is real data loaded into memory and turning up the DRAM refresh rate whereas when nothing is loaded, the dynamic power is as low as possible.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#25

Post by buildbot »

unglaublich wrote: post #20
amelius wrote: post #13

Sounds like a side channel attack waiting to happen.

So I guess we'll all be applying a random rotation to our matrices now to obscure their contents, like TurboQuant does. https://arkaung.github.io/interactive-turboquant/#rotation

Not that it super matters, but random hadamards for quantization have been a thing since way before turboquant.

https://arxiv.org/abs/2404.00456

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#26

Post by Aurornis »

Lerc wrote: post #18
Aurornis wrote: post #7

Earlier quoted context omitted.

This is not observable from LLM inference, where you would not encounter uniform matrices. Power limiting does not improve performance but it does improve efficiency. You might be able to get 90% of the performance for only 70% of the power usage, for example. It does not make the card go faster though.

When thermal throttling occurs you can perform faster by running slower. This is precicely because of the efficiency. The lower efficiency of the higher speed triggers a much lower performance sooner.

> When thermal throttling occurs you can perform faster by running slower.

This is not true unless the throttling algorithm is so broken that it's oscillating between extremes.

The parts have a curve of clock speed versus voltage. More clock speed means higher performance. That goes further up the voltage curve, meaning more power.

Throttling just moves the card further down the voltage to clock speed curve. It reduces clock speed, reducing performance.

The cards don't "perform faster by running slower". If you run the card slower, it performs slower.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#27

Post by jetsamflotsam »

I feel like many of the comments missed the point or didn't read the article. What I believe this article is stating (and I've read this many times during my PhD for various reasons), is that the input data distributions affect how many transistor state changes there are during multiplication. Since these events are a large portion of energy loss/heat generation, the clocks won't be throttled as much for certain data patterns.

There was a workshop paper from SC24 that did more experiments around this I believe. I can't find it now though.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#28

Post by bee_rider »

nzach wrote: post #6

I went in expecting to find 'branch prediction'[0] as the answer, but apparently things are even more complex nowadays. [0] - https://stackoverflow.com/questions/11227809/why-is-conditio...

I expected a “torch is smart enough to keep track of cases where it just initialized the C in C <= A*B+C to zero, avoiding the add” type situation but I was wrong.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#29

Post by School-Cotton »

dan_sbl wrote: post #12

> For example, when the GPU is fully idle, nvidia-smi tells me that it’s only pulling 88W of power. I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?

I suspect the act of running nvidia-smi itself prevents the GPU from being put into a low-power state.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#30

Post by ryanisnan »

nzach wrote: post #6

I went in expecting to find 'branch prediction'[0] as the answer, but apparently things are even more complex nowadays. [0] - https://stackoverflow.com/questions/11227809/why-is-conditio...

That's exactly what I thought.
Post reply on HN