Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

thonking.ai

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#31

Post by PcChip »

Aurornis wrote: post #26
Lerc wrote: post #18

Earlier quoted context omitted.

When thermal throttling occurs you can perform faster by running slower. This is precicely because of the efficiency. The lower efficiency of the higher speed triggers a much lower performance sooner.

> When thermal throttling occurs you can perform faster by running slower. This is not true unless the throttling algorithm is so broken that it's oscillating between extremes. The parts have a curve of clock speed versus voltage. More clock speed means higher performance. That goes further up the voltage curve, meaning more power. Throttling just moves the card further down the voltage to clock speed curve. It reduc…

with a lower power cap set, it runs cooler, which sometimes allows the GPU to reach higher boost speeds. This is a real effect on gaming GPUs - however I have no idea if it applies to datacenter GPUs

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#32

Post by zacmps »

School-Cotton wrote: post #29
dan_sbl wrote: post #12

> For example, when the GPU is fully idle, nvidia-smi tells me that it’s only pulling 88W of power. I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?

I suspect the act of running nvidia-smi itself prevents the GPU from being put into a low-power state.

From memory this is true and nvml (Nvidia management library) is the way to get stats that doesn't cause the GPU to wake.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#34

Post by legitronics »

dan_sbl wrote: post #12

> For example, when the GPU is fully idle, nvidia-smi tells me that it’s only pulling 88W of power. I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?

Yes, I have some of these cards and AFAICT the HBM2e chips just always run at full speed. I have different variants of the pcie cards and while I can get the gpu itself into a lower power state the memory just runs full tilt. Though I see 40w on my “normal” cards and 60w on the Frankenstein card that thinks it’s an sxm4.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#35

Post by Thorondor »

ggambetta wrote: post #33

I'd have guessed multiply-by-0 and multiply-by-1 can be special-cased to run much faster and simpler code paths, like you'd do when writing MUL for a processor that doesn't have it (I <3 z80)

Hardware engineer here. Special casing the multiply by 0 and multiply by 1 paths is harder than it sounds. In software, the cost of adding special cases is simply performance. You're adding more instructions that execute in sequence on a CPU that already physically exists. Doing this for your multiply case is worthwhile because the speedup is large for 0 and 1 while the cost is not that large (relative to the time taken for the whole multiplication operation) for other values.

Hardware is different. Every operation that can be performed in hardware by a chip needs dedicated circuitry. Special casing 0 and 1 means adding at least OR reduction on each operand and a dedicated multiplexer for every bit of the output. Those transistors use power even when they're not in use (leakage power is a huge issue on modern semiconductor processes). They also degrade timing by adding more gates on critical paths through the multipliers. (The timing issue here is that all operations that happen between one flip-flop and another flip-flop need to finish within one clock cycle.) And unless there are whole blocks of 0's and 1's (this does happen in certain neural networks), you typically won't see a direct speedup anyway. In software terms, the matrix multiply is scheduled as many parallel operations that cannot be accelerated much overall by skipping a few operations in some "threads."

All of this makes zero skipping a nontrivial topic. People do still try to do it but it needs serious consideration as, depending on the application, the case is rarely one-sided.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#36

Post by nomel »

Thorondor wrote: post #35
ggambetta wrote: post #33

I'd have guessed multiply-by-0 and multiply-by-1 can be special-cased to run much faster and simpler code paths, like you'd do when writing MUL for a processor that doesn't have it (I <3 z80)

Hardware engineer here. Special casing the multiply by 0 and multiply by 1 paths is harder than it sounds. In software, the cost of adding special cases is simply performance. You're adding more instructions that execute in sequence on a CPU that already physically exists. Doing this for your multiply case is worthwhile because the speedup is large for 0 and 1 while the cost is not that large (relative to the time ta…

You didn't touch on the most important aspect for cost: die area!

How much die space ($) will that circuitry, that's probably statistically near zero chance for you main customers workload (who has model weight of 0 or 1!?), add. And, if you can stomach the cost, what else could you put there instead?

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#37

Post by DiabloD3 »

buildbot wrote: post #25
unglaublich wrote: post #20

Earlier quoted context omitted.

So I guess we'll all be applying a random rotation to our matrices now to obscure their contents, like TurboQuant does. https://arkaung.github.io/interactive-turboquant/#rotation

Not that it super matters, but random hadamards for quantization have been a thing since way before turboquant. https://arxiv.org/abs/2404.00456

Which llama.cpp now does.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#38

Post by Thorondor »

nomel wrote: post #36
Thorondor wrote: post #35

Earlier quoted context omitted.

Hardware engineer here. Special casing the multiply by 0 and multiply by 1 paths is harder than it sounds. In software, the cost of adding special cases is simply performance. You're adding more instructions that execute in sequence on a CPU that already physically exists. Doing this for your multiply case is worthwhile because the speedup is large for 0 and 1 while the cost is not that large (relative to the time ta…

You didn't touch on the most important aspect for cost: die area! How much die space ($) will that circuitry, that's probably statistically near zero chance for you main customers workload (who has model weight of 0 or 1!?), add. And, if you can stomach the cost, what else could you put there instead?

Weights should not be 0 (at least not frequently) but in a ReLU-based neural network, activations are 0 pretty often. You're absolutely right about die area though.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#39

Post by eff-nix »

kangalioo wrote: post #9
nzach wrote: post #6

I went in expecting to find 'branch prediction'[0] as the answer, but apparently things are even more complex nowadays. [0] - https://stackoverflow.com/questions/11227809/why-is-conditio...

To be fair, the culprit in the article is _less complex_ than branch prediction: "with random data, bits are flipped often, and bit flips in transistors inherently draw power" is less mental gymnastics than "with random data, the cpu fails to predict the future, causing redundant speculative execution"

But why do we expect random data to result in more bit flips? That seems harder to argue than the mechanics of a basic branch prediction system.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#40

Post by Lerc »

Aurornis wrote: post #26
Lerc wrote: post #18

Earlier quoted context omitted.

When thermal throttling occurs you can perform faster by running slower. This is precicely because of the efficiency. The lower efficiency of the higher speed triggers a much lower performance sooner.

> When thermal throttling occurs you can perform faster by running slower. This is not true unless the throttling algorithm is so broken that it's oscillating between extremes. The parts have a curve of clock speed versus voltage. More clock speed means higher performance. That goes further up the voltage curve, meaning more power. Throttling just moves the card further down the voltage to clock speed curve. It reduc…

>This is not true unless the throttling algorithm is so broken that it's oscillating between extremes.

That algorithm is doing exactly the task I described. If it could temporarily run faster but in a way that would cause occilation, that literally means it can run faster but it is choosing not to to preserve overall performance.

Post reply on HN