Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

thonking.ai

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#51

Post by ggambetta »

Thorondor wrote: post #35
ggambetta wrote: post #33

I'd have guessed multiply-by-0 and multiply-by-1 can be special-cased to run much faster and simpler code paths, like you'd do when writing MUL for a processor that doesn't have it (I <3 z80)

Hardware engineer here. Special casing the multiply by 0 and multiply by 1 paths is harder than it sounds. In software, the cost of adding special cases is simply performance. You're adding more instructions that execute in sequence on a CPU that already physically exists. Doing this for your multiply case is worthwhile because the speedup is large for 0 and 1 while the cost is not that large (relative to the time ta…

Thanks for the detailed explanation, I had no idea about any of this.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#52

Post by imtringued »

eff-nix wrote: post #46
atq2119 wrote: post #44

Earlier quoted context omitted.

Think about it from the other end. Why would any bits flip at all in the data path of your matrix multiplier when all the matrices are 0?

Sure, when comparing 0’s to anything else. But what about normal distribution to uniform in 0,1? The author hand waves something about signs but it’s not very well reasoned - that’s just a single bit in floats. And what of the Pi test - I’d expect that to flip many more bits than the 1-bit one.

Field effect transistors are basically a capacitor. They store energy.

If you switch a not gate's input from zero to one to zero and so on, the gate capacitance will have to charge and discharge. The entire idea behind CMOS is that if you have n and p channel transistors together, you can take advantage of the fact that electrons are more mobile than holes. Filling and draining electrons gives you a greater switching speed.

If the input stays the same, then the charge at the input inside the flip flop is the same as the charge inside the not gate. No charge differential means no electrons move, which means there is no ohmic resistance that causes the internal metal and polysilicon interconnect to heat up and less power gets lost and no switching obviously happens faster than some switching.

TL;DR If you randomize the data, you will constantly charge and discharge the capacitors.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#53

Post by imtringued »

nomel wrote: post #36
Thorondor wrote: post #35

Earlier quoted context omitted.

Hardware engineer here. Special casing the multiply by 0 and multiply by 1 paths is harder than it sounds. In software, the cost of adding special cases is simply performance. You're adding more instructions that execute in sequence on a CPU that already physically exists. Doing this for your multiply case is worthwhile because the speedup is large for 0 and 1 while the cost is not that large (relative to the time ta…

You didn't touch on the most important aspect for cost: die area! How much die space ($) will that circuitry, that's probably statistically near zero chance for you main customers workload (who has model weight of 0 or 1!?), add. And, if you can stomach the cost, what else could you put there instead?

Nvidia has added structural sparsity to their GPUs and every time they pull out a flops or tops number, they assume you will use structural sparsity.

The die area argument here makes no sense. Supporting structural sparsity can be done either by duplicating the multipliers with and without the support or you have a single general purpose multiplier that does both, in which case you can have twice as many of them.

Also, in ReLU^2 networks, 90%+ parameters are zero.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#54

Post by rowanG077 »

nomel wrote: post #36
Thorondor wrote: post #35

Earlier quoted context omitted.

Hardware engineer here. Special casing the multiply by 0 and multiply by 1 paths is harder than it sounds. In software, the cost of adding special cases is simply performance. You're adding more instructions that execute in sequence on a CPU that already physically exists. Doing this for your multiply case is worthwhile because the speedup is large for 0 and 1 while the cost is not that large (relative to the time ta…

You didn't touch on the most important aspect for cost: die area! How much die space ($) will that circuitry, that's probably statistically near zero chance for you main customers workload (who has model weight of 0 or 1!?), add. And, if you can stomach the cost, what else could you put there instead?

I expect the degraded critical path will most likely be worse than a bit of die area. On modern processes you have A LOT of transistors to play with.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#55

Post by rowanG077 »

bitwize wrote: post #14

It wouldn't surprise me to see some ML algorithm in silico somewhere to select faster matmul paths on favorable data. Yo dawg, I heard you like AI, so we put some AI in your AI so you can infer while you're inferring.

This is old news. AMD was this in their CPUs years ago.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#56

Post by kzrdude »

kangalioo wrote: post #9
nzach wrote: post #6

I went in expecting to find 'branch prediction'[0] as the answer, but apparently things are even more complex nowadays. [0] - https://stackoverflow.com/questions/11227809/why-is-conditio...

To be fair, the culprit in the article is _less complex_ than branch prediction: "with random data, bits are flipped often, and bit flips in transistors inherently draw power" is less mental gymnastics than "with random data, the cpu fails to predict the future, causing redundant speculative execution"

Good point but this forum leans heavily towards software, so we are used to the latter! I have worked close to IC development so had to learn and tangle with the former idea too, was interesting.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#58

Post by Ardon »

Aurornis wrote: post #7
gdevenyi wrote: post #2

People have been noticing the effects of this in local LLM inference. Power limiting seems to improve overall performance!

This is not observable from LLM inference, where you would not encounter uniform matrices. Power limiting does not improve performance but it does improve efficiency. You might be able to get 90% of the performance for only 70% of the power usage, for example. It does not make the card go faster though.

This isn't necessarily true, especially with consumer GPUs. Some actually can clock higher with less voltage. It's pretty rare, and mostly comes from factory overclocked cards. For it to help you need a card that 1. Thermal throttles 2. can sustain its max OC with less voltage than set at the factory. In that rare case you are removing thermal pressure which allows you to clock higher for longer. It's the silicon lottery though, and (often) lazy board partners just smashing the voltage up as high as the chip maker allows. You definitely won't get more performance from a datacenter GPU this way.

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#59

Post by jayd16 »

saagarjha wrote: post #48
jayd16 wrote: post #11

I can't tell from the blog, is this actually verified or is it theory and then numbers showing plausibility? I could certainly come up with alternative theories about memory compression and prefetching if we were talking about texture reads.

It’s real, you can measure it yourself on modern Nvidia hardware

I can test the symptoms, but what is the proof that gate switching is the actual problem?

Re: Matrix Multiplications on GPUs Run Faster When Given “Predictable” Data (2024)

#60

Post by nomel »

imtringued wrote: post #53
nomel wrote: post #36

Earlier quoted context omitted.

You didn't touch on the most important aspect for cost: die area! How much die space ($) will that circuitry, that's probably statistically near zero chance for you main customers workload (who has model weight of 0 or 1!?), add. And, if you can stomach the cost, what else could you put there instead?

Nvidia has added structural sparsity to their GPUs and every time they pull out a flops or tops number, they assume you will use structural sparsity. The die area argument here makes no sense. Supporting structural sparsity can be done either by duplicating the multipliers with and without the support or you have a single general purpose multiplier that does both, in which case you can have twice as many of them. Als…

> The die area argument here makes no sense.

Any logic you add to the GPU is physical silicon and metal that take up physical space.

> duplicating the multipliers with and without the support or you have a single general purpose multiplier that does both

That would be extra physical logic, which would be extra physical space on the die. "can be done" isn't my point, it's that "doing requires surface area".

Post reply on HN