41–50 of 60 posts
I went in expecting to find 'branch prediction'[0] as the answer, but apparently things are even more complex nowadays. [0] - https://stackoverflow.com/questions/11227809/why-is-conditio...
When you make it so the computer does not have to compute all possible states of matter it finishes faster.
> For example, when the GPU is fully idle, nvidia-smi tells me that it’s only pulling 88W of power. I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?
Yes, I have some of these cards and AFAICT the HBM2e chips just always run at full speed. I have different variants of the pcie cards and while I can get the gpu itself into a lower power state the memory just runs full tilt. Though I see 40w on my “normal” cards and 60w on the Frankenstein card that thinks it’s an sxm4.
That aside idle power consumption is a driver-to-driver affair from both amd and novideo, sometimes I'm only pulling 15-30W when nothing is happening and other times it decides it needs 110w for a static 500hz screen
Earlier quoted context omitted.
To be fair, the culprit in the article is _less complex_ than branch prediction: "with random data, bits are flipped often, and bit flips in transistors inherently draw power" is less mental gymnastics than "with random data, the cpu fails to predict the future, causing redundant speculative execution"
But why do we expect random data to result in more bit flips? That seems harder to argue than the mechanics of a basic branch prediction system.
Earlier quoted context omitted.
You didn't touch on the most important aspect for cost: die area! How much die space ($) will that circuitry, that's probably statistically near zero chance for you main customers workload (who has model weight of 0 or 1!?), add. And, if you can stomach the cost, what else could you put there instead?
Weights should not be 0 (at least not frequently) but in a ReLU-based neural network, activations are 0 pretty often. You're absolutely right about die area though.
What percent of this hardware is running inference for ReLU models? ;)
Earlier quoted context omitted.
But why do we expect random data to result in more bit flips? That seems harder to argue than the mechanics of a basic branch prediction system.
Think about it from the other end. Why would any bits flip at all in the data path of your matrix multiplier when all the matrices are 0?
And what of the Pi test - I’d expect that to flip many more bits than the 1-bit one.
Earlier quoted context omitted.
Think about it from the other end. Why would any bits flip at all in the data path of your matrix multiplier when all the matrices are 0?
Sure, when comparing 0’s to anything else. But what about normal distribution to uniform in 0,1? The author hand waves something about signs but it’s not very well reasoned - that’s just a single bit in floats. And what of the Pi test - I’d expect that to flip many more bits than the 1-bit one.
Normal vs uniform is less clear, but also not as much of a difference. The arguments about signs isn't just about a signs bit, though. The way you negate during accumulation is that you flip all the bits. Only the final float representation is sign+magnitude, the accumulation itself has two's complement steps. I don't actually know the analysis here, just pointing out that it's not that simple.
I can't tell from the blog, is this actually verified or is it theory and then numbers showing plausibility? I could certainly come up with alternative theories about memory compression and prefetching if we were talking about texture reads.
Earlier quoted context omitted.
Sure, when comparing 0’s to anything else. But what about normal distribution to uniform in 0,1? The author hand waves something about signs but it’s not very well reasoned - that’s just a single bit in floats. And what of the Pi test - I’d expect that to flip many more bits than the 1-bit one.
If the inputs are constant, then all the multiplies are constant and the only thing that toggles is the accumulation. Which explains the pi situation. Normal vs uniform is less clear, but also not as much of a difference. The arguments about signs isn't just about a signs bit, though. The way you negate during accumulation is that you flip all the bits. Only the final float representation is sign+magnitude, the accum…
> For example, when the GPU is fully idle, nvidia-smi tells me that it’s only pulling 88W of power. I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?
Server cards are not optimized for idle power usage. They’re expected to be fully utilized. For server gear it’s more common to have less dynamic power and voltage switching because it produces more predictable performance and latency.