11–20 of 60 posts
I could certainly come up with alternative theories about memory compression and prefetching if we were talking about texture reads.
I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?
> For example, when the GPU is fully idle, nvidia-smi tells me that it’s only pulling 88W of power. I haven't used a non-laptop GPU in some time, but that is a crazy amount of "idle" power consumption. Is this normal for cards like this?
For server gear it’s more common to have less dynamic power and voltage switching because it produces more predictable performance and latency.
Earlier quoted context omitted.
In general, constraints require optimizations and rearchitectures. I'd also expect the ram shortage for instance to have a big impact on the software industry as a whole, specially in games. They will need to make do with what people have, a ps5/pro or similar in PC power.
I actually think it is a good thing to introduce constraints to AI and the overall tech industry. Hopefully everyone will have to look at improving performance without having to add RAM or increase CPU/GPU performance.
It wouldn't surprise me to see some ML algorithm in silico somewhere to select faster matmul paths on favorable data. Yo dawg, I heard you like AI, so we put some AI in your AI so you can infer while you're inferring.
People have been noticing the effects of this in local LLM inference. Power limiting seems to improve overall performance!
This is not observable from LLM inference, where you would not encounter uniform matrices. Power limiting does not improve performance but it does improve efficiency. You might be able to get 90% of the performance for only 70% of the power usage, for example. It does not make the card go faster though.
This is precicely because of the efficiency. The lower efficiency of the higher speed triggers a much lower performance sooner.
I went in expecting to find 'branch prediction'[0] as the answer, but apparently things are even more complex nowadays. [0] - https://stackoverflow.com/questions/11227809/why-is-conditio...
GPUs do branch prediction? I thought they didn't bother and try to minimize wasted effort by using high amounts of concurrent threads?
Sounds like a side channel attack waiting to happen.