They are claiming a 25x reduction in power consumption. That can't be right. Anyone understand where this number is coming from?
Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
171–180 of 340 posts
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#172What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading. [0] https://youtu.be/Y2F8yisiS6E?t=4698
How can 4 bits possibly be enough? Are intermediate calculations done at a higher width and then converted down back to FP4?
The various sigmoid activation functions have the effect of keeping bit growth under control, by virtue of clamping to the +/- 1 range.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#173Earlier quoted context omitted.
Tell me more about why you believe their stock is hilariously overvalued.
Their market cap is 2.2T $. In the past year, they had a revenue of 60B $ and net income of 30B $. Absolutely amazing numbers, I agree. The year before they had a revenue of 30B $ and a net income of 4.5B $ - and it was a rather good year. What happens next of course depend of how you judge the situation - was it a peak hype demand ? Will it stabilize now ? Grow at current extraordinary rates ? Scenario 1 - margins g…
It does no good to design similar or even superior chips if you can't get them fabbed. How much of the world's fab capacity has Nvidia already reserved?
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#174What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading. [0] https://youtu.be/Y2F8yisiS6E?t=4698
How can 4 bits possibly be enough? Are intermediate calculations done at a higher width and then converted down back to FP4?
There are research papers where even 1 bit (not floating point) was enough, with some quality loss.
4 bits is effectively 16 different float point numbers - 8 positive, 8 negative, no zero and no NaN/inf. 1 bit for sign and 3 bits for exponent, 0 bits for mantissa, mantissa is implied to be 4. It’s logarithmic - representing numbers in the range from -4^3 to 4^3, smallest numbers are 4^-3.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#175Earlier quoted context omitted.
Is that how it works? Why don't we just put many chips in one computer?
the massive blackwell SoC he showed is two Blackwell dies with an interconnect. It’s very similar to what Apple does with their Ultra series. Then the B200 package is 2 of these plus a CPU. So a total of 4 GPU dies in each unit.
> Then the B200 package is 2 of these plus a CPU.
That's GB200.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#176Which Blackwell is it named after?
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#177"Platform company" means multi-chip in this case? Seems logical since it's becoming impractical to cram so many transistors on a single die.
no it means rent seeking. imagine aws if they also sold all computers in the world, now you can only rent from them
"For only 100$ a month, you'll be able to turn on the gpu you already paid for"
--Nvidia, pretty soon
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#178> “Nvidia … is becoming less of a mercenary chip provider and more of a platform provider, like Microsoft or Apple, on which other companies can build software.
I can understand from a growth perspective why it’s more profitable for Nvidia if it can become more of a platform service for AI. However, that’s difficult to balance that and partnerships the company already has with AWS and Microsoft. I’d expect to see some acquisitions or competing custom solutions in the future. Fortunately for Nvidia, a lot of AI is still dependent on CUDA. I’m interested to see how this plays out.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#179What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading. [0] https://youtu.be/Y2F8yisiS6E?t=4698
>bit misleading. Only partially, because in LLMs FP4 isn't half as useful as FP8. So if you have gear that crushes at FP4 then that's what you use and you benefit from that increased speed (at minimal accuracy loss). Definitely some marketing creativity in there, but its not entirely wrong as a measure of real world usage
curiously, what real world usage actually uses FP4? AFICT, most of the LLMs still use BF16, and even the quantizations down to 4bits and 2bits end up back to 16bit or INT8 for actual computations.
Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’
#180They are claiming a 25x reduction in power consumption. That can't be right. Anyone understand where this number is coming from?
Did you read that in the linked article? I couldn’t find it. But maybe due to the better efficiency with regard to the performance boost (5x) and the ability to now use 27 trillion parameters versus 1.7 Trillion, one can presumably finish the same amount of work in 1/25th of the time and bam, reduction in power consumption. As you say, I’m skeptical the max power draw itself is 25x lower.