Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’ • Hacker News Board

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#181

Post by dheera » Tue, Mar 19, 2024, 3:25 AM UTC

cebert wrote: post #178

> “Nvidia … is becoming less of a mercenary chip provider and more of a platform provider, like Microsoft or Apple, on which other companies can build software. I can understand from a growth perspective why it’s more profitable for Nvidia if it can become more of a platform service for AI. However, that’s difficult to balance that and partnerships the company already has with AWS and Microsoft. I’d expect to see som…

My prediction is eventually there will be anti-trust ligitation, they will be required to open the CUDA standard, after which AMD will become a competitor.

NVIDIA could voluntarily open the standard to avoid this ligitation if they wanted to, though, and IMO it would be the smart thing to do, but almost every corporation in history has chosen the ligitation instead.

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#182

Post by anon291 » Tue, Mar 19, 2024, 3:27 AM UTC

s_m_t wrote: post #121

qwertox wrote: post #64

What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading. [0] https://youtu.be/Y2F8yisiS6E?t=4698

How can 4 bits possibly be enough? Are intermediate calculations done at a higher width and then converted down back to FP4?

The fundamental 'unit' of NN computation is not an individual vector element but rather an entire vector. One of the first results you often learn about in linear algebra is that some axes are more important than others (principal components, singular value decomposition). Thus, it totally stands to reason that the underlying field of the vector is inconsequential but rather the entire vector machinery. All you have to do is make sure that there are enough elements in the vector to get the job done for whatever bit size of element.

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#183

Post by wmf » Tue, Mar 19, 2024, 3:32 AM UTC

cebert wrote: post #178

> “Nvidia … is becoming less of a mercenary chip provider and more of a platform provider, like Microsoft or Apple, on which other companies can build software. I can understand from a growth perspective why it’s more profitable for Nvidia if it can become more of a platform service for AI. However, that’s difficult to balance that and partnerships the company already has with AWS and Microsoft. I’d expect to see som…

I think they're planning for a world where half their customers (hyperscalers) just use GPUs and CUDA while the other half (the long tail) use more profitable, higher-level parts of the platform. They don't have the leverage to force customers one way or the other. It would be easier to just sell GPUs, but they know that sophisticated customers can switch to other chips while the platform provides lock-in for smaller customers.

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#184

Post by wmf » Tue, Mar 19, 2024, 3:39 AM UTC

dheera wrote: post #181

cebert wrote: post #178

> “Nvidia … is becoming less of a mercenary chip provider and more of a platform provider, like Microsoft or Apple, on which other companies can build software. I can understand from a growth perspective why it’s more profitable for Nvidia if it can become more of a platform service for AI. However, that’s difficult to balance that and partnerships the company already has with AWS and Microsoft. I’d expect to see som…

My prediction is eventually there will be anti-trust ligitation, they will be required to open the CUDA standard, after which AMD will become a competitor. NVIDIA could voluntarily open the standard to avoid this ligitation if they wanted to, though, and IMO it would be the smart thing to do, but almost every corporation in history has chosen the ligitation instead.

It would be kind of genius for Nvidia to "open" the CUDA APIs (which have already been unofficially reverse engineered anyway) but not the code. Maybe they'd also officially support HIP and SYCL. Maybe they could open SXM after all competitors have already committed to OAM. They'd create the appearance of opening up while giving up very little.

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#185

Post by jimmySixDOF » Tue, Mar 19, 2024, 3:40 AM UTC

azeirah wrote: post #9

dagmx wrote: post #6

FP8 being 2.5x Hopper is kind of disappointing after such a long time. Since its 2 fused chips, that means it’s 25% effective delta. though it seems most of the progress has been on memory throughput and power use which is still very impressive. I wonder how this will trickle down to the consumer segment.

Jensen revealed later that the LLM inference is 30x due to architectural improvements, it's massive. I don't know if it's latency or just 2-3x performance boost with 30x more customers served in the same chip. Either way, 30x is massive.

This is also the only place Nvidia are getting competitive pressure - from the likes of Groq (and likely but less published from Cerebras) with higher inferance T/s and concurrency utilization/batching [1] so if this proves to be the true then the case for big chip systems (on todays specs) will be harder.

[1]https://twitter.com/swyx/status/1760065636410274162?t=rpbcr8...

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#186

Post by s_m_t » Tue, Mar 19, 2024, 3:43 AM UTC

yalok wrote: post #174

s_m_t wrote: post #121

Earlier quoted context omitted.

How can 4 bits possibly be enough? Are intermediate calculations done at a higher width and then converted down back to FP4?

There are research papers where even 1 bit (not floating point) was enough, with some quality loss. 4 bits is effectively 16 different float point numbers - 8 positive, 8 negative, no zero and no NaN/inf. 1 bit for sign and 3 bits for exponent, 0 bits for mantissa, mantissa is implied to be 4. It’s logarithmic - representing numbers in the range from -4^3 to 4^3, smallest numbers are 4^-3.

Thanks, I was thinking that zero, negative zero, inf, negative inf, and the NaN's were included like in IEEE 754

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#187

Post by s_m_t » Tue, Mar 19, 2024, 3:44 AM UTC

anon291 wrote: post #182

s_m_t wrote: post #121

Earlier quoted context omitted.

How can 4 bits possibly be enough? Are intermediate calculations done at a higher width and then converted down back to FP4?

The fundamental 'unit' of NN computation is not an individual vector element but rather an entire vector. One of the first results you often learn about in linear algebra is that some axes are more important than others (principal components, singular value decomposition). Thus, it totally stands to reason that the underlying field of the vector is inconsequential but rather the entire vector machinery. All you have…

I see, so the idea is that enough of the quantization errors are sort of averaged out across the dimensions of the vector space to still be useful?

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#188

Post by wmf » Tue, Mar 19, 2024, 3:48 AM UTC

LTL_FTC wrote: post #180

jakobov wrote: post #171

They are claiming a 25x reduction in power consumption. That can't be right. Anyone understand where this number is coming from?

Did you read that in the linked article? I couldn’t find it. But maybe due to the better efficiency with regard to the performance boost (5x) and the ability to now use 27 trillion parameters versus 1.7 Trillion, one can presumably finish the same amount of work in 1/25th of the time and bam, reduction in power consumption. As you say, I’m skeptical the max power draw itself is 25x lower.