Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

cnbc.com

Post reply on HN
Print view View original

311–320 of 340 posts

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#311

Post by bayindirh »

gammalost wrote: post #305
bayindirh wrote: post #263

Earlier quoted context omitted.

Wasteful software development is easy and keeps momentum for development. As long as growth is king, quick and dirty will always beat well optimized and smaller systems. This is not a problem with AI only, but with every software we use. Only two groups try to optimize things and try to fit into smaller systems. Passionate programmers and people who is paid to do this (e.g.: phone manufacturers' software teams, etc.)…

I don't think it is thanks to wasteful software development. The libraries used for LLMs do a lot to squeeze out the full potential of GPUs. I think it is more of an information problem. How can we store enough information in weights so that it is possible to train models without a budget similar to OpenAI

Just because you're squeezing out every bit of performance from a processor doesn't mean all the work you're doing is meaningful or can't be optimized.

I work on material simulations. I make processors hit their TDPs, saturate their pipelines and make them go as fast as they can. However, sometimes we come up with a formula optimization which does things 1-2% faster, which means we can save hours on a bigger computation. Utilization doesn't change, but speed does.

> I think it is more of an information problem.

It's an interesting point of view, and partially true. However, we're still wasting "space" by just adding bits to the network to make it contain more data.

There's a long way to go. "Wasteful development" is a phase and always be part of software development. The important part is not forgetting that optimization exists. Otherwise we can't sustain ourselves much with all that energy use.

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#312

Post by sitkack »

incrudible wrote: post #249
sitkack wrote: post #207

Earlier quoted context omitted.

By "Opening Up" they cement their leadership position. AI frameworks are already targeting CL, SPIR-V, etc. The low level details will fade and so will Nvidias api dominance. The MI300 smokes the H100 yet here we are.

Just because they are a target doesn’t mean things just work. Historically, AMD hardware for GPGPU becomes obsolete well before the software landscape catches up. I am not going to risk my time and money finding out whether history repeats itself, just for a few potential FLOPS per dollar.

Don't disagree, but it is is nuts how AMD is leaving billions on the table by not finishing the project by writing the software.

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#313

Post by DanielHB »

cebert wrote: post #178

> “Nvidia … is becoming less of a mercenary chip provider and more of a platform provider, like Microsoft or Apple, on which other companies can build software. I can understand from a growth perspective why it’s more profitable for Nvidia if it can become more of a platform service for AI. However, that’s difficult to balance that and partnerships the company already has with AWS and Microsoft. I’d expect to see som…

AWS is pushing ARM hard, yet people still buy x86/x64 compute en mass

Even if AWS has its own hardware+software solution for neural networks it would still take years if not decades to tear off the CUDA platform

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#314

Post by AYBABTME »

rapsey wrote: post #222
elbear wrote: post #211

Earlier quoted context omitted.

Why's that?

Because it is a different type of engineering. If you manage software development like you manage hardware development your software is going to be bad. That has always been AMD's problem and it is not likely to get fixed.

2t$ problem of egos?

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#315

Post by lotsofpulp »

blagie wrote: post #308
lotsofpulp wrote: post #301

Earlier quoted context omitted.

>There's a reason Facebook is worth about 10x what SpaceX is worth, and most of that value doesn't come from the ability to build software. Facebook isn't even particularly good at building software. One is a publicly listed business with as much of an objective look at real time "worth" as possible in today's world, and the other is a private business with confidential financials. Seems like you would be unable to e…

SpaceX raised $750M at a valuation of $137B in January 2023. A private investment at this scale should have a lot more transparency and due diligence than disclosures from a SEC disclosures. If I were investing $750M, I'd have engineers under NDA review SpaceX technologies, financial auditors, legal auditors, etc. Secondary sales place it a little bit higher (but those typically have all the issues you describe).

Fair enough, didn’t know about that recent round. Still, I would assume that number is higher than it would be if the business were publicly listed, but the $140B should be close enough.

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#316

Post by omikun »

phh wrote: post #253
yalok wrote: post #174

Earlier quoted context omitted.

There are research papers where even 1 bit (not floating point) was enough, with some quality loss. 4 bits is effectively 16 different float point numbers - 8 positive, 8 negative, no zero and no NaN/inf. 1 bit for sign and 3 bits for exponent, 0 bits for mantissa, mantissa is implied to be 4. It’s logarithmic - representing numbers in the range from -4^3 to 4^3, smallest numbers are 4^-3.

Thanks. First source i see for what fp4 is. Gotta say I'm surprised: I would have chosen to lose one value, but have a zero. (though I have no doubt those people are much more clever and knowledgeable than I am)

If the weight is zero it doesn’t need to exist

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#317

Post by fisf »

djtango wrote: post #219
__mharrison__ wrote: post #204

My take from being at the keynote and the content I've seen so far at the conference is that Nvidia's is moving up the stack (like all good hardware vendors are prone to do). Obviously they are going to keep doing bigger. But the takeaway for me is that they are building "docker for llms" - NIM. They are building a container system where you can download/buy(?) NIMs and easily deploy them on their hardware. Going to…

I'm not that abreast of all the developments in the AI space. What specific class of AI startups do you have in mind here? AI-aaS startups who provide the "infra"?

AI startups who just wrap a (standard) API or model in a thin UI layer. The backend part will be a commodity and the UI layer offers no value proposition.

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#318

Post by dogma1138 »

matt-p wrote: post #302
nemothekid wrote: post #194

Earlier quoted context omitted.

> My prediction is eventually there will be anti-trust ligitation, they will be required to open the CUDA standard, after which AMD will become a competitor. If AMD isn't a competitor before government intervention, I don't the government forcing nvidia to open up CUDA changes much. CUDA's moat isn't due to some secret sauce - nvidia put in the developer hours; and if AMDs CUDA implementation is still broken, people…

It changes alot. It is not legal to make a 'CUDA' driver for an AMD GPU as Nvidia own cuda. You can see there was a open implementation of this that AMD sponsored until they got threatened with a lawsuit by Nvidia

ZLUDA ate the dust not because they implemented CUDA but because they were misusing complied NVIDIA libraries.

If it was a clean room implementation of the API NVIDIA wouldn’t care. Heck that’s exactly what AMD did with HIP.

But what you cannot do is essentially intercept calls to and reverse engineer NVIDIA binaries in real time because you can’t be arsed to build your own.

Re: Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

#319

Post by nemothekid »

matt-p wrote: post #302
nemothekid wrote: post #194

Earlier quoted context omitted.

> My prediction is eventually there will be anti-trust ligitation, they will be required to open the CUDA standard, after which AMD will become a competitor. If AMD isn't a competitor before government intervention, I don't the government forcing nvidia to open up CUDA changes much. CUDA's moat isn't due to some secret sauce - nvidia put in the developer hours; and if AMDs CUDA implementation is still broken, people…

It changes alot. It is not legal to make a 'CUDA' driver for an AMD GPU as Nvidia own cuda. You can see there was a open implementation of this that AMD sponsored until they got threatened with a lawsuit by Nvidia

>that AMD sponsored

The problem currently, as people like Hotz and many others are discovering, it not the lack of CUDA. Most people use PyTorch and don't care what the underlying software is. Infact most CUDA is hand tuned to nvidia hardware anyways and is optimized to make the most on nvidia. The problem is AMD's drivers - the piece that actually sends the code to run on the GPU, tends to be broken. AMD cannot "sponsor" an outsider to fix this. A legal, but broken, AMDCUDA will not be any better than the current situation; so no, having CUDA on AMD wouldn't change anything.

The problem is not "CUDA is not AMD", the problem is AMD has not, does not, and for some reason will not invest adequately in GPU compute. CUDA is a mirage; if AMD had a similar platform someone would have done the work already to ensure PyTorch works on it. PyTorch already supports ROCm, people don't use it because the performance is bad and it's buggy. When nvidia had this problem, nvidia hired engineers to work on open source projects and debug issues in open source libraries (not even limited to AI, you will find nvidia engineers debugging issues in a wide range of CUDA projects). When AMD has this issue, they barely acknowledge it.

Post reply on HN