Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

anandtech.com

Post reply on HN
Print view View original

31–40 of 111 posts

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#31

Post by JonChesterfield »

andrewstuart wrote: post #25

GPUs are going to be weird, underconfigured and overpriced until there is real competition. Whether or not there is real competition depends entirely on whether Intels Arc line of GPUs stays in the market. AMD strangely has decided not to compete. Its newest GPU the 7900 XTX is an extremely powerful card, close to the top of the line Nvidia RTX 4090 in raster performance. If AMD had introduced it with an aggressively…

GPUs strike me as absurdly cheap given the performance they can offer. I'd just like them to be easier to program.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#32

Post by Symmetry »

tpmx wrote: post #17

Does AMD have a chance here in the short term (say 24 months)?

AMD seems to be focusing on traditional HPC, they've got a ton of 64 bit flops in their recent commercial model. I expect their server GPUs are mostly for chasing supercomputer contracts, which can be pretty lucrative, while they cede model training to NVidia.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#33

Post by wyldfire »

ecshafer wrote: post #16

I was wondering today if we would start to see the reverse of this. Small ASICS or some kind of optimized for LLM Gpu for desktop / or maybe even laptops of mobile. It is evident I think that LLM are here to stay and will be a major part of computing for a while. Getting this local, so we aren't reliant on clouds would be a huge boon for personal computing. Even if its a "worse" experience, being able to load up an L…

In fact, Qualcomm has announced a "Cloud AI" PCIe card designed for inference (as opposed to training & inference) [1, 2]. It's populated with NSPs like the ones in mobile SoCs.

[1] https://www.qualcomm.com/products/technology/processors/clou...

[2] https://github.com/quic/software-kit-for-qualcomm-cloud-ai-1...

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#34

Post by sargun »

What exactly is an SXM5 socket? It sounds like a PCIe competitor, but proprietary to nvidia. Looking at it, it seems specific to nvidia DGX (mother?)boards. Is this just a "better" alternative to PCIe (with power delivery, and such), or fundamentally a new technology?

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#36

Post by int_19h »

pixl97 wrote: post #22
int_19h wrote: post #13

I wonder how soon we'll see something tailored specifically for local applications. Basically just tons of VRAM to be able to load large models, but not bleeding edge perf. And eGPU form factor, ideally.

I'm not a ML scientist my any means, but Perf seems as important as RAM from what I'm reading. Running prompts in internal chain of thought (eating up more TPU time) appears to give much better output.

It's not that perf is not important, but not having enough VRAM means you can't load the model of a given size at all.

I'm not saying they shouldn't bother with RAM at all, mind you. But given some target price, it's a balance thing between compute and RAM, and right now it seems that RAM is the bigger hurdle.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#37

Post by dragontamer »

andrewstuart wrote: post #25

GPUs are going to be weird, underconfigured and overpriced until there is real competition. Whether or not there is real competition depends entirely on whether Intels Arc line of GPUs stays in the market. AMD strangely has decided not to compete. Its newest GPU the 7900 XTX is an extremely powerful card, close to the top of the line Nvidia RTX 4090 in raster performance. If AMD had introduced it with an aggressively…

The root cause is that TSMC raised prices in everyone.

Since Intel GPUs are again TSMC manufactured, you really aren't going to see price improvements unless Intel subsidizes all of this.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#38

Post by 0xbadc0de5 »

sargun wrote: post #34

What exactly is an SXM5 socket? It sounds like a PCIe competitor, but proprietary to nvidia. Looking at it, it seems specific to nvidia DGX (mother?)boards. Is this just a "better" alternative to PCIe (with power delivery, and such), or fundamentally a new technology?

It's one of those /If you have to ask, you can't afford it/ scenarios.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#39

Post by andrewstuart »

JonChesterfield wrote: post #31
andrewstuart wrote: post #25

GPUs are going to be weird, underconfigured and overpriced until there is real competition. Whether or not there is real competition depends entirely on whether Intels Arc line of GPUs stays in the market. AMD strangely has decided not to compete. Its newest GPU the 7900 XTX is an extremely powerful card, close to the top of the line Nvidia RTX 4090 in raster performance. If AMD had introduced it with an aggressively…

GPUs strike me as absurdly cheap given the performance they can offer. I'd just like them to be easier to program.

Depends on the GPU of course but at the top end of the market AUD$3000 / USD$1,600 is not cheap and certainly not absurdly cheap.

Much less powerful GPUs represent better value but the market is ridiculously overpriced at the moment.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#40

Post by andrewstuart »

dragontamer wrote: post #37
andrewstuart wrote: post #25

GPUs are going to be weird, underconfigured and overpriced until there is real competition. Whether or not there is real competition depends entirely on whether Intels Arc line of GPUs stays in the market. AMD strangely has decided not to compete. Its newest GPU the 7900 XTX is an extremely powerful card, close to the top of the line Nvidia RTX 4090 in raster performance. If AMD had introduced it with an aggressively…

The root cause is that TSMC raised prices in everyone. Since Intel GPUs are again TSMC manufactured, you really aren't going to see price improvements unless Intel subsidizes all of this.

>> The root cause is that TSMC raised prices in everyone.

This is not correct.

Post reply on HN