Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

anandtech.com

Post reply on HN
Print view View original

41–50 of 111 posts

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#41

Post by breatheoften »

binarymax wrote: post #5
TylerE wrote: post #4

Earlier quoted context omitted.

Why bother when you can get cryptobros paying way over MSRP for 3090s?

Not just cryptobros. A100s are the current top of the line and it’s hard to find them available on AWS and Lambda. Vast.AI has plenty if you trust renting from a stranger. AMD really needs to pick up the pace and make a solid competitive offering in deep learning. They’re slowly getting there but they are at least 2 generations out.

It's crazy to me that no other hardware company has sought to compete for the deep learning training/inference market yet ...

The existing ecosystems (cuda, pytorch etc) are all pretty garbage anyway -- aside from the massive number of tutorials it doesn't seem like it would actually be hard to build a vertically integrated competitor ecosystem ... it feels a little like the rise of rails to me -- is a million articles about how to build a blog engine really that deep a moat ..?

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#43

Post by dragontamer »

andrewstuart wrote: post #40
dragontamer wrote: post #37

Earlier quoted context omitted.

The root cause is that TSMC raised prices in everyone. Since Intel GPUs are again TSMC manufactured, you really aren't going to see price improvements unless Intel subsidizes all of this.

>> The root cause is that TSMC raised prices in everyone. This is not correct.

https://www.tomshardware.com/news/tsmc-ups-chip-production-p...

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#44

Post by Sol- »

ecshafer wrote: post #16

I was wondering today if we would start to see the reverse of this. Small ASICS or some kind of optimized for LLM Gpu for desktop / or maybe even laptops of mobile. It is evident I think that LLM are here to stay and will be a major part of computing for a while. Getting this local, so we aren't reliant on clouds would be a huge boon for personal computing. Even if its a "worse" experience, being able to load up an L…

Why is the sentiment here so much that LLMs will somehow be decentralized and run locally at some point? Has the story of the internet so far not been that centralization has pretty much always won?

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#45

Post by solarmist »

ipsum2 wrote: post #42

I would sell a kidney for one of these. It's basically impossible to train language models on a consumer 24GB card. The jump up is the A6000 ADA, at 48GB for $8,000. This one will probably be priced somewhere in the $100k+ range.

You think? It’s double 48 GB (per card) so why wouldn’t it be in the $20k range?

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#47

Post by __anon-2023__ »

neilmovva wrote: post #2

A bit underwhelming - H100 was announced at GTC 2022, and represented a huge stride over A100. But a year later, H100 is still not generally available at any public cloud I can find, and I haven't yet seen ML researchers reporting any use of H100. The new "NVL" variant adds ~20% more memory per GPU by enabling the sixth HBM stack (previously only five out of six were used). Additionally, GPUs now come in pairs with 6…

Yes, I was expecting a RAM-doubled edition of the H100, this is just a higher-binned version of the same part.

I got an email from vultr, saying that they're "officially taking reservations for the NVIDIA HGX H100", so I guess all public clouds are going to get those soon.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#48

Post by YetAnotherNick »

brucethemoose2 wrote: post #7

The really interesting upcoming LLM products are from AMD and Intel... with catches. - The Intel Falcon Shores XPU is basically a big GPU that can use DDR5 DIMMS directly, hence it can fit absolutely enormous models into a single pool. But it has been delayed to 2025 :/ - AMD have not mentioned anything about the (not delayed) MI300 supporting DIMMs. If it doesn't, its capped to 128GB, and its being marketed as an HP…

> DDR5 DIMMS directly

That's the problem. Good DDR5 RAM's memory speed is <100GB/s, while nvidia could has up to 2TB/s, and still the bottleneck lies on memory speed for most applications.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#49

Post by frankchn »

int_19h wrote: post #13

I wonder how soon we'll see something tailored specifically for local applications. Basically just tons of VRAM to be able to load large models, but not bleeding edge perf. And eGPU form factor, ideally.

The Apple M-series CPUs with unified RAM is interesting in this regard. You can get an 16-inch MBP with an M2 Max 96GB of RAM for $4300 today, and I expect the M2 Ultra go to 192GB.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#50

Post by YetAnotherNick »

ipsum2 wrote: post #42

I would sell a kidney for one of these. It's basically impossible to train language models on a consumer 24GB card. The jump up is the A6000 ADA, at 48GB for $8,000. This one will probably be priced somewhere in the $100k+ range.

Use 4 consumer grade 4090 then. It would be much cheaper and better in almost every aspect. Also even with this, forget about training foundational models. Meta spent 82k GPU hours on the smallest llama and 1M hours on largest.
Post reply on HN