Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

anandtech.com

Post reply on HN
Print view View original

11–20 of 111 posts

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#11

Post by rerx »

neilmovva wrote: post #2

A bit underwhelming - H100 was announced at GTC 2022, and represented a huge stride over A100. But a year later, H100 is still not generally available at any public cloud I can find, and I haven't yet seen ML researchers reporting any use of H100. The new "NVL" variant adds ~20% more memory per GPU by enabling the sixth HBM stack (previously only five out of six were used). Additionally, GPUs now come in pairs with 6…

You can also join a pair of regular PCIe H100 GPUs with an NVLink bridge. So that topology is not so new either.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#16

Post by ecshafer »

I was wondering today if we would start to see the reverse of this. Small ASICS or some kind of optimized for LLM Gpu for desktop / or maybe even laptops of mobile. It is evident I think that LLM are here to stay and will be a major part of computing for a while. Getting this local, so we aren't reliant on clouds would be a huge boon for personal computing. Even if its a "worse" experience, being able to load up an LLM into our computer, tell it to only look at this directory and help out would be cool.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#18

Post by rerx »

metadat wrote: post #12

How is this card (which is really two physical cards occupying 2 PCIe slots) exposed to the OS? Does it show up as a single /dev/gfx0 device, or is the unification a driver trick?

The two cards show as two distinct GPUs to the host, connected via NVLink. Unification / load balancing happens via software.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#19

Post by andy81 »

TylerE wrote: post #4
binarymax wrote: post #3

Earlier quoted context omitted.

It is interesting that hopper isn’t widely available yet. I have seen some benchmarks from academia but nothing in the private sector. I wonder if they thought they were moving too fast and wanted to milk amphere/ada as long as possible. Not having any competition whatsoever means Nvidia can release what they like when they like.

Why bother when you can get cryptobros paying way over MSRP for 3090s?

GPU mining died last year.

There's so little liquidity post-merge that it's only worth mining as a way to launder stolen electricity.

The bitcoin people still waste raw materials, and prices are relatively sticky with so few suppliers and a backlog of demand, but we've already seen prices drop heavily since then.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#20

Post by TylerE »

andy81 wrote: post #19
TylerE wrote: post #4

Earlier quoted context omitted.

Why bother when you can get cryptobros paying way over MSRP for 3090s?

GPU mining died last year. There's so little liquidity post-merge that it's only worth mining as a way to launder stolen electricity. The bitcoin people still waste raw materials, and prices are relatively sticky with so few suppliers and a backlog of demand, but we've already seen prices drop heavily since then.

Right, that's why NVidia is acutally trying again. The money printer has run out of ink.
Post reply on HN