Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

anandtech.com

Post reply on HN
Print view View original

81–90 of 111 posts

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#81

Post by enlyth »

riku_iki wrote: post #80
enlyth wrote: post #65

Earlier quoted context omitted.

Exactly, we're just below that sweet spot right now. For example on 24GB, Llama 30B runs only in 4bit mode and very slowly, but I can imagine a RLHF finetuned 30B or 65B version running in at least 8bit would be actually useful, and you could run it on your own computer easily.

> For example on 24GB, Llama 30B runs only in 4bit mode and very slowly why do you think adding vram, but not cores will make it run faster?..

I've been told the 4 bit quantization slows it down, but don't quote me on this since I was unable to benchmark at 8 bit locally

In any case, you're right it might not be as significant, however, the quality of the output increases with 8/16bit, and running 65B is completely impossible on 24GB

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#82

Post by waboremo »

jacquesm wrote: post #74
Sol- wrote: post #44

Earlier quoted context omitted.

Why is the sentiment here so much that LLMs will somehow be decentralized and run locally at some point? Has the story of the internet so far not been that centralization has pretty much always won?

Because that is pretty much the pendulum swinging in the IT world. Right now it is solidly in 'centralization' territory, hopefully it will go back towards decentralization again in the future. The whole PC revolution was an excellent datapoint for decentralization, now we're back to 'dumb terminals' but as local compute strengthens the things that you need a whole farm of servers for today can probably fit in your p…

Not sure this really tracks. Local compute has always been strengthening as a steady incline. Yet we haven't really experienced any sort of pendulum shift, it's always been centralization territory.

The reasoning seems mostly obvious to me here: people do not care for the effort that decentralization requires. If given the option to run AI off some website to generate all you want, people will gladly do this over using their local hardware due to the setup required.

The unfortunate part is that it takes so much longer to create not for profit tooling that is just as easy to use, especially when the calling to turn that into your for profit business in such a lucrative field is so tempting. Just ask the people who have contributed to Blender for a decade now.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#83

Post by KeplerBoy »

breatheoften wrote: post #41
binarymax wrote: post #5

Earlier quoted context omitted.

Not just cryptobros. A100s are the current top of the line and it’s hard to find them available on AWS and Lambda. Vast.AI has plenty if you trust renting from a stranger. AMD really needs to pick up the pace and make a solid competitive offering in deep learning. They’re slowly getting there but they are at least 2 generations out.

It's crazy to me that no other hardware company has sought to compete for the deep learning training/inference market yet ... The existing ecosystems (cuda, pytorch etc) are all pretty garbage anyway -- aside from the massive number of tutorials it doesn't seem like it would actually be hard to build a vertically integrated competitor ecosystem ... it feels a little like the rise of rails to me -- is a million articl…

How could their moat possibly be deeper?

First of all you need hardware with cutting-edge chips. Chips which can only be supplied by TSMC and Samsung.

Then you need the software ranging all the way from the firmware and driver over something analogous to CUDA with libraries like cuDNN, cuBLAS and many others to integrations into pytorch and tensorflow.

And none of that will come for free, like it came to Nvidia. Nvidia built CUDA and people built their DL frameworks around it in the last decade, but nobody will invest their time into doing the same for a competitor, when they could just do their research on Nvidia hardware instead.

Realistically it's up to AMD or Intel.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#84

Post by enlyth »

koheripbal wrote: post #60
Melatonic wrote: post #58

Earlier quoted context omitted.

That was always why the Titan line was so great - they typically unlocked features in between Quadro and Gaming cards. Sometimes it was subtle (like very good FP32 AND FP16 performance) or adding full 10 bit colour support if you had a Titan only. Now it seems like they have opened up even more of those features to consumer cards (at least the creative ones) with the studio drivers.

Isn't a new Titan RTX 4090 coming out soon?

An alleged photo of an engineering sample was spotted in the wild a while ago, but no one knows if it's actually going to end up being a thing you can buy.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#85

Post by YetAnotherNick »

bick_nyers wrote: post #79
throwaway743 wrote: post #55

Earlier quoted context omitted.

Go with 2x 3090s instead. 4000 series doesn't support SLI, so you're stuck with the max of whatever one card you get.

If I remember correctly the NVLINK adds 100GB/s (where PCIE 4.0 is 64GB/s). Is it really worth getting 3090 performance (roughly half) for that extra bus speed?

PCIE 4.0*16 is 32 GB/s.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#86

Post by jacquesm »

waboremo wrote: post #82
jacquesm wrote: post #74

Earlier quoted context omitted.

Because that is pretty much the pendulum swinging in the IT world. Right now it is solidly in 'centralization' territory, hopefully it will go back towards decentralization again in the future. The whole PC revolution was an excellent datapoint for decentralization, now we're back to 'dumb terminals' but as local compute strengthens the things that you need a whole farm of servers for today can probably fit in your p…

Not sure this really tracks. Local compute has always been strengthening as a steady incline. Yet we haven't really experienced any sort of pendulum shift, it's always been centralization territory. The reasoning seems mostly obvious to me here: people do not care for the effort that decentralization requires. If given the option to run AI off some website to generate all you want, people will gladly do this over usi…

Absolutely not. Computers used to be extremely centralized and the decentralization revolution powered a ton of progress in both software development and hardware development.

You can run many AI applications locally today that would have required a massive investment in hardware not all that long ago. It's just that the bleeding edge is still in that territory. One major optimization avenue is the improvement of the models themselves, they are large because they have large numbers of parameters, but the bulk of those parameters has little to no effect on the model output and there is active research on 'model compression', which has the potential to be able to extract the working bits from a model while discarding the non-working bits without affecting the output and realize massive gains in efficiency (both in power consumption as well as for running the model).

Have a look at the kind of progress that happened in the chess world with the initial huge ML powered engines that are beaten by the kind of program that you can run on your phone nowadays.

https://en.wikipedia.org/wiki/Stockfish_(chess)

I fully expect something similar to happen to language models.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#87

Post by enlyth »

andrewstuart wrote: post #25

GPUs are going to be weird, underconfigured and overpriced until there is real competition. Whether or not there is real competition depends entirely on whether Intels Arc line of GPUs stays in the market. AMD strangely has decided not to compete. Its newest GPU the 7900 XTX is an extremely powerful card, close to the top of the line Nvidia RTX 4090 in raster performance. If AMD had introduced it with an aggressively…

I suspect that the lack of CUDA is a dealbreaker for too many people when it comes to AMD, with the recent explosion in machine learning.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#88

Post by waboremo »

jacquesm wrote: post #86
waboremo wrote: post #82

Earlier quoted context omitted.

Not sure this really tracks. Local compute has always been strengthening as a steady incline. Yet we haven't really experienced any sort of pendulum shift, it's always been centralization territory. The reasoning seems mostly obvious to me here: people do not care for the effort that decentralization requires. If given the option to run AI off some website to generate all you want, people will gladly do this over usi…

Absolutely not. Computers used to be extremely centralized and the decentralization revolution powered a ton of progress in both software development and hardware development. You can run many AI applications locally today that would have required a massive investment in hardware not all that long ago. It's just that the bleeding edge is still in that territory. One major optimization avenue is the improvement of the…

The bleeding edge will always be in that territory. It still requires a massive investment today to run AI applications locally to produce anywhere near as good results. People are spending upwards of $2000 for a GPU just to get decent results when it comes to image generation, many forgoing this entirely and just giving Google a monthly fee to use their hardware.

Which is the point, decentralization will always be playing catch up here unless something really interesting happens. It has absolutely nothing to do with local compute power, that has always been on an incline. We just get fed scraps down the line.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#89

Post by rcme »

KeplerBoy wrote: post #83
breatheoften wrote: post #41

Earlier quoted context omitted.

It's crazy to me that no other hardware company has sought to compete for the deep learning training/inference market yet ... The existing ecosystems (cuda, pytorch etc) are all pretty garbage anyway -- aside from the massive number of tutorials it doesn't seem like it would actually be hard to build a vertically integrated competitor ecosystem ... it feels a little like the rise of rails to me -- is a million articl…

How could their moat possibly be deeper? First of all you need hardware with cutting-edge chips. Chips which can only be supplied by TSMC and Samsung. Then you need the software ranging all the way from the firmware and driver over something analogous to CUDA with libraries like cuDNN, cuBLAS and many others to integrations into pytorch and tensorflow. And none of that will come for free, like it came to Nvidia. Nvid…

There will probably be Chinese options as well. China has an incentive to provide a domestic competitor due to deteriorating relations with the U.S.

Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models

#90

Post by cavisne »

Sol- wrote: post #44
ecshafer wrote: post #16

I was wondering today if we would start to see the reverse of this. Small ASICS or some kind of optimized for LLM Gpu for desktop / or maybe even laptops of mobile. It is evident I think that LLM are here to stay and will be a major part of computing for a while. Getting this local, so we aren't reliant on clouds would be a huge boon for personal computing. Even if its a "worse" experience, being able to load up an L…

Why is the sentiment here so much that LLMs will somehow be decentralized and run locally at some point? Has the story of the internet so far not been that centralization has pretty much always won?

Nvidia's business model encourages this for starters. They charge a huge markup for their datacenter GPU's through some clever licensing restrictions. So it is cheaper per FLOP to run inference on a personal device.

Centralization of compute has not always won (even if that compute is mostly controlled by a single company). The failure of cloud gaming vs consoles, and the success of Apple (which is very centralized but pushes a lot of ML compute out to the edge) for example.

Post reply on HN