Earlier quoted context omitted.
You think? It’s double 48 GB (per card) so why wouldn’t it be in the $20k range?
Machine learning is so hyped right now (with good reason) so customers are price insensitive.
I guess we'll see.
Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
71–80 of 111 posts
Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
#72I was wondering today if we would start to see the reverse of this. Small ASICS or some kind of optimized for LLM Gpu for desktop / or maybe even laptops of mobile. It is evident I think that LLM are here to stay and will be a major part of computing for a while. Getting this local, so we aren't reliant on clouds would be a huge boon for personal computing. Even if its a "worse" experience, being able to load up an L…
Apple, Intel, AMD, Qualcomm, Samsung, etc. already have "neural engines" in their SoCs. These engines continue to evolve to better support common types of models.
Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
#73How is this card (which is really two physical cards occupying 2 PCIe slots) exposed to the OS? Does it show up as a single /dev/gfx0 device, or is the unification a driver trick?
The two cards show as two distinct GPUs to the host, connected via NVLink. Unification / load balancing happens via software.
Kinda depressing if you consider how they removed NVLink in the 4090, stating the following reason:
> “The reason we took [NVLink] off is that we need I/O for other things, so we’re using that area to cram in as many AI processors as possible,” Jen-Hsun Huang explained of the reason for axing NVLink.[0]
"NVLink is bad for your games and AI, trust me bro."
But then this card, actually aimed at ML applications, uses it.
0. https://www.techgoing.com/nvidia-rtx-4090-no-longer-supports...
Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
#74I was wondering today if we would start to see the reverse of this. Small ASICS or some kind of optimized for LLM Gpu for desktop / or maybe even laptops of mobile. It is evident I think that LLM are here to stay and will be a major part of computing for a while. Getting this local, so we aren't reliant on clouds would be a huge boon for personal computing. Even if its a "worse" experience, being able to load up an L…
Why is the sentiment here so much that LLMs will somehow be decentralized and run locally at some point? Has the story of the internet so far not been that centralization has pretty much always won?
Because that is pretty much the pendulum swinging in the IT world. Right now it is solidly in 'centralization' territory, hopefully it will go back towards decentralization again in the future. The whole PC revolution was an excellent datapoint for decentralization, now we're back to 'dumb terminals' but as local compute strengthens the things that you need a whole farm of servers for today can probably fit in your pocket tomorrow, or at the latest in a few years.
Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
#75Earlier quoted context omitted.
We're NOT business users, we just want to run our own LLM at home. Given the size of LLMs, this should be possible with just a little bit of extra VRAM.
Exactly, we're just below that sweet spot right now. For example on 24GB, Llama 30B runs only in 4bit mode and very slowly, but I can imagine a RLHF finetuned 30B or 65B version running in at least 8bit would be actually useful, and you could run it on your own computer easily.
Do you know where the cutoff is? Does 32GB VRAM give us 30B int8 with/without a RLHF layer? I don't think 5090 is going to go straight to 48GB, I'm thinking either 32 or 40GB (if not 24GB).
Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
#76I was wondering today if we would start to see the reverse of this. Small ASICS or some kind of optimized for LLM Gpu for desktop / or maybe even laptops of mobile. It is evident I think that LLM are here to stay and will be a major part of computing for a while. Getting this local, so we aren't reliant on clouds would be a huge boon for personal computing. Even if its a "worse" experience, being able to load up an L…
Why is the sentiment here so much that LLMs will somehow be decentralized and run locally at some point? Has the story of the internet so far not been that centralization has pretty much always won?
I think the sentiment is both. There will be advanced centralized LLM's and people want the option to have a personal one (or two). There needn't be a single solution.
Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
#77Earlier quoted context omitted.
Nvidia don't want consumers using consumer GPUs for business. If you are a business user then you must pay Nvidia gargantuan amounts of money. This is the outcome of a market leader with no real competition - you pay much more for lower power than the consumer GPUs and you are forced into ujsing their business GPUs through software license restrictions on the drivers.
We're NOT business users, we just want to run our own LLM at home. Given the size of LLMs, this should be possible with just a little bit of extra VRAM.
I don't think you understand though, they don't WANT you. They WANT the version of you who makes $150k+ a year and will splurge $5k on a Quadro.
If they had trouble selling stock we would see this niche market get catered to.
Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
#78I'm super duper curious if there are ways to glob together VRAM between consumer-grade hardware to make this whole market more accessible to the common hacker?
I remember reading about a guy who soldered 2GB VRAM modules on his 3060 12GB (replacing the 1GB modules) and was able to attain 24GB on that card. Or something along those lines.
Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
#79Earlier quoted context omitted.
Use 4 consumer grade 4090 then. It would be much cheaper and better in almost every aspect. Also even with this, forget about training foundational models. Meta spent 82k GPU hours on the smallest llama and 1M hours on largest.
Go with 2x 3090s instead. 4000 series doesn't support SLI, so you're stuck with the max of whatever one card you get.
If I remember correctly the NVLINK adds 100GB/s (where PCIE 4.0 is 64GB/s). Is it really worth getting 3090 performance (roughly half) for that extra bus speed?
Re: Nvidia Announces H100 NVL – Max Memory Server Card for Large Language Models
#80Earlier quoted context omitted.
We're NOT business users, we just want to run our own LLM at home. Given the size of LLMs, this should be possible with just a little bit of extra VRAM.
Exactly, we're just below that sweet spot right now. For example on 24GB, Llama 30B runs only in 4bit mode and very slowly, but I can imagine a RLHF finetuned 30B or 65B version running in at least 8bit would be actually useful, and you could run it on your own computer easily.
> For example on 24GB, Llama 30B runs only in 4bit mode and very slowly
why do you think adding vram, but not cores will make it run faster?..