151–160 of 285 posts
I don't get how this is unique to GPU clusters. As a general rule, underwriters are not qualified to operate and maintain the assets they underwrite loans for. That's why houses, cars, equipment, ... go at auction at a fraction of their value. And why lenders have insurance.
Auction prices account for that risk, you don't get to do all of the verification you do for normal purchases.
Earlier quoted context omitted.
Why would they be shot? Unlike the consumer cards that are basically factory overclocked to look good on benchmarks, the datacenter GPUs are designed to run at full tilt 24/7 and survive for years.
According to the article these cards have a 9% annual failure rate.
From an annualized number on the llama 3 training report. would be interesting to see if we have a better idea given that we're already on rubin.
Earlier quoted context omitted.
Most crypto mining on GPUs would use 100% of memory bandwidth, but only a fraction of the compute available. This is a consequence of ASIC resistance of their mining algorithms -- custom silicon can only offer a modest benefit over GPUs if the hard part is memory bandwidth.
Why is that true? Can’t you just make more stuff parallel and shrink the ASIC chips accordingly?
[flagged]
That still doesn't tell you what GPUs will be worth in 5 years, because it depends upon what inference demand is like and what the alternatives are. What will an hour of NVL72 be worth in 2029? (And other things, that we know partially but not fully-- like what failure rate for current generation parts will be under this loading). So we have big uncertainties about the revenue, moderate uncertainty about the proporti…
Earlier quoted context omitted.
Why would they be shot? Unlike the consumer cards that are basically factory overclocked to look good on benchmarks, the datacenter GPUs are designed to run at full tilt 24/7 and survive for years.
According to the article these cards have a 9% annual failure rate.
I half expect Nvidia to have buyback contacts like Ferrari with the larger customers to prevent a price crash when they all upgrade and to keep them scarce. I hope not though, perhaps I can pick up a H100 in a few years if they get sold on the open market.
There's going to be a golden age of GPGPU compute in the next few years once A100/H100 are fully obsolete for running frontier models efficiently and the price plummets It will be perfect for stuff like GPU-accelerated query engines, "classical ML" and every other CPU-based workload that could conceivably be offloaded to GPU
You can get used 16GB P100s on AliExpress for ~$100 if you want obsolete GPUs. Allegedly new AMD BC 250s are only slightly more.
I've looked at this some but I already have a GTX1070 which is only supported upto CUDA 11.9.
That's precludes some interesting modern optimizations out of the box. I've spend a lot of LLM tokens backporting some things, but I'm really not sure the hassle is worth it.
New hardware is just better. I think in maybe 5 years when supply and demand are back in equilibrium we are going to have some killer technology for decent prices, and 15yo H100s won't look attractive.
Earlier quoted context omitted.
GPGPU? General Purpose GPU? If embarrassingly parallel CPU algorithms weren't offloaded to the GPU previously, why would the A100/H100 price drop make a difference? We had cheap GPU in the past and we still left plenty of performance on the table with CPU programs because they were easier to build. Is the idea that previously maintaining GPU programs was expensive whereas now AI makes it cheap? If so, I could buy tha…
We've not had cheap GPUs with this much VRAM before, though. Might be an interesting change, though I also doubt it personally.
Earlier quoted context omitted.
They're pretty specific to the datacenter use case with no outputs and they need to be cooled externally, principally through the very loud high speed fans used in data centers. I suppose you could strap a fan to one and put it in a normal case or maybe make a dedicated after market cooler (like the water blocks made for water cooling cases).
I think there's a market for a home AI server that can run an LLM or video gen model behind a web frontend. Not literally a raspberry pi strapped to an H100, but something with lopsided enough specs that people joke it is. And precisely because it's such a huge headache to do yourself, I think a small company could make a nice business wrapping up used datacenter cards in that sort of server.
Models are too big now
I half expect Nvidia to have buyback contacts like Ferrari with the larger customers to prevent a price crash when they all upgrade and to keep them scarce. I hope not though, perhaps I can pick up a H100 in a few years if they get sold on the open market.
That should be illegal. Sounds like a very fraudulent business tactic.
I don't think they need some special protection against this kind of contract.
Earlier quoted context omitted.
That still doesn't tell you what GPUs will be worth in 5 years, because it depends upon what inference demand is like and what the alternatives are. What will an hour of NVL72 be worth in 2029? (And other things, that we know partially but not fully-- like what failure rate for current generation parts will be under this loading). So we have big uncertainties about the revenue, moderate uncertainty about the proporti…
perhaps! but you can look at what a five year old GPU (Nvidia A100 80gb) sells for today. it's not cheap! I wish it was!
We are increasing production of GPUs and RAM a lot.