Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody knows what a used GPU cluster is worth

ciphertalk.substack.com

Post reply on HN
Print view View original

141–150 of 285 posts

Re: Nobody knows what a used GPU cluster is worth

#141

Post by smalltorch »

everforward wrote: post #135
jtolmar wrote: post #31

Earlier quoted context omitted.

I think there's a market for a home AI server that can run an LLM or video gen model behind a web frontend. Not literally a raspberry pi strapped to an H100, but something with lopsided enough specs that people joke it is. And precisely because it's such a huge headache to do yourself, I think a small company could make a nice business wrapping up used datacenter cards in that sort of server.

I'm doubtful they can, based on current inference prices. An H100 draws about 50W while idling, which is ~$8/month at average US electricity prices. They also draw ~400W while active. The electricity prices are relevant because if you paid $0 for your H100 and didn't use it a single time, you could buy millions of tokens in inference just on the electricity it draws while idling. If you can't keep that thing saturate…

Most houses have a 200amp panel and extra space for larger breakers.

A electrician can plop in a electric car charger for instance, that is a 40-50 amp circuit.

Re: Nobody knows what a used GPU cluster is worth

#143

Post by noosphr »

dantillberg wrote: post #109
azeemba wrote: post #106

Earlier quoted context omitted.

Why weren't crypto gpus running at 100%?

Most crypto mining on GPUs would use 100% of memory bandwidth, but only a fraction of the compute available. This is a consequence of ASIC resistance of their mining algorithms -- custom silicon can only offer a modest benefit over GPUs if the hard part is memory bandwidth.

So does llm inference. You're lucky if you hit 40% of the advertised flops.

Re: Nobody knows what a used GPU cluster is worth

#144

Post by andix »

nikanj wrote: post #96

Essentially nothing, fractions of a penny on a dollar, because players who could afford paying real money won't risk the crusty old hardware - so you're limited to buyers who still need a massive cluster but don't have AI infinite money glitch enabled

Are failing GPUs really a big issue? I don't know how they behave when they die, but if it can be detected quickly, the affected nodes can just be removed from the pool.

Re: Nobody knows what a used GPU cluster is worth

#145

Post by gessha »

rtkwe wrote: post #14
echelon wrote: post #7

Earlier quoted context omitted.

Wouldn't that be crazy - a hobbyist market for H100s? Maybe someone could start a business buying up and rehousing these.

They're pretty specific to the datacenter use case with no outputs and they need to be cooled externally, principally through the very loud high speed fans used in data centers. I suppose you could strap a fan to one and put it in a normal case or maybe make a dedicated after market cooler (like the water blocks made for water cooling cases).

If you check eBay for V100 SXM variant, you will often see them sold in combination with a cold plate, together with a PCIe carrier board. There’s definitely a market for them.

Re: Nobody knows what a used GPU cluster is worth

#146

Post by Chaosvex »

christina97 wrote: post #128
lightbendover wrote: post #84

Earlier quoted context omitted.

Who is buying these at 70% of new pricing given the sky high likelihood of them being shot? Maybe it's safe to buy from small labs that went under quickly, but I can't imagine a cluster that has been operating near its thermal limits for a couple years fetching that kind of resale.

Why would they be shot? Unlike the consumer cards that are basically factory overclocked to look good on benchmarks, the datacenter GPUs are designed to run at full tilt 24/7 and survive for years.

Given the decades of consumer and enterprise GPUs often being identical or near identical hardware, it'd be interesting to see if there's any evidence of this actually being true.

Re: Nobody knows what a used GPU cluster is worth

#147

Post by mickael-kerjean »

tracker1 wrote: post #36
fancyfredbot wrote: post #28

The relatively slow depreciation of GPU value is an artifact of supply constraints. If you run fp4 inference and could choose freely between Hopper and a Rubin, the performance per watt would make the Hopper unattractive even if you paid zero for the hardware and only for the power. You can't get the Rubin, or even the Blackwell, so you will pay for the H100 but this won't last if fabs ramp up capacity.

Not to mention, physical limits to lithography are slowing down significantly... so tech will continue to evolve more slowly... it'll never be the jump from 1080-1990 again, for example, even though 1990-2000 was pretty close, 2000-2010 much slower and since 2010 slower still. What's as or more weird is how much hardware is backordered, and how much live hardware is allocated, but waiting on facilities for operation.…

The infamous attention is all you need paper has this: "The Transformer ... reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs.", the p100 is now quite a bit under 100$ and that's from a time where nvidia was valued 100x less than it is today. Lets wait and see

Re: Nobody knows what a used GPU cluster is worth

#148

Post by bigbuppo »

christina97 wrote: post #128
lightbendover wrote: post #84

Earlier quoted context omitted.

Who is buying these at 70% of new pricing given the sky high likelihood of them being shot? Maybe it's safe to buy from small labs that went under quickly, but I can't imagine a cluster that has been operating near its thermal limits for a couple years fetching that kind of resale.

Why would they be shot? Unlike the consumer cards that are basically factory overclocked to look good on benchmarks, the datacenter GPUs are designed to run at full tilt 24/7 and survive for years.

According to the article these cards have a 9% annual failure rate.

Re: Nobody knows what a used GPU cluster is worth

#149

Post by AYBABTME »

I don't get how this is unique to GPU clusters. As a general rule, underwriters are not qualified to operate and maintain the assets they underwrite loans for. That's why houses, cars, equipment, ... go at auction at a fraction of their value. And why lenders have insurance.

Re: Nobody knows what a used GPU cluster is worth

#150

Post by Der_Einzige »

latchkey wrote: post #15

I like Meg a lot a human, but Meg is all doom and gloom. Every single post she makes is about how GPUs fail [0] and now she's onto how financing is a big thing just waiting to crash and "nobody knows what a used GPU cluster is worth"... Actually, we do, people offer them to me all the time. A used box of MI300x is $257k. "There is no GPU futures market"... actually there are a few of them that people have pitched to…

It’s always the people who know the least about GPU economics who want to doom post. Until I can reliably get on demand A100s in large quantities for less than 2.00 an hour there’s no AI bubble.

Most of the people who shit talk GPUs don’t even understand why it relatively speaking doesn’t matter that much if I’m on ampere or Vera Rubin for full precision models. These same people are wildly misinforming investors.

To quote the ADATA ceo, “don’t talk about an AI bubble until 2040”

Post reply on HN