[flagged]
Nobody knows what a used GPU cluster is worth
211–220 of 285 posts
Re: Nobody knows what a used GPU cluster is worth
#212Earlier quoted context omitted.
To advertise the maximum possible performance numbers, CPUs and GPUs massively shoot up the power draw for that final few percentage points. If you are worried about actual calculations/$, cutting back on the power will save your energy bill for practically no loss. When I tested "ECO" mode on my AMD CPU, performance was ~97% of regular mode, and temperatures dropped 5-10C. I think ECO mode drops TDP from something l…
Last point should also be for llm. Not just crypto
Not true. A B200 "only" draws 1 kW. At 16¢ USD / kWh that's $1400 / year. When the hardware costs multiple tens of thousands you might be looking at a yearly electricity bill of ~3% of that. And that's before accounting for the datacenter itself.
The difference is that much of the hardware crypto was running on was dirt cheap in comparison. Add to that a lot of crypto was being run on discount gear in cobbled together setups. Some people were even stealing electricity which often involved installing large clusters of relatively cheap devices in places without proper cooling.
Re: Nobody knows what a used GPU cluster is worth
#213Earlier quoted context omitted.
As noted in my other comment you can get obsolete GPUs (P100s, BC250s) with lots of RAM on AliExpress now. It hasn't proven revolutionary.
is 16Gb "lots of RAM" in an LLM world? How many of these would you need to stack on a motherboard to inference a decent size model?
At least 4, probably 6+.
Re: Nobody knows what a used GPU cluster is worth
#214Earlier quoted context omitted.
> and not purchase any more. why would anyone sign such a contract?
I don't know if this was IT shrugging me off or if it was something real but some IT person at this big ISP I worked at told me that they cannot just buy an SSD — my windows box at work was running off of a hard disk in 2019 — and that there was some contract that said any computer hardware we bought had to be through HP or something like that and it takes many months it something like that.
> I don't know if this was IT shrugging me off or if it was something real but some IT person at this big ISP I worked at told me that they cannot just buy an SSD — my windows box at work was running off of a hard disk in 2019 — and that there was some contract that said any computer hardware we bought had to be through HP or something like that and it takes many months it something like that.
Plausible - the hardware might be leased, and so you have no right to modify it. You have to pay them to modify it.
Same as if you leased a car, you cannot do the services yourself, you have to pay them (and an approved agent of theirs) to do it.
Re: Nobody knows what a used GPU cluster is worth
#215Earlier quoted context omitted.
We've not had cheap GPUs with this much VRAM before, though. Might be an interesting change, though I also doubt it personally.
As noted in my other comment you can get obsolete GPUs (P100s, BC250s) with lots of RAM on AliExpress now. It hasn't proven revolutionary.
Those don't have a lot of RAM though. Not like these 40, 80GB ones.
Re: Nobody knows what a used GPU cluster is worth
#216I half expect Nvidia to have buyback contacts like Ferrari with the larger customers to prevent a price crash when they all upgrade and to keep them scarce. I hope not though, perhaps I can pick up a H100 in a few years if they get sold on the open market.
The only reason a datacenter would ditch their H100 is if it becomes uneconomical to run them, with newer silicon providing much more power efficiency. When that happens they'll look like a used V100 looks today: horribly inefficient, lacking modern data types and engines, requiring screaming server fans with weird adapters to not melt, way beyond end of life in terms of cuda support. Almost completely damn useless u…
Honestly, the primary reason you would ditch a H100 is because the ML models have moved onto a different data type, e.g. nobody uses fp8 anymore and everyone jumped onto fp4 or some custom block float thing.
Re: Nobody knows what a used GPU cluster is worth
#217Earlier quoted context omitted.
My workstations are usually tower servers, which are the same design as their 4 and 5u rack counterparts. Until the thermal management kicks in, they sound like jet planes. When the thermal management starts and assesses the required cooling, it’ll throttle down the fans to reasonable levels. That is, until the moment you push the machine to its limits. When then happens, you might get back to the same levels of the…
Yes, but the line between "reasonably quiet" and 100% isn't a step function. If these do end up in Home Labs, it's going to be a server rack in the basement and not the server rack at the other end of the room.
> If these do end up in Home Labs, it's going to be a server rack in the basement and not the server rack at the other end of the room.
If they are too loud, yes. In my part of the world basements are not common, but my shed would be a sensible place for one.
Re: Nobody knows what a used GPU cluster is worth
#218Earlier quoted context omitted.
According to the article these cards have a 9% annual failure rate.
Most hardware has an exponential (memoryless) failure distribution in practice. The bathtub curve is a myth.
Where is this exponential thing coming from?
Why would hardware failure raise exponential with linear scale? Doesn't make sense
Re: Nobody knows what a used GPU cluster is worth
#219Earlier quoted context omitted.
Who is buying these at 70% of new pricing given the sky high likelihood of them being shot? Maybe it's safe to buy from small labs that went under quickly, but I can't imagine a cluster that has been operating near its thermal limits for a couple years fetching that kind of resale.
Honest question. Does silicon wear out due to high temperatures, or is it more like lightbulbs where it wears out from thermal cycles? Does it really wear out at all?
For a long time it was assumed it wouldn't wear out and tbh if you look how long it takes, how rare it is, its not a real issue...
besides what intel did with Raptor Lake. This series had massive issues with oxidiation. Was the first time ever i became aware of this issue on scale.
Nontheless there are papers out there that silicon can degenerate and does.
Re: Nobody knows what a used GPU cluster is worth
#220[flagged]