Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Was my $48K GPU server worth it?

rosmine.ai

Post reply on HN
Print view View original

311–320 of 480 posts

Re: Was my $48K GPU server worth it?

#311

Post by gbgarbeb »

digitaltrees wrote: post #273
gbgarbeb wrote: post #262

Earlier quoted context omitted.

The ones sold for $25k from established sellers are legit. Filter by "sold." The 0-reputation account in Spain selling an M3U 512GB for $4200 is 100% fraud.

oh young grasshopper, I see you dont know that money launderers love the ebay hype cycle. Its REALLY common on high dollar hot items to have phantom transactions where parties are on both sides of the transaction to clean illicit money. The high price tag and high volume amount of transactions hides the illicit signal. I have tried to buy a few of these mac studios only to have the transaction cancelled because I was…

Bizarre. They don't care that eBay takes 14%?

Re: Was my $48K GPU server worth it?

#312

Post by IshKebab »

orsorna wrote: post #104
throwatdem12311 wrote: post #51

Earlier quoted context omitted.

Not even a single mention of gaming. No wonder gamers hate AI bros.

I would probably hate someone if they were buying the same hardware as me but doing something actually useful with it. Any game worth playing doesn't require high specs anyway. There is such a large catalog of old games.

I don't think you can dismiss gaming as not "actually useful".

Re: Was my $48K GPU server worth it?

#313

Post by adastra22 »

girvo wrote: post #291
andoando wrote: post #256

Earlier quoted context omitted.

You could just rent a bare metal server with those specs

Yes I could, but that is annoying because of spot pricing and having my instance shut down, and it has fluctuating prices It’s also annoying because then I need to make sure my little “lab” setup is well automated, and I’m lazy :) Also, I literally said “ It's not financially a good idea” so I’m confused why you think I don’t know that.

Spot pricing and instance availability don’t apply to on metal hosting. You’d have your own machine dedicated to your own use only, at a locked in price.

Re: Was my $48K GPU server worth it?

#314

Post by gbgarbeb »

joefourier wrote: post #60
bigyabai wrote: post #48

Earlier quoted context omitted.

> Why didn't you take into account [...] the fact that a laptop can still hold a decent % of its resale value, and is useful for many other tasks than running an LLM? Because that wasn't what they claimed to research? >> for inference it's definitely not worth it. It's entirely fine if you enjoy local LLMs on your computer, there are people doing horribly inefficient inference on smartphones now. But for pure inferen…

Who is going to buy a $4299 M5 Max MBP with 64GB of RAM just to run Gemma 4 31b? Firstly you don't need 64GB for that model. Secondly if you want a machine that sits in the corner and does nothing but LLM inference, you don't buy a MacBook Pro, you buy some GPUs which are going to cost you a fraction of that (~$1k for ~64GB of VRAM is possible). The people buying Apple Silicon for inference general aim for the Mac St…

24GB GPUs are $700-2500. Please show me the 64GB GPU for $1k.

Re: Was my $48K GPU server worth it?

#316

Post by ai_fry_ur_brain »

adastra22 wrote: post #310
ai_fry_ur_brain wrote: post #308

Earlier quoted context omitted.

Its basic math, go calculate max sessions for a certain tps on any hardware. Session# * tps * 86400 (secs in a day) * 30 days. You'll realize real quick its not profitible. You cant just say things you don't like to hear are unsubstantiated without verifying. Not to mention, subscriptions.. $2mm in GPUs being given out for 5 hrs a day at a cost of $200 a month. I could easily say that everyone who says its profitible…

You got numbers? Because it seems perfectly possible to me. OpenAI and Anthropic’s marginal cost for inference is certainly far less than their API pricing.

How can you say that with such certainty? You have no idea what it costs to run a 10T parameter model at extremely high concurrency.

These 1T param models running at <$3.00 per 1mm are certainly not profitable.

Re: Was my $48K GPU server worth it?

#317

Post by IshKebab »

mrandish wrote: post #108
LandoCalrissian wrote: post #95

'If you google “plugging a PC into multiple outlets”, you get lots of warnings that if you even consider such a setup you will instantly burst into flames. So I hired a professional PC builder make sure it was safe.' Not really sure how that makes it safe but OK!

I guess it was supposed to be a humorous aside, but it wasn't actually helpful because the relevant issue is when you pull more total amps from a single circuit than it's fused for (usually 15 or 20 amps in U.S. residences). The failure mode is usually tripping the circuit breaker. That issue can often be addressed fairly easily by splitting the power draw between two adjacent circuits. You can have an electrician do…

I used to work for a company where we made test rigs and their safety guys were strictly against having a single machine with multiple power inputs. It wasn't about power draw. Once you have two plugs:

1. You no longer have the nice property that unplugging it guarantees (more or less) that it isn't electrified.

2. You open up the possibility of mains voltage from one plug appearing on the unplugged prongs of the other plug.

3. It possibly messes with RCDs, depending on what you do exactly.

Although in this case it's probably fine because he's just plugging totally separate power supplies in and they're already fully enclosed.

Re: Was my $48K GPU server worth it?

#318

Post by callmeal »

adastra22 wrote: post #310
ai_fry_ur_brain wrote: post #308

Earlier quoted context omitted.

Its basic math, go calculate max sessions for a certain tps on any hardware. Session# * tps * 86400 (secs in a day) * 30 days. You'll realize real quick its not profitible. You cant just say things you don't like to hear are unsubstantiated without verifying. Not to mention, subscriptions.. $2mm in GPUs being given out for 5 hrs a day at a cost of $200 a month. I could easily say that everyone who says its profitible…

You got numbers? Because it seems perfectly possible to me. OpenAI and Anthropic’s marginal cost for inference is certainly far less than their API pricing.

See: https://www.wheresyoured.at/ He's been "numbering" for quite a while now.

Re: Was my $48K GPU server worth it?

#319

Post by bredren »

adrianN wrote: post #257
miki123211 wrote: post #227

Earlier quoted context omitted.

This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, a…

Historically it was not uncommon for beds to be rented out to multiple people.

The word for this type of boarding is “flophouse.”

This is the type of place one might be “waiting for the other shoe to drop.” Which carries a variety of potential meanings in this moment of AI.

Tangentially related: Mack and the boys lived in the “Palace Flophouse and Grill” in Cannery Row.

I suppose I must have looked up flophouse when reading all the Steinbeck I could get my hands on and it’s stuck w me.

Re: Was my $48K GPU server worth it?

#320

Post by kriro »

Nice analysis, I would have loved a short overview of the kinds of experiments that were running on the machine (I know the results are given).

I find the "independent researcher" business model quite interesting. In the linked post he writes """DFT is a proprietary training algorithm, however, I’m currently offering a beta for a model training service where I will train your model for you using DFT.""" I'm curious how successful this is. Essentially market some AI breakthrough as a service instead of publishing a paper like my academic brain is trained to do.

As an aside, one thing that I always loved about our field was that the startup cost for many business ideas was "a laptop, internet connection and some some grit". In the age of AI it's quite a bit more and I feel one of the sad side effects of this is that it crowds out poorer and younger developers.

Post reply on HN