Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Was my $48K GPU server worth it?

rosmine.ai

Post reply on HN
Print view View original

411–420 of 480 posts

Re: Was my $48K GPU server worth it?

#411

Post by dpark »

miki123211 wrote: post #227
freediddy wrote: post #10

In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I'v…

This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, a…

> You can't get cheaper GPUs

You absolutely can. OpenAI et al are paying a fortune for GPUs but they are not paying retail prices.

The entire business model of retail is to sell above cost.

Re: Was my $48K GPU server worth it?

#412

Post by yard2010 »

iwontberude wrote: post #125
root_axis wrote: post #121

Earlier quoted context omitted.

You spent 50k for plex hosting? Why so expensive?

Half a petabyte of RAID6 is the biggest line item, then the redundant 40gb networking and compute follow closely. I have a lot… too much even?

Dam this is just art in this scale

Re: Was my $48K GPU server worth it?

#413

Post by yard2010 »

LtdJorge wrote: post #173
marai2 wrote: post #151

Earlier quoted context omitted.

How many Blurays are we talking about?

Can't reply to the other poster, but I have 4K HDR Blu-ray copies from discs I found in the street too, which are more in the 60GB ballpark.

In what kind of "streets" you guys are hanging around?

Re: Was my $48K GPU server worth it?

#414

Post by oblio »

baq wrote: post #369
oblio wrote: post #366

Earlier quoted context omitted.

Yup, just like crypto and tulips were rational behaviors for the global economy. Or investing 5% of world GDP in half-baked AI is.

wanting to get rich quick is completely rational if you ask me ;)

Only if they do get rich, if they lose money, it's stupid. Technically stupidity is also rational? The bottom of rationality? :-))

Re: Was my $48K GPU server worth it?

#415

Post by dpark »

rithdmc wrote: post #373
digitaltrees wrote: post #273

Earlier quoted context omitted.

oh young grasshopper, I see you dont know that money launderers love the ebay hype cycle. Its REALLY common on high dollar hot items to have phantom transactions where parties are on both sides of the transaction to clean illicit money. The high price tag and high volume amount of transactions hides the illicit signal. I have tried to buy a few of these mac studios only to have the transaction cancelled because I was…

The transaction was cancelled? Sounds like you weren't defrauded.

GP didn’t say they were defrauded. They said the listing was a cover for laundering money.

Re: Was my $48K GPU server worth it?

#416

Post by krupan »

brightball wrote: post #404
ai_fry_ur_brain wrote: post #308

Earlier quoted context omitted.

Its basic math, go calculate max sessions for a certain tps on any hardware. Session# * tps * 86400 (secs in a day) * 30 days. You'll realize real quick its not profitible. You cant just say things you don't like to hear are unsubstantiated without verifying. Not to mention, subscriptions.. $2mm in GPUs being given out for 5 hrs a day at a cost of $200 a month. I could easily say that everyone who says its profitible…

To some degree I think there's a hope that it becomes like a gym membership. If everybody used their membership, the gym would be too crowded. It's all of those memberships that people feel like they need to have but don't use where the extra profit comes in. As long as the power users are paying per token, everything is good.

Really? This is what we expect from this amazing world changing technology? People will sign up for it and not use it? Good business plan, how can I invest? /s

Re: Was my $48K GPU server worth it?

#417

Post by krupan »

And the net result is a way for LLMs to use more variety in their writing style.

Didn't Sam Altman create LLMs to cure cancer and stuff? Why does their writing style matter as long as the information they are conveying is accurate?

Re: Was my $48K GPU server worth it?

#418

Post by btbuildem »

dekhn wrote: post #50

I can't imagine spending $48K on a home GPU server, but I did just splurge and buy a PC with an RTX 5090, specifically to hold the largest models you can fit in 32GB. It's a top of the line PC with water cooled high end CPUs, 64GB RAM, RTX 5090 for $5K. To me the jury is still out whether this was a worthwhile investment, but I do expect to use this machine for a decade. I don't run it at 100% power (it's mostly idle…

Last fall, seeing the writing on the wall, I pieced together an "AI" rig, 96GB ram, 2x RTX 3090, 9950X - not exactly top of the line, but it came in around CDN$3000 all in all, with most parts second hand. I don't think I could build that for CDN$10000 today.

I've been using it pretty steadily for a variety of personal projects, and the only improvement (aside from the obvious "more VRAM") I feel pressed to make is a portable AC unit / some kind of a focused cooling solution. The rig raises the ambient temperature in the office by 4C at least.

Now with the murmurs of even the large players reconsidering their AI spend, and usage-based pricing shifts, having a self-contained, owned, and independently administered compute resource is looking better and better.

Re: Was my $48K GPU server worth it?

#419

Post by iwontberude »

simplyluke wrote: post #383
swiftcoder wrote: post #327

Earlier quoted context omitted.

does one really need 40gb networking to stream bluerays?

I am (clearly) not as far down the rabbithole as the commenter you're replying to, but almost certainly not. Streaming 4k blueray is on the order or ~100Mb/s, which means on a LAN bog-standard gigabit ethernet and associated networking hardware would be more than sufficient. This is taking a hobby to its extremes, in much the same way that a $5k boat and $500k boat let you catch the same fish.

It’s about being able to rapidly move files between the arrays and future proofing.

Re: Was my $48K GPU server worth it?

#420

Post by iwontberude »

swiftcoder wrote: post #327
iwontberude wrote: post #125

Earlier quoted context omitted.

Half a petabyte of RAID6 is the biggest line item, then the redundant 40gb networking and compute follow closely. I have a lot… too much even?

does one really need 40gb networking to stream bluerays?

You are totally right, it’s mostly just for backups and transfers to rebalance data.
Post reply on HN