Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Was my $48K GPU server worth it?

rosmine.ai

Post reply on HN
Print view View original

421–430 of 480 posts

Re: Was my $48K GPU server worth it?

#421

Post by theYipster »

arjie wrote: post #279
theYipster wrote: post #207

Great article. I'm about to embark on a similar journey.... Doing a ton of AI development right now. Don't need a server, but a very, very high end workstation is super appealing to me right now. Looking at $50-$80k. 1TB RAM. 2x RTX Pro 6000s. 64 core Threadripper Pro. As many 4tb or 8tb nvme drives as I can stuff. I envision NixOS at the core... then everything I need virtualized on top with KVM/QEMU. Maybe a dual b…

What do you want to do with the workstation? I have a similar setup: - 512 GB - Epyc 9684x - 2x RTX 6000 Pro - 1400 W PSU x 2 but in redundant mode Mine is in a colo where it stays nice and cool. In my case, I went with less RAM and more GPUs (bought 4). Secondarily, the Max-Q blower version of an RTX 6000 Pro Blackwell is easier to keep cool and also only needs 300 W at the cost of very little performance. The non-m…

Very nice. Primary use case is application development, where the applications leverage a mixture of cloud based and local models. Modelling complex architectures. My work is primarily in the aerospace and defense arena, so hybrid and on-prem are important, as are ITAR and CMMC compliance. The idea is to have the local rig to build and validate architectural deployments that can sit on prem on customer hardware, in cloud, in gov cloud, or in a mix.

Not really looking at colocation, as this machine would double as a heavy duty gaming and flight sim rig. That means at least one regular RTX 6000 Pro. Not sure if I can mix and match with the Max-Q version, or if I even want blower fans in a desktop case (last time I did that was about 16-18 years ago with an ATI card... wasn't a fan--pun intended.)

Re: Was my $48K GPU server worth it?

#422

Post by beernet »

freediddy wrote: post #10

In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I'v…

Yep, the great theoretical promise of local models remains theoretical, no matter how much die hard-engineers want to push it...Who would have thought, right?

Re: Was my $48K GPU server worth it?

#423

Post by razakel »

bionsystem wrote: post #363
digitaltrees wrote: post #273

Earlier quoted context omitted.

oh young grasshopper, I see you dont know that money launderers love the ebay hype cycle. Its REALLY common on high dollar hot items to have phantom transactions where parties are on both sides of the transaction to clean illicit money. The high price tag and high volume amount of transactions hides the illicit signal. I have tried to buy a few of these mac studios only to have the transaction cancelled because I was…

Funny, they have the "honesty" to cancel the transaction and not take your money, just to keep their ebay reputation high ?

eBay and PayPal almost always side with the buyer.

Re: Was my $48K GPU server worth it?

#424

Post by rithdmc »

dpark wrote: post #415
rithdmc wrote: post #373

Earlier quoted context omitted.

The transaction was cancelled? Sounds like you weren't defrauded.

GP didn’t say they were defrauded. They said the listing was a cover for laundering money.

"weren't scammed" might have been a better choice of words, which they said one post up.

Re: Was my $48K GPU server worth it?

#425

Post by dpark »

rithdmc wrote: post #424
dpark wrote: post #415

Earlier quoted context omitted.

GP didn’t say they were defrauded. They said the listing was a cover for laundering money.

"weren't scammed" might have been a better choice of words, which they said one post up.

Ah, I see what you mean now.

Re: Was my $48K GPU server worth it?

#427

Post by ai_fry_ur_brain »

fauigerzigerk wrote: post #323
ai_fry_ur_brain wrote: post #308

Earlier quoted context omitted.

Its basic math, go calculate max sessions for a certain tps on any hardware. Session# * tps * 86400 (secs in a day) * 30 days. You'll realize real quick its not profitible. You cant just say things you don't like to hear are unsubstantiated without verifying. Not to mention, subscriptions.. $2mm in GPUs being given out for 5 hrs a day at a cost of $200 a month. I could easily say that everyone who says its profitible…

>Its basic math Yes, once you have modeled the problem correctly and you know all the input parameters. This is not that: Session# * tps * 86400 (secs in a day) * 30 days. I don't think there is enough public information to check Anthropic's claims regarding inference profitability. It depends not just on unknown technical factors but also on agreements they have with other companies.

I agree that we dont know how expensive SOTA is. But yes my math should give you the max amount of tokens you can sell per month, and its not remotely profitible for most of the larger open source models (at their current pricing). Im not sure why a 10x larger model that is more in demand would be profitible when its only 5x the price.

Its possible you could pay off hardware for Kimi 2.6 after maybe 2-3 yrs (by providing low tps / high concurrency) but you're now out of warranty and have been running your machines full throttle 24/7 for 2-3 years.

This is why moonshot attempted to double the price when they released 2.6 but then it got driven down by North American capital subsidies.

Re: Was my $48K GPU server worth it?

#428

Post by vessenes »

freediddy wrote: post #10

In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I'v…

I don't think this changes the final conclusion - but have you considered calculating against depreciation -- i.e. figuring out how much your M3 ultra is worth today, and only charging yourself for the delta? In my mind you might even have made money on the hardware.
Post reply on HN