Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Was my $48K GPU server worth it?

rosmine.ai

Post reply on HN
Print view View original

291–300 of 480 posts

Re: Was my $48K GPU server worth it?

#291

Post by girvo »

andoando wrote: post #256
girvo wrote: post #163

Earlier quoted context omitted.

I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platform. Theres not much else that lets me fiddle with 128GB of RAM for my graphics processor, and it's quite lovely to be able to run things as long as I like without worrying about my cloud instance being shut down. It's not financially a good idea: renting reall…

You could just rent a bare metal server with those specs

Yes I could, but that is annoying because of spot pricing and having my instance shut down, and it has fluctuating prices

It’s also annoying because then I need to make sure my little “lab” setup is well automated, and I’m lazy :)

Also, I literally said “ It's not financially a good idea” so I’m confused why you think I don’t know that.

Re: Was my $48K GPU server worth it?

#292

Post by girvo »

nostrebored wrote: post #223
girvo wrote: post #163

Earlier quoted context omitted.

I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platform. Theres not much else that lets me fiddle with 128GB of RAM for my graphics processor, and it's quite lovely to be able to run things as long as I like without worrying about my cloud instance being shut down. It's not financially a good idea: renting reall…

$6500 AUD can get you a good chunk of B200 time on any of the GPU neoclouds :)

Less than I expected, though! And I get to run this all through the night

I do still use Vast and Runpod for things too, but it’s much nicer to test a fine tuning run here to make sure I’m in the ballpark

I also did literally say “It's not financially a good idea, renting is better than owning” so I’m confused why I have two people telling me that

Also it’s just far more fun to play with something tangible to me :)

Re: Was my $48K GPU server worth it?

#293

Post by girvo »

a1o wrote: post #209
girvo wrote: post #163

Earlier quoted context omitted.

I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platform. Theres not much else that lets me fiddle with 128GB of RAM for my graphics processor, and it's quite lovely to be able to run things as long as I like without worrying about my cloud instance being shut down. It's not financially a good idea: renting reall…

When you say you are not just doing inference, you mean you are also training your own llms? I am curious what other things can be done.

Fine tuning, and yeah training my own, experimenting with architectures and learning how it all works. Been a lot of fun

Re: Was my $48K GPU server worth it?

#294

Post by jononor »

nextos wrote: post #30
jmyeet wrote: post #20

So some things have changed since this rig was first built (2024). The most relevant is that $6800 RTX 6000 Ada 48GB has arguably been supplanted by the $9500 RTX 6000 Pro 96GB. The Ada has a memory bandwidth of 960GB/s. The Pro has 1.8TB/s and about 40-50% better performance so is at least equivalent in processing power, much better in memory bandwidth (important for inference) and can hold larger models on a single…

I am also considering to buy 3-4x RTX 6000 Pro 96GB plus some Ryzen workstation with a grant. Is this the best general-purpose choice as of 2026 with $50k for training, fine-tuning and running large open models?

Foe multi GPU make sure you have enough PCIE lanes! That rules out consumer grade sockets like AM5, you would need Threadripper or EPYC.

Re: Was my $48K GPU server worth it?

#295

Post by r0b05 »

kgeist wrote: post #150

I administer a simple AI server in the office, which just uses a single RTX 5090 but is able to serve ~80 people throughout the day. I'm impressed by Qwen3.6-27b's capabilities in agentic coding/tasks so far. Devs say it's not much different from Sonnet 4.6 on many tasks (sometimes it even outperformed it), 40-60 tok/sec, up to 260k context. The server cost about $10k with all the bells and whistles. I spent a lot of…

How can a single 5090 serve 80 people? Something doesn't add up here.

Re: Was my $48K GPU server worth it?

#296

Post by utopiah »

cpard wrote: post #113

UPDATE: Launch was a success! 400K+ views, and multiple companies reached to use my IP. Read more here It seems that he managed to get what he wanted from the hardware and I'm happy for them. He said something interesting at the beginning of his post, he compared the cost of the hardware to the cost of his time based on his FAANG salary. Which is an interesting way to think of this, but the rest of the article didn't…

Doubt it, feels basically like just an ad to get attention "Oh look, that's where the magic happens" vs running their code on existing infrastructure and thus just showing the results, like everybody else. This "feels" more "tangible".

Re: Was my $48K GPU server worth it?

#297

Post by utopiah »

timothyduong wrote: post #164
datadrivenangel wrote: post #35

I did the math at least on a Macbook pro, and for inference it's definitely not worth it. - https://www.williamangel.net/blog/2026/05/17/offline-llm-ene... - Discussion: https://news.ycombinator.com/item?id=48168198

That's the case with Self-hosting anything. It is the privacy that matters.

Privacy of what in this case?

Re: Was my $48K GPU server worth it?

#298

Post by PowerElectronix »

miki123211 wrote: post #227
freediddy wrote: post #10

In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I'v…

This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, a…

On top of that, AI providers are also eating a big loss on the service.

Re: Was my $48K GPU server worth it?

#299

Post by utopiah »

dekhn wrote: post #50

I can't imagine spending $48K on a home GPU server, but I did just splurge and buy a PC with an RTX 5090, specifically to hold the largest models you can fit in 32GB. It's a top of the line PC with water cooled high end CPUs, 64GB RAM, RTX 5090 for $5K. To me the jury is still out whether this was a worthwhile investment, but I do expect to use this machine for a decade. I don't run it at 100% power (it's mostly idle…

Well a lot of people have that in their garage, even "worst" it's on wheels so even easier to steal.

I'm not saying it's worth it just that it's not such a crazy amount in comparison.

Re: Was my $48K GPU server worth it?

#300

Post by heisenbit »

datadrivenangel wrote: post #35

I did the math at least on a Macbook pro, and for inference it's definitely not worth it. - https://www.williamangel.net/blog/2026/05/17/offline-llm-ene... - Discussion: https://news.ycombinator.com/item?id=48168198

One value of learning on my Macbook is that mps is not as well supported as cuda which forces me to go down roads I would not have traveled.
Post reply on HN