Was my $48K GPU server worth it?
rosmine.ai
Was my $48K GPU server worth it?
1–10 of 480 posts
Re: Was my $48K GPU server worth it?
#2> Because of this I got a motherboard with slow GPU interconnect. It’s good for running many small experiments in parallel (which is my main use case) but horrible for any models split across gpus.
:( you paid a professional pc builder and you weren't told this?
Re: Was my $48K GPU server worth it?
#3> Because of this I got a motherboard with slow GPU interconnect. It’s good for running many small experiments in parallel (which is my main use case) but horrible for any models split across gpus. :( you paid a professional pc builder and you weren't told this?
Don't those Ada 6000 GPUs support NVLink? I think I can even see the cover for the connectors in OP's pic.
edit: Hm, finding mixed information online on whether that's still supported or not. Apparently it was removed in workstation GPUs.
Re: Was my $48K GPU server worth it?
#4> Because of this I got a motherboard with slow GPU interconnect. It’s good for running many small experiments in parallel (which is my main use case) but horrible for any models split across gpus. :( you paid a professional pc builder and you weren't told this?
Don't those Ada 6000 GPUs support NVLink? I think I can even see the cover for the connectors in OP's pic. edit: Hm, finding mixed information online on whether that's still supported or not. Apparently it was removed in workstation GPUs.
Nope, they don't support it. And afair even if they did, you would be limited to connecting only in pairs, not all 6 together
Re: Was my $48K GPU server worth it?
#5> Because of this I got a motherboard with slow GPU interconnect. It’s good for running many small experiments in parallel (which is my main use case) but horrible for any models split across gpus. :( you paid a professional pc builder and you weren't told this?
Consumer motherboards can still make sense even if you leave some performance on the table. Running an actual 8x GPU server is not something you'd want to do in an apartment. Imagine the old Lucasfilm "THX" trailer where an unearthly-sounding foghorn whine rises to a sweeping crescendo at reference level, only without the decay at the end.
At the time he put this rig together, there weren't a lot of open-weight LLMs that could run well on 6x48=288 GB, so it probably wasn't a huge loss. There still aren't, really.
Right now I'm in the process of cramming Blackwell cards into an old DDR4-based Milan server, where the important thing is to be able to run large models at all. The GPU fans alone burn over 400 watts at full throttle.
Re: Was my $48K GPU server worth it?
#6It doesn't cover risk. If one or more gpus dies, who pays for it? If you rent, you are guaranteed to be insulated from this risk. But owning, you might not have the best return policy from the vendor. And if you are actually at fault for breaking it, they have every right to deny a return. Or if your apartment is burglarized or catches fire (possibly from overloading the circuit) you are out the entire investment.
Re: Was my $48K GPU server worth it?
#7I have four old 24gb Nvidia cards. They're not great but they're not useless either. The problem is that I haven't really figured out a good way to actually use them.
Genuine question; would anyone here recommend any specific motherboard to best utilize these cards?
Re: Was my $48K GPU server worth it?
#8Just curious OP (if you're the one posting) -- what do you mean by independent researcher? What are you researching and are you making $$ from it or are you living off previous built up savings? Seems like an interesting path. What research have you looked into so far?
Re: Was my $48K GPU server worth it?
#9> Because of this I got a motherboard with slow GPU interconnect. It’s good for running many small experiments in parallel (which is my main use case) but horrible for any models split across gpus. :( you paid a professional pc builder and you weren't told this?
I wonder why using 2 PSUs resulted in having slower interconnect.
There is no specs in this blogpost regarding cpu/motherboard choice, but if you go with threadripper pro they have 128 pci-e lanes for some time now, so using all GPUs at full speed shouldn't be a problem
Re: Was my $48K GPU server worth it?
#10In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment.
In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I've been running against Gemma4:31b, Qwen3.5 and 3.6, and getting local LLMs to solve AMC 8/10 math questions and it's about 10-100x slower than just doing it online. When I tried it with ChatGPT late last year, it took about one night and $25 to solve about 1000 questions. Using my RTX 6000 and M3 Ultra and Gemma4:31b on both, it answered about 40 questions in 7 hours and I haven't checked how good the answer is yet. At 800 watts (600 for RTX and 200 for M3 Ultra) and running for 7 hours, it solved around 40 questions.
At the very least I'm going to try to sell my M3 Ultra if I can find a reliable place to sell it without getting ripped off by scammers.