Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Was my $48K GPU server worth it?

rosmine.ai

Post reply on HN
Print view View original

101–110 of 480 posts

Re: Was my $48K GPU server worth it?

#101

Post by rosmine »

janalsncm wrote: post #63

(For reference I’m talking about the DFT post from the same blog.) I love that ML is still in the “gentleman researcher” stage where relatively small amounts of startup capital can buy a ticket into frontier research. For a lot of research questions 6 GPUs is even overkill. It’s one of the reasons I’m skeptical of the “trillion dollar supercluster” idea [0]. I think what we need is more reasonably smart people invest…

I agree :) Also, I heard Teknium trained the original Hermes model on 2x 4090. You can do a lot with a little compute

Re: Was my $48K GPU server worth it?

#102

Post by 4chandaily »

rosmine wrote: post #94

Hi! Thank you so much for posting this! I got back luck/timing when I tried, so happy it made it to the front page! (I am the author)

I did this with used parts and cheaper consumer cards (3090s) and did much of the same calculations. I found it was way cheaper for me as well.

The main advantage, however, is that the friction of "this is going to cost me in tokens to even try" goes away. I was so much more willing to take chances and try new things on my own hardware than I would have been if I were paying API costs. I feel like this point isn't made clearly enough by those of us who run these absurd self-hosted inference systems.

Thanks for the write up, was a fun read. I spent an order of magnitude less, but I could relate to your story from beginning to end.

Epyc (Milan), 512gb ram, 4x 3090

Re: Was my $48K GPU server worth it?

#103

Post by dekhn »

fortyseven wrote: post #92
dekhn wrote: post #59

Earlier quoted context omitted.

I have a second computer with an RTX 4090 for gaming (running Windows). I also used the new RTX 5090 running Linux to evaluate whether Proton/Wine allow me to run Windows games on linux (yes, it works, but the compatibility and frame rate issues make me stick to native Windows for now).

I wonder what's going wrong there? Personally I found compatibility and performance on Linux to be extremely good. And just keeps getting better. And that's not even just me, that's all kinds of benchmarks out there. Sorry to hear that. : ' (

No idea. I agree that in principle I should have close to the same performance on Linux. I just didn't want to spend a bunch of time customizing configs and updating software so I could reach parity with Windows when I had two computers.

Re: Was my $48K GPU server worth it?

#104

Post by orsorna »

throwatdem12311 wrote: post #51
dekhn wrote: post #50

I can't imagine spending $48K on a home GPU server, but I did just splurge and buy a PC with an RTX 5090, specifically to hold the largest models you can fit in 32GB. It's a top of the line PC with water cooled high end CPUs, 64GB RAM, RTX 5090 for $5K. To me the jury is still out whether this was a worthwhile investment, but I do expect to use this machine for a decade. I don't run it at 100% power (it's mostly idle…

Not even a single mention of gaming. No wonder gamers hate AI bros.

I would probably hate someone if they were buying the same hardware as me but doing something actually useful with it. Any game worth playing doesn't require high specs anyway. There is such a large catalog of old games.

Re: Was my $48K GPU server worth it?

#105

Post by rib3ye »

> I thought that I could not get a standard datacenter server because my apartment wouldn’t let me upgrade the circuits, so I needed to have 2 power supplies plugged into different circuits.

Why didn't they just put a higher amp breaker in the box?

Re: Was my $48K GPU server worth it?

#106

Post by kccqzy »

LandoCalrissian wrote: post #95

'If you google “plugging a PC into multiple outlets”, you get lots of warnings that if you even consider such a setup you will instantly burst into flames. So I hired a professional PC builder make sure it was safe.' Not really sure how that makes it safe but OK!

Probably means hiring someone who has more knowledge about PSUs and especially about having two simultaneous PSUs. There are questions like: when you press the power button how do the two PSUs turn up and in what sequence? How do you deal with the PWR_OK signal? What if there are voltage differences between the two PSUs? What about power backfeeding?

Re: Was my $48K GPU server worth it?

#107

Post by phkahler »

dekhn wrote: post #50

I can't imagine spending $48K on a home GPU server, but I did just splurge and buy a PC with an RTX 5090, specifically to hold the largest models you can fit in 32GB. It's a top of the line PC with water cooled high end CPUs, 64GB RAM, RTX 5090 for $5K. To me the jury is still out whether this was a worthwhile investment, but I do expect to use this machine for a decade. I don't run it at 100% power (it's mostly idle…

>> To me the jury is still out whether this was a worthwhile investment, but I do expect to use this machine for a decade.

The high cost and power consumption are both signs of the death of Moore's law, so you are probably correct that this system will be near state of the art for some time.

Re: Was my $48K GPU server worth it?

#108

Post by mrandish »

LandoCalrissian wrote: post #95

'If you google “plugging a PC into multiple outlets”, you get lots of warnings that if you even consider such a setup you will instantly burst into flames. So I hired a professional PC builder make sure it was safe.' Not really sure how that makes it safe but OK!

I guess it was supposed to be a humorous aside, but it wasn't actually helpful because the relevant issue is when you pull more total amps from a single circuit than it's fused for (usually 15 or 20 amps in U.S. residences). The failure mode is usually tripping the circuit breaker.

That issue can often be addressed fairly easily by splitting the power draw between two adjacent circuits. You can have an electrician do it permanently or temporarily DIY it with an appropriately rated extension cord. The real issue was OP was in an apartment at the time so an electrician would have been difficult. I assume they decided to just have a system integrator build it because they didn't want to figure out how to segment and route the power rails in a dual power supply system, but it's not exactly rocket science. Problems are often more due to choosing power supplies that aren't up to their claimed spec, not pre-testing them under load or using incorrect or under-spec cables.

Re: Was my $48K GPU server worth it?

#109

Post by chaidhat »

rib3ye wrote: post #105

> I thought that I could not get a standard datacenter server because my apartment wouldn’t let me upgrade the circuits, so I needed to have 2 power supplies plugged into different circuits. Why didn't they just put a higher amp breaker in the box?

It is unsafe for wires to be handling higher power than it was rated cause the wires act like very low ohm resistors. At some high enough I, you’re still gonna be generating power P=I^2R which is mainly thermal and melt the wires.

Re: Was my $48K GPU server worth it?

#110

Post by ttshaw1 »

mrandish wrote: post #108
LandoCalrissian wrote: post #95

'If you google “plugging a PC into multiple outlets”, you get lots of warnings that if you even consider such a setup you will instantly burst into flames. So I hired a professional PC builder make sure it was safe.' Not really sure how that makes it safe but OK!

I guess it was supposed to be a humorous aside, but it wasn't actually helpful because the relevant issue is when you pull more total amps from a single circuit than it's fused for (usually 15 or 20 amps in U.S. residences). The failure mode is usually tripping the circuit breaker. That issue can often be addressed fairly easily by splitting the power draw between two adjacent circuits. You can have an electrician do…

I think the relevant issue is you could conceivably have a house with two outlets with opposite phases. Bussing them together in the PSU will then create a short
Post reply on HN