Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an AI server on a budget

informationga.in

Post reply on HN
Print view View original

101–110 of 113 posts

Re: Building an AI server on a budget

#101

Post by reginald78 »

DogRunner wrote: post #15

I used a similar budget and build something like this: 7x RTX 3060 - 12 GB which results in 84GB Vram AMD Ryzen 5 - 5500GT with 32GB Ram All in a 19-inch rack with a nice cooling solution and a beefy power supply. My costs? 1300 Euro, but yeah, I sourced my parts on ebay / second hand. (Added some 3d printed parts into the mix: https://www.printables.com/model/1142963-inter-tech-and-gene... https://www.printables.com…

Great info in this post with some uncommon questions answered. I have a 3060 with unimpressive idle power consumption, interesting that it varies so much.

I know it would increase the idle power consumption, but have you considered a server platform instead of Ryzen to get more lanes?

Even so, you could probably get at least 4x for 4 cards without getting to crazy. 2 m.2 -> pcie adapters, the main GPU slot and the fairly common 4x wired secondary slot.

Splitting the main 16x GPU slot is possible but whenever I looked into this I kind of found the same thing you did. In addition to being a cabling/mounting nightmare the necessary hardware started to eat up enough total system cost that just ponying up for a 3090 started to make more sense.

Re: Building an AI server on a budget

#102

Post by ijk »

nickpsecurity wrote: post #56
semi-extrinsic wrote: post #34

Earlier quoted context omitted.

> save on the heartbreak of buying an RTX 5090 only to find that even that doesn’t help much with LLM inference and we’re all gonna need the cheaper-but-more-VRAM Intel Arc B60s When going for more VRAM, with an RTX 5090 currently sitting at $3000 for 32GB, I'm curious why people aren't trying to get the Dell C4140s. Those seem to go for $3000-$4000 for the whole server with 4x V100 16GB, so 64GB total VRAM. Maybe it…

Don't the parallelizing techniques of a 4x build make using them more difficult than a 1x build with no extra parallelism? Couldn't the 32GB 4090 handle more models in their original configurations?

For LLM inference parallel GPUs is mostly fine (you take some performance hit but llama.cpp doesn't care what cards you use and other stuff handles 4 symmetric GPUs just fine). You get more problems when you're doing anything training related, though.

Re: Building an AI server on a budget

#105

Post by Uehreka »

numpad0 wrote: post #49
Uehreka wrote: post #7

Love the attention to detail, I can tell this was a lot of work to put together and I hope it helps people new to PC building. I will note though, 12GB of VRAM and 32GB of system RAM is a ceiling you’re going to hit pretty quickly if you’re into messing with LLMs. There’s basically no way to do a better job at the budget you’re working with though. One thing I hear about a lot is people using things like RunPod to br…

I don't understand why some people build a "rig", put a lot of thoughts into ever so slightly differently binned CPUs, and then don't max out RAM(put aside DDR5 quirk considerations). It's like buying a sports car only to cheap out on tires. It makes no sense.

I built my current computer last fall. The Ryzen 7950X was on an awesome sale for black Friday and after looking at the math buying a 9950X just didn’t make sense. So I got the 7950X and 96GB of DDR5 RAM (2 sticks, so I can double later if I need to). Loving it, it was the perfect choice.

All this to say some people do in fact do this ;)

Re: Building an AI server on a budget

#107

Post by [deleted] »

7speter wrote: post #6

I dunno everyone, but I think Intel has something big on their hands with their announced workstation gpus. The b50 is a low profile card that doesn’t have a powersupply hookup because it only uses something like 60 watts, and comes with 16gb vram at a msrp of 300 dollars. I imagine companies will have first dibs via the likes of agreements with suppliers like CDW, etc, but if Intel had enough of these battlemage die…

[deleted]

Re: Building an AI server on a budget

#108

Post by AzN1337c0d3r »

usercvapp wrote: post #37

I have a server at home sitting IDLE for the last 2 years with 2 TB of RAM and 4 CPUs. I am gonna push it this week and launch some LLM models to see how they perform! How much electric bill efficient are they running locally?

Depends on the server. Probably not going to be cost effective. I get barely ~0.5 tokens/sec.

I have Dual E5-2699A v4 w/1.5 TB DDR4-2933 spread across 2 sockets.

The full Deepseek-R1 671B (~1.4 TB) with llama.cpp seems to have a in that local engines that run the LLMs don't do NUMA aware allocation, so cores will often have to pull the weights in from another socket's memory controllers through the inter-socket links (QPI/UPI/Hypertransport) and bottleneck there.

For my platform that's 2x QPI links @ ~39.2GB/s/link that get saturated.

I give it a prompt, go to work and check back on it at lunch and sometimes it's still going.

If you're going to want to achieve interactively I'd aim for 7-10 tokens/s, so realistically it means you'll run one of the 8b models on a GPU (~30 tokens/s) or maybe a 70b model on an M4 Max (~8 tokens/s).

Re: Building an AI server on a budget

#109

Post by rwyinuse »

garyfirestorm wrote: post #91
rwyinuse wrote: post #78

Earlier quoted context omitted.

I use a Proxmox server with RTX 3060 to generate paintings (I have a couple of old jailbroken Amazon Kindle's attached to walls for that purpose), and to run ollama, which is connected to Home Assistant & their voice preview device, allowing me to talk with LLM without transmitting anything to cloud services. Admittedly with that amount of VRAM the models I can run are fairly useless for stuff like controlling lights…

Why do you say that? You can easily finetune 8B parameter model for function calling.

That's good to know. The model I was using supported function calling, but seemed to get the calls often wrong. Perhaps I should try a more fine-tuned model for the purpose.
Post reply on HN