> 12GB vram
waste of effort, why would you go through the trouble of building + blogging for this?
Building an AI server on a budget
31–40 of 113 posts
Re: Building an AI server on a budget
#32In January 2024 there was a similar post ( https://news.ycombinator.com/item?id=38985152 ) wherein the author selected dual NVidia 4060 Ti's for an at-home-LLM-with-voice-control -- because they were the cheapest cost per GB of well-supported VRAM at the time. (They probably still are, or at least pretty close to it.) That informed my decision shortly after, when I built something similar - that video card model was…
> which implies the hubris of a North American No need for that.
True, though
Re: Building an AI server on a budget
#33In January 2024 there was a similar post ( https://news.ycombinator.com/item?id=38985152 ) wherein the author selected dual NVidia 4060 Ti's for an at-home-LLM-with-voice-control -- because they were the cheapest cost per GB of well-supported VRAM at the time. (They probably still are, or at least pretty close to it.) That informed my decision shortly after, when I built something similar - that video card model was…
> which implies the hubris of a North American No need for that.
He did soften the blow by saying North American, rather than the more correctly appropos, American
Re: Building an AI server on a budget
#34Love the attention to detail, I can tell this was a lot of work to put together and I hope it helps people new to PC building. I will note though, 12GB of VRAM and 32GB of system RAM is a ceiling you’re going to hit pretty quickly if you’re into messing with LLMs. There’s basically no way to do a better job at the budget you’re working with though. One thing I hear about a lot is people using things like RunPod to br…
> save on the heartbreak of buying an RTX 5090 only to find that even that doesn’t help much with LLM inference and we’re all gonna need the cheaper-but-more-VRAM Intel Arc B60s
When going for more VRAM, with an RTX 5090 currently sitting at $3000 for 32GB, I'm curious why people aren't trying to get the Dell C4140s. Those seem to go for $3000-$4000 for the whole server with 4x V100 16GB, so 64GB total VRAM.
Maybe it's just because they produce heat and noise like a small turbojet.
Re: Building an AI server on a budget
#353090 for ~1000 is much more solid choice. Also these old mining mobos play very well for multi-gpu ollama.
Re: Building an AI server on a budget
#36In January 2024 there was a similar post ( https://news.ycombinator.com/item?id=38985152 ) wherein the author selected dual NVidia 4060 Ti's for an at-home-LLM-with-voice-control -- because they were the cheapest cost per GB of well-supported VRAM at the time. (They probably still are, or at least pretty close to it.) That informed my decision shortly after, when I built something similar - that video card model was…
> TFA doesn't say where they are
"the 1,440W limit on wall outlets in California" is a pretty good hint.
Re: Building an AI server on a budget
#37I have a server at home sitting IDLE for the last 2 years with 2 TB of RAM and 4 CPUs.
I am gonna push it this week and launch some LLM models to see how they perform!
How much electric bill efficient are they running locally?
Re: Building an AI server on a budget
#38I would consider adding $400 for something like this instead:
https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto...
Re: Building an AI server on a budget
#39The RTX market is particularly irritating right now, even second-hard 4090s are still going for MSRP if you can find them at all. Most of the recommendations for this budget AI system are on point - the only thing I'd recommend is more RAM. 32GB is not a lot - particularly if you start to load larger models through formats such as GGUF and want to take advantage of system ram to split the layers at the cost of infere…
I thought those 4090’s were weird. You pay more for them than the brand new 5090. And then there’s AMD, which everyone loves to hate, but has similar GPU’s that cost 1/4th of what a similar Nvidia GPU costs.
Re: Building an AI server on a budget
#40I've been dreaming on pcpartpicker. I think Radeon RX 7900 XT - 20 GB has been the best bang for your buck. Enables full gpu 32B? Looking at what other people have been doing lately, they arent doing this. They are getting 64+ core cpus and 512GB of ram. Keeping it on cpu and enabling massive models. This setup lets you do deepseek 671B. It makes me wonder, how much better is 671B vs 32B?
I bought an RX 7900 XTX with 24GB, and it’s everything I expected of it. It’s absolutely massive though. I thought I could add one extra for more memory, but that’s a pipe dream in my little desktop box.
Cheap too, compared to a lot of what I’m seeing.