Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an AI server on a budget

informationga.in

Post reply on HN
Print view View original

41–50 of 113 posts

Re: Building an AI server on a budget

#41

Post by rjsw »

JKCalhoun wrote: post #26

Someone posted that they had used a "mining rig" [0] from AliExpress for less than $100. It even has RAM and a CPU. He picked up a 2000W (!) DELL server PS for cheap off eBay. The GPUs were NVIDIA TESLAs (M40 for example) since they often have a lot of RAM and are less expensive. I followed in those footsteps to create my own [1] (photo [2]). I picked up a 24GB M40 for around $300 off eBay. I 3D printed a "cowl" for…

There was an article on Tom's Hardware recently where someone was using a CPU cooler with a GPU [1].

[1] https://www.tomshardware.com/pc-components/gpus/crazed-modde...

Re: Building an AI server on a budget

#43

Post by Havoc »

> You pay a lot upfront for the hardware, but if your usage of the GPU is heavy, then you save a lot of money in the long run.

Last I saw data on this wasn’t true. A like for like comparison (same model and quant) API is cheaper than elec so you never make back hardware cost. That was a year ago and api costs have plummeted so I’d imagine it’s even worse now.

Datacenters have cheaper elec, can do batch inference at scale and more efficient cards. And that’s before we consider the huge free allowances by Google etc

Own AI gear is cool…but not due to economics

Re: Building an AI server on a budget

#44

Post by dfc »

topato wrote: post #33
throwaway314155 wrote: post #21

Earlier quoted context omitted.

> which implies the hubris of a North American No need for that.

He did soften the blow by saying North American, rather than the more correctly appropos, American

The author also refers to Californian power limits. So it seems the criticism is misplaced.

Re: Building an AI server on a budget

#46

Post by dcassett »

T-A wrote: post #36
Jedd wrote: post #9

In January 2024 there was a similar post ( https://news.ycombinator.com/item?id=38985152 ) wherein the author selected dual NVidia 4060 Ti's for an at-home-LLM-with-voice-control -- because they were the cheapest cost per GB of well-supported VRAM at the time. (They probably still are, or at least pretty close to it.) That informed my decision shortly after, when I built something similar - that video card model was…

> TFA doesn't say where they are "the 1,440W limit on wall outlets in California" is a pretty good hint.

San Francisco specifically:

"I prompted ChatGPT to give me recommendations. Prompt: ... The final build will be located at my residence in San Francisco, CA, ..."

Re: Building an AI server on a budget

#47

Post by 1shooner »

Jedd wrote: post #9

In January 2024 there was a similar post ( https://news.ycombinator.com/item?id=38985152 ) wherein the author selected dual NVidia 4060 Ti's for an at-home-LLM-with-voice-control -- because they were the cheapest cost per GB of well-supported VRAM at the time. (They probably still are, or at least pretty close to it.) That informed my decision shortly after, when I built something similar - that video card model was…

>TFA doesn't say where they are, or what currency they're using

They say California, and I'm seeing the dollar amount in the title and metadata as $1,3k, was that an edit?

Re: Building an AI server on a budget

#48

Post by numpad0 »

Couple best vram for buck && borderline space heater GPUs off top of my head: Tesla K80(12GBx2), M40(24GB), Radeon Instinct MI(25|50|60|100)(8-32GB?), Radeon Pro V340(16GBx2), bunch of other Radeon Vega 8GB cards e.g. Vega 56, NVIDIA P102/P104(~16GB), Intel A770(16GB). Note: some of these are truly just space heaters.

I'm not sure if right now is the best timing for building an LLM rig, as Intel Arc B60(24GBx2) is about to go on sale. Or maybe it is to secure multiples of 16GB cards hastily offloaded before its launch?

Re: Building an AI server on a budget

#49

Post by numpad0 »

Uehreka wrote: post #7

Love the attention to detail, I can tell this was a lot of work to put together and I hope it helps people new to PC building. I will note though, 12GB of VRAM and 32GB of system RAM is a ceiling you’re going to hit pretty quickly if you’re into messing with LLMs. There’s basically no way to do a better job at the budget you’re working with though. One thing I hear about a lot is people using things like RunPod to br…

I don't understand why some people build a "rig", put a lot of thoughts into ever so slightly differently binned CPUs, and then don't max out RAM(put aside DDR5 quirk considerations). It's like buying a sports car only to cheap out on tires. It makes no sense.

Re: Building an AI server on a budget

#50

Post by stevenhuang »

lelanthran wrote: post #28
olowe wrote: post #14

Earlier quoted context omitted.

> I thought prevailing wisdom was that a used 3090 with it's larger vram was the best budget gpu choice? The trick is memory bandwidth - not just the amount of VRAM - is important for LLM inference. For example, the B50 specs list a memory bandwidth of 224 GB/s [1], whereas the Nvidia RTX 3090 has over 900GB/s [2]. The 4070's bandwidth is "just" 500GB/s [3]. More VRAM helps run larger models but with lower bandwidth…

> The trick is memory bandwidth - not just the amount of VRAM - is important for LLM inference. I'm not really knowledgeable about this space, so maybe I'm missing something: Why does the bus performance affect token generation? I would expect it to cause a slow startup when loading the model, but once the model is loaded, just how much bandwidth can the token generation possibly use? Token generation is completely o…

During inference, each token passes through each parameter of the model as a matrix vector products. And then as context grows, each new token passes through all current context tokens as matrix vector products.

This means bandwidth requirements grow as context sizes grow.

For datacenter workloads batching can be used to efficiently use this memory bandwidth and make things compute bound instead

Post reply on HN