What are the practical uses of a self hosted LLM? Is it actually possible to approach the likes of Claude or one of the other big ones on your own hardware for a reasonable budget? I don’t know if this is something that’s actually worth it or if people are just building these rigs for fun or niche use cases that don’t require the intelligence of a hosted LLM.
Building an AI server on a budget
51–60 of 113 posts
Re: Building an AI server on a budget
#52This is interesting. We recently built a similar machine to implement a product that we're building on a customer site.
I didn't buy second hand parts since i wasn't sure of the quality so it was a little pricey but we have the entire thing working now and over the last week, we added the llm server to the mix. Haven't released it yet though.
I wrote about some "fun" we had getting it together here but it's not as technically detailed as the original article.
https://blog.hpcinfra.com/when-linkedin-met-reality-our-bang...
Re: Building an AI server on a budget
#53I would consider adding $400 for something like this instead: https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto...
Do you use this? If so, what's your use case and performance?
Re: Building an AI server on a budget
#54I would rather spend $1,300 on openai/anthropic credits. The performance from that 4070 cannot be worth the squeeze.
[dead]
Re: Building an AI server on a budget
#55Someone posted that they had used a "mining rig" [0] from AliExpress for less than $100. It even has RAM and a CPU. He picked up a 2000W (!) DELL server PS for cheap off eBay. The GPUs were NVIDIA TESLAs (M40 for example) since they often have a lot of RAM and are less expensive. I followed in those footsteps to create my own [1] (photo [2]). I picked up a 24GB M40 for around $300 off eBay. I 3D printed a "cowl" for…
I had an old Tesla M40 12 GB lying around and figured I’d try it out with some 8-13B llms, but was disappointed to find that it’s around the same speed as my mac mini m2. I suppose the mac mini is a 10 years newer chip, but it’s crazy that mobile today matches data center from 10 years ago
Re: Building an AI server on a budget
#56Love the attention to detail, I can tell this was a lot of work to put together and I hope it helps people new to PC building. I will note though, 12GB of VRAM and 32GB of system RAM is a ceiling you’re going to hit pretty quickly if you’re into messing with LLMs. There’s basically no way to do a better job at the budget you’re working with though. One thing I hear about a lot is people using things like RunPod to br…
> save on the heartbreak of buying an RTX 5090 only to find that even that doesn’t help much with LLM inference and we’re all gonna need the cheaper-but-more-VRAM Intel Arc B60s When going for more VRAM, with an RTX 5090 currently sitting at $3000 for 32GB, I'm curious why people aren't trying to get the Dell C4140s. Those seem to go for $3000-$4000 for the whole server with 4x V100 16GB, so 64GB total VRAM. Maybe it…
Don't the parallelizing techniques of a 4x build make using them more difficult than a 1x build with no extra parallelism? Couldn't the 32GB 4090 handle more models in their original configurations?
Re: Building an AI server on a budget
#57If the author is reading this I'll point out that the cuda toolkit you find in the repositories is generally older. You can find the latest versions straight from Nvidia: https://developer.nvidia.com/cuda-downloads?target_os=Linux&... The caveat is that sometimes a library might be expecting an older version of cuda. The vram on the GPU does make a difference, so it would at some point be worth looking at another GPU…
The other main caveat is that installing from custom sources using apt is a massive pain in the ass.
I tried running an LLM locally today, installed cuda toolkit, and it was missing cudann.h
I gave up.
Re: Building an AI server on a budget
#58Good value but a 12GB card isn't going to let you do too much given the low quality of small models. Curious what "home AI" use cases small models are being used for?
It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc.
My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB RAM, 32/64 core Xeon 1U rack server. Nothing amazing but wanted the most GPU VRAM before paying the premium Nvidia tax on their larger GPUs.
This works well for Ollama/llama-server which can make use of all GPU VRAM unfortunately ComfyUI can't make use of all GPU VRAM to run larger models, so on the lookout for a lot more RAM in my next GPU Server.
Really hoping Intel can deliver with its upcoming Arc Pro B60 Dual GPU for a great value 48GB option which can be run 4x in an affordable 192GB VRAM workstation [1]. If it runs Ollama and ComfyUI efficiently I'm sold.
[1] https://www.servethehome.com/maxsun-intel-arc-pro-b60-dual-g...
Re: Building an AI server on a budget
#59why not one of these?
https://www.amazon.sg/NVIDIA-Jetson-Orin-64GB-Developer/dp/B...
Re: Building an AI server on a budget
#60Earlier quoted context omitted.
> The trick is memory bandwidth - not just the amount of VRAM - is important for LLM inference. I'm not really knowledgeable about this space, so maybe I'm missing something: Why does the bus performance affect token generation? I would expect it to cause a slow startup when loading the model, but once the model is loaded, just how much bandwidth can the token generation possibly use? Token generation is completely o…
During inference, each token passes through each parameter of the model as a matrix vector products. And then as context grows, each new token passes through all current context tokens as matrix vector products. This means bandwidth requirements grow as context sizes grow. For datacenter workloads batching can be used to efficiently use this memory bandwidth and make things compute bound instead
[I'm still not understanding]
It seems to me that even if you pass in a long context on every prompt, that context is still tiny compared to the execution time on the processor/GPU/tensorcore/etc.
Lets say I load up a model of 12GB on my 12GB VRAM GPU. I pass in a prompt with 1MB of context which causes a response of 500kb after 1s. That's still only 1.5MB of IO transferred in 1s, which kept the GPU busy for 1s. Increasing the prompt is going to increase the duration to a response accordingly.
Unless the GPU is not fully utilised on each prompt-response cycle, I feel that the GPU is still the bottleneck here, not the bus performance.