Did you not use any thermal paste at all, or did you just forget to mention it in your post?
Building an AI server on a budget
71–80 of 113 posts
Re: Building an AI server on a budget
#72In January 2024 there was a similar post ( https://news.ycombinator.com/item?id=38985152 ) wherein the author selected dual NVidia 4060 Ti's for an at-home-LLM-with-voice-control -- because they were the cheapest cost per GB of well-supported VRAM at the time. (They probably still are, or at least pretty close to it.) That informed my decision shortly after, when I built something similar - that video card model was…
> which implies the hubris of a North American No need for that.
Probably true.
But for those of us outside the USA bubble, it's incredibly tring to have to intuit geo information (when geo information would add to the understanding).
As others noted in sibling comments, TFA had in fact mentioned in passing their location (in their quoted prompt to chatgpt, and at the very end of the third supporting point for the decision to go for an Nvidia 4070) 'California, CA'. I confess that I skimmed over both those paragraphs.
Now, sure, CA is a country code, but I stand corrected that the author completely hid their location. Had I spotted those clues I'd not have to have made any assumptions around wall power capabilities & costs, new & second hand market availability / costs, etc.
I think I mostly catered for those considerations in the rest of my original comment though - asserted power sensitivity makes it surprising that a higher-power-requiring, smaller-RAM-capacity, more-expensive-than-a-sibling-generation-16GB card was selected.
Re: Building an AI server on a budget
#73>DECISION: Nvidia RTX 4070
I'm curiuos why OP didn't go for the more recent Nvidia RTX 4060 Ti with 16 GB VRAM that cost cheaper (~USD500) brand new and lesser power consumption at 165W [1].
[1] RTX 5060 Ti 16GB sucks for gaming, but seems like a diamond in the rough for AI:
Re: Building an AI server on a budget
#74Earlier quoted context omitted.
During inference, each token passes through each parameter of the model as a matrix vector products. And then as context grows, each new token passes through all current context tokens as matrix vector products. This means bandwidth requirements grow as context sizes grow. For datacenter workloads batching can be used to efficiently use this memory bandwidth and make things compute bound instead
[I'm still not understanding] It seems to me that even if you pass in a long context on every prompt, that context is still tiny compared to the execution time on the processor/GPU/tensorcore/etc. Lets say I load up a model of 12GB on my 12GB VRAM GPU. I pass in a prompt with 1MB of context which causes a response of 500kb after 1s. That's still only 1.5MB of IO transferred in 1s, which kept the GPU busy for 1s. Incr…
1MB of context can maybe hold 10 tokens depending on your model.
For reference. llama 3.2 8B used to take 4 KiB per token per layer. At 32 layers that is 128KiB or 8 tokens per MiB of KV cache (context). If your context holds 8000 tokens including responses then you need around 1GB.
>Unless the GPU is not fully utilised on each prompt-response cycle, I feel that the GPU is still the bottleneck here, not the bus performance.
Matrix vector multiplication implies a single floating point multiplication and addition (2 flops) per parameter. Your GPU can do way more flops than that without using tensor cores at all. In fact, this workload bores your GPU to death.
Re: Building an AI server on a budget
#75Earlier quoted context omitted.
My home AI machine does image classification.
What kind of image classification do you do at home?
My side project accepts and publishes user generated content. To stay compliant with regulations, I use ML to remove adult content:
Re: Building an AI server on a budget
#76Earlier quoted context omitted.
My home AI machine does image classification.
Using just an Ollama VL Model (gemma3/mistral-small3.1/qwen2.5vl) or a specific library?
My home server detects NSFW images in user generated content on my side project.
source code: https://github.com/KevinColemanInc/NSFW-FLASK
Re: Building an AI server on a budget
#77Good value but a 12GB card isn't going to let you do too much given the low quality of small models. Curious what "home AI" use cases small models are being used for? It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc. My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB…
It's really not going to let you train much which IMO is the only reason I'd personally bother with a big GPU. Gradients get huge and everything does them with single/half precision floating point.
Re: Building an AI server on a budget
#78Good value but a 12GB card isn't going to let you do too much given the low quality of small models. Curious what "home AI" use cases small models are being used for? It would be nice to see a best value home AI setups under different budgets or RAM tiers, e.g. best value configuration for 128 GPU VRAM, etc. My 48GB GPU VRAM "Home AI Server" cost ~$3100 from all parts on eBay running 3x A4000's in a Supermicro 128GB…
I use a Proxmox server with RTX 3060 to generate paintings (I have a couple of old jailbroken Amazon Kindle's attached to walls for that purpose), and to run ollama, which is connected to Home Assistant & their voice preview device, allowing me to talk with LLM without transmitting anything to cloud services.
Admittedly with that amount of VRAM the models I can run are fairly useless for stuff like controlling lights via Home Assistant, occasionally does what I tell it to do but usually not. It is pretty okay for telling me information, like temperature or value of some sensors I have connected to HA. For generating AI paintings it's enough. My server also hosts tons of virtual machines, docker containers and is used for remote gameplay, so the AI thing is just an extra.
Re: Building an AI server on a budget
#79Let me try to put this in the scale of coffee:
--
Using LLM via api: Starbucks.
Inference at home: Nespresso capsules.
Fine-tune a small model at home: Owning a grinder and an italian espresso machine.
Pre-training a model: Owning a moderate coffee plantation.
Re: Building an AI server on a budget
#80Earlier quoted context omitted.
> save on the heartbreak of buying an RTX 5090 only to find that even that doesn’t help much with LLM inference and we’re all gonna need the cheaper-but-more-VRAM Intel Arc B60s When going for more VRAM, with an RTX 5090 currently sitting at $3000 for 32GB, I'm curious why people aren't trying to get the Dell C4140s. Those seem to go for $3000-$4000 for the whole server with 4x V100 16GB, so 64GB total VRAM. Maybe it…
Don't the parallelizing techniques of a 4x build make using them more difficult than a 1x build with no extra parallelism? Couldn't the 32GB 4090 handle more models in their original configurations?
> Don't the parallelizing techniques of a 4x build make using them more difficult than a 1x build with no extra parallelism?
For inference, no. For training, only slightly.