Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

31–40 of 236 posts

Re: Building a personal, private AI computer on a budget

#31

Post by joshstrange »

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past.

I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I feel like the ROI just isn’t there, especially for an industry that is moving so fast.

Privacy is not something to ignore at all but the cost of inference online is very hard to beat, especially when I’m still learning how best to use LLMs.

Re: Building a personal, private AI computer on a budget

#32

Post by sethd »

jmyeet wrote: post #17

The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac M…

For sure and the Mac Mini M4 Pro with 64GB of RAM feels like the sweet spot right now.

That said, the base storage option is only 512GB, and if this machine is also a daily driver, you’re going to want to bump that up a bit. Still, it’s an amazing machine for under $3K.

Re: Building a personal, private AI computer on a budget

#33

Post by walterbell »

elorant wrote: post #20
walterbell wrote: post #9

Earlier quoted context omitted.

200+ comments, https://news.ycombinator.com/item?id=42897205 > This runs the 671B model in Q4 quantization at 3.5-4.25 TPS for $2K on a single socket Epyc server motherboard using 512GB of RAM.

Runs is an overstatement though. With 4 tokens/second you can't use it on production.

Some LLM use cases are async, e.g. agents, "deep research" clones.

Re: Building a personal, private AI computer on a budget

#34

Post by darkwater »

iamleppert wrote: post #30
jmyeet wrote: post #17

The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac M…

The hassle of not being able to work with native CUDA isn't worth it for a huge amount of AI. Good luck getting that latest paper or code working quickly just to try it out, if the author didn't explicitly target M4 (unlikely but all the most mainstream of stuff).

In a homelab scenario, having your own AI assistant not ran by someone else, that is not an issue. If you want to tinker/learn AI it's definitely an issue.

Re: Building a personal, private AI computer on a budget

#35

Post by mechagodzilla »

elorant wrote: post #20
walterbell wrote: post #9

Earlier quoted context omitted.

200+ comments, https://news.ycombinator.com/item?id=42897205 > This runs the 671B model in Q4 quantization at 3.5-4.25 TPS for $2K on a single socket Epyc server motherboard using 512GB of RAM.

Runs is an overstatement though. With 4 tokens/second you can't use it on production.

I have a similar setup running at about 1.5 tokens/second, and it's perfectly usable for the sorts of difficult tasks one needs a frontier model like this for - give it a prompt and come back an hour or two later. You interact with it like e-mailing a coworker. If I need an answer back in seconds, it's probably not a very complicated question, and a much smaller model will do.

Re: Building a personal, private AI computer on a budget

#36

Post by diggan »

walterbell wrote: post #33
elorant wrote: post #20

Earlier quoted context omitted.

Runs is an overstatement though. With 4 tokens/second you can't use it on production.

Some LLM use cases are async, e.g. agents, "deep research" clones.

Not to mention even simpler things, like wanting to tag all of your local notes based on the content, basically a bash loop you can run indefinitely and speed doesn't matter much, as long as it eventually finishes

Re: Building a personal, private AI computer on a budget

#37

Post by walterbell »

One reason to bother with private AI: cloud AI ToS for consumers may have legal clauses about usage of prompt and context data, e.g. data that is not already on the Internet. Enterprise customers can exclude their data from future training.

https://stratechery.com/2025/deep-research-and-knowledge-val...

> Unless, of course, the information that matters is not on the Internet. This is why I am not sharing the Deep Research report that provoked this insight: I happen to know some things about the industry in question — which is not related to tech, to be clear — because I have a friend who works in it, and it is suddenly clear to me how much future economic value is wrapped up in information not being public. In this case the entity in question is privately held, so there aren’t stock market filings, public reports, barely even a webpage! And so AI is blind.

(edited for clarity)

Re: Building a personal, private AI computer on a budget

#38

Post by Eisenstein »

apples_oranges wrote: post #12

Does using 2x24GB VRAM mean that the model can be fully loaded into memory if it's between 24 and 48 GB in size? I somehow doubt it, at least ollama wouldn't work like that I think. But does anyone know?

No, you need to have extra space for the context (which requires more space the larger the model is).

But it should be said that basing model quality on its size in GB is like qualifying a video based on its size in GB. You can have the same video be small or huge with anywhere from negligible to huge differences in quality between the two.

You will be running quantizied model weights, which can range in precision from 1 to 16 bits per parameter (the B for billion in the model name). Model weights at Q8 are generally their parameter size without the B in GB (Llama 3 8B at Q8 would be ~8GB). There are many different strategies for quantizing as well, so this is just a rough guide.

So basically if you can't fit the 48GB model into your 48GB of VRAM, just download a lower precision quant.

Re: Building a personal, private AI computer on a budget

#39

Post by magicalhippo »

numpad0 wrote: post #7
hexomancer wrote: post #2

Isn't the fact the P40 has horrible fp16 performance a deal breaker for local setups?

Is it cheaper in $/GB than used Vega 56(HBM2 8GB) besides? There are mining boards with bunch of x1 slots that probably can run half a dozen of them for same 48GB.

AFAIK this doesn't really work for interactive use, as LLMs process data serially. So your request needs to pass through all of the cards for each token, one at a time. Thus a lot of PCIe traffic and hence latency. Better than nothing, but only really useful if you can batch requests so you can keep each GPU working all the time, rather than just one at a time.

Re: Building a personal, private AI computer on a budget

#40

Post by ImPostingOnHN »

lioeters wrote: post #25
dgrabla wrote: post #4

Great breakdown!. The "own your own AI" at home is a terrific hobby if you like to tinker, but you are going to spend a ton of time and money on hardware that will be underutilized most of the time. If you want to go nuts check out Mitko Vasilev's dream machine. It makes no sense if you don't have a very clear use case that only requires small models or really slow token generation speeds. If the goal however is not…

> spend a ton of time and money Not necessarily. For non-professional purposes, I've spent zero dollars (no additional memory or GPU) and I'm running a local language model that's good enough to help with many kinds of tasks including writing, coding, and translation. It's a personal, private, budget AI that requires no network connection or third-party servers.

on what hardware (and how much did you spend on it)?
Post reply on HN