Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

141–150 of 236 posts

Re: Building a personal, private AI computer on a budget

#141

Post by fragmede »

gmueckl wrote: post #138
cwoolfe wrote: post #136

As others have said, a high powered Mac could be used for the same purpose at a comparable price and lower power usage. Which makes me wonder: why doesn't Apple get into the enterprise AI chip game and compete with Nvidia? They could design their own ASIC for it with all their hardware & manufacturing knowledge. Maybe they already are.

The primary market for such a product would be businesses. And Apple isn't particularly good at selling to companies. The consumer product focus may just be too ingrained to be successful with such a move. A beefed up home pod with a local LLM-based assistant would be a more typical Apple product. But they'd probably need LLMs to become much, much more reliable to not ruin their reputation over this.

Why? Siri's still total crap but that doesn't seem to have slowed down iPhone sales.

Re: Building a personal, private AI computer on a budget

#142

Post by anonzzzies »

ollybee wrote: post #126

The middle ground is to rent a GPU VPS as needed. You can get an H100 for $2/h. Not quite the same privacy as fully local offline, but better than a SASS API and good enough for me. Hopefully in a year or three it will truly be cost effective to run something useful locally and then I can switch.

That is what I do but it costs a lot of $, more than just using openrouter. I would like to have a machine so I can have a model talk to itself 24/7 for a realtively fixed price. I have enough solar and wind + cheap net electric so it would basically be free after buying. Just hard to pick what to buy without just forking out a fortune on GPU's.

Re: Building a personal, private AI computer on a budget

#143

Post by diggan »

achierius wrote: post #135
diggan wrote: post #112

Earlier quoted context omitted.

So Apple M2 Studio does ~15 tks/second and A100-SXM4-80GB does 9 tks/second? I'm not sure I'm reading the results wrong or missing some vital context, but that sounds unlikely to me.

The studio has a lot more ram available to the GPU (up to 192gb) than the a100 (80gb), and iirc at least comparable memory bandwidth -- those are what matter when you're doing LLM inference, so the studio tends to win out there. Where the a100 and other similar chips dominate is in training &c, which is mostly a question of flops.

> and iirc at least comparable memory bandwidth

I don't think they do.

From Wikipedia:

> the M2 Pro, M2 Max, and M2 Ultra have approximately 200 GB/s, 400 GB/s, and 800 GB/s respectively

From techpowerup:

> NVIDIA A100 SXM4 80 GB - Memory bandwidth - 2.04 TB/s

Seems to be a magnitude of difference, and that's just the bandwidth.

Re: Building a personal, private AI computer on a budget

#144

Post by oofbaroomf »

anthonyskipper wrote: post #134
oofbaroomf wrote: post #118

Earlier quoted context omitted.

The bottleneck for single batch inference is memory bandwidth. The M4 Pro has less memory bandwidth than the P40, so it would be slower. Also, the setup presented in the OP has system RAM, allowing you to run models than what fits in 48GB of VRAM (and with good speeds too if you offload with something like ktransformers).

>>M4 Pro has less memory bandwidth than the P40, so it would be slower Why do you say this? I thought the p40 only had a memory bandwidth of 346 Gbytes/sec. The m4 is 546 GB/s. So the macbook should kick the crap out of the p40.

The M4 Max has up to 546 GB/s. The M4 Pro, what GP was talking about, has only 273 GB/s. An M4 Max with that much RAM would most likely exceed OP's budget.

Re: Building a personal, private AI computer on a budget

#145

Post by gmueckl »

fragmede wrote: post #141
gmueckl wrote: post #138

Earlier quoted context omitted.

The primary market for such a product would be businesses. And Apple isn't particularly good at selling to companies. The consumer product focus may just be too ingrained to be successful with such a move. A beefed up home pod with a local LLM-based assistant would be a more typical Apple product. But they'd probably need LLMs to become much, much more reliable to not ruin their reputation over this.

Why? Siri's still total crap but that doesn't seem to have slowed down iPhone sales.

Siri mostly hit the expectations they themselves were able to set through their ads when launching that product - having a voice based assistant at all was huge back then. With an LLM-based assistant, the market has set the expectations for them and they are just unreasonably high and don't mirror reality. That's a potentially big trap for Apple now.

Re: Building a personal, private AI computer on a budget

#146

Post by joshstrange »

nickthegreek wrote: post #62
joshstrange wrote: post #59

Earlier quoted context omitted.

Yeah, really I'd love for my Home Assistant to be able to use a local LLM/TTS/STT which I did get working but was way too slow. Also it would fun to just throw some problems/ideas at the wall without incurring (more) cost, that's a big part of it. But each time I run the numbers I would be better off using Anthropic/OpenAI/DeepSeek/other. I think sooner or later I'll break down and buy a server for local inference ev…

You shouldn't be having slow response issues with LLM/TTS/STT for HA on a mbp m3 max 128gb. I'd either limit the entities exposed or choose a smaller model.

Oh, I can get smaller models to run reasonably fast but I'm very interested in tool calling and I'm having a hard time finding a model that runs fast and is good at calling tools locally (I'm sure that's due to my own ignorance).

Re: Building a personal, private AI computer on a budget

#147

Post by oofbaroomf »

jmyeet wrote: post #17

The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac M…

Unified memory is great because it's fast, but you can also get a lot of system memory on a "conventional" machine like OP's, and offload MOE layers like what Ktransformers did, so you can run huge models with acceptable speeds. While the Mac mini may have better value for anything that fits in the unified memory, if you want to run Deepseek R1 or other large models, then it's best to max out system RAM and get a GPU to offload.

Re: Building a personal, private AI computer on a budget

#148

Post by mock-possum »

miniwark wrote: post #107

2 x Nvidia Tesla P40 card for €660 is not a thing i consider to be "on a budget". People can play with "small" or "medium" models less powerfull and cheaper cards. A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market (and they are around 300~350 new). In my opinion, 48Gb of VRAM is overkill to call it "on a budget", for me this setup is nice but it's for semi-profess…

Yeesh, yeah, that was my first thought too - who’s budget??

less than $500 total feels more fitting as a ‘budget’ build - €1700 is more along the lines of ‘enthusiast’ or less charitably “I am rich enough to afford expensive hobbies”

If it’s your business and you expect to recoup the cost and write off the cost on your taxes, that’s one thing - but if you’re just looking to run a personal local LLM for funnies, that’s not an accessible price tag.

I suppose “or you could just buy a Mac” should have tipped me off though.

Re: Building a personal, private AI computer on a budget

#149

Post by moffkalast »

joshstrange wrote: post #31

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past. I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I fee…

A Strix Halo minipc might be a good mid tier option once they're out, though AMD still isn't clear on how much they'll overprice them.

Core Ultra Arc iGPU boxes are pretty neat too for being standalone and can be loaded up with DDR5 shared memory, efficient and usable in terms of speed, though that's definitely low end performance, plus SYCL and IPEX are a bit eh.

Re: Building a personal, private AI computer on a budget

#150

Post by Capricorn2481 »

motorest wrote: post #105
kamranjon wrote: post #70

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

> For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. Around half that price tag was attributed to the blogger reusing an old workstation he had lying around. Beyond this point, OP slapped two graphics cards into an old rig. A better description would be something like "what buying two graphics cards gets you in terms of AI".

> Beyond this point, OP slapped two graphics cards into an old rig

Meaning what? This is largely what you do on a budget since RAM is such a difference maker in token generation. This is what's recommended. OP could buy an a100, but that wouldn't be a budget build.

Post reply on HN