Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

121–130 of 236 posts

Re: Building a personal, private AI computer on a budget

#121

Post by cwalv »

joshstrange wrote: post #31

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past. I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I fee…

> but every time I price out a beefier box I feel like the ROI just isn’t there, especially for an industry that is moving so fast.

Same, esp. if you factor in the cost of renting. Even if you run 24/7 it's hard to see it paying off in half the time it will take to be obsolete

Re: Building a personal, private AI computer on a budget

#122

Post by diggan »

fkyoureadthedoc wrote: post #120
diggan wrote: post #113

Earlier quoted context omitted.

When you run that, what quantization do you get? The library website of Ollama ( https://ollama.com/library/gemma2:27b ) isn't exactly a good use case in surfacing useful information like what the default quantization is.

not sure how to tell, but here's the full output from ollama serve https://pastes.io/ollama-run-gemma2-27b

Thanks, that seems to indicate Q4 for the quantization, you're probably able to run that on the 4090 as well FWIW, the size of the model is just 14.55 GiB.

Re: Building a personal, private AI computer on a budget

#123

Post by dandanua »

mrbonner wrote: post #77
dandanua wrote: post #45

I doubt it is that efficient. Even though it has 48GB of VRAM, it's more than twice slower than a single 3090 GPU. In my budget AI setup I use 7840 Ryzen based miniPC with USB4 port and connect 3090 to it via the eGPU adapter (ADT-link UT3G). It costed me about $1000 total and I can easily achieve 35 t/s with qwen2.5-coder-32b using ollama.

Wouldn't eGPU defeat the purpose of having fast memory bandwidth? Have you tried it with stable diffusion?

40Gbps of USB4 is plenty. I've tried this pytorch tests https://github.com/aime-team/pytorch-benchmarks/ and saw only 10% drop in performance. No drop in performance for LLM inference, if a model is already loaded to the VRAM.

Re: Building a personal, private AI computer on a budget

#124

Post by GTP »

motorest wrote: post #102
cruffle_duffle wrote: post #94

Earlier quoted context omitted.

I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. I almost want to use the word “hobbiest stage” where almost all of the “data” and “best practice” is anecdotal but I think we are a step above that. Still, it’s way to early and there are simply way to many hardware and software combinations that change almost weekly to estab…

> I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. What's hard about it? You get the hardware, you run the software, you take measurements.

Yes, but we don't have enough people doing that to get quality data. Not many people are building this kind of setup, and even less are publishing their results. Additionally, if I just run a test a couple of time and then average the results, this is still far from a solid measurement.

Re: Building a personal, private AI computer on a budget

#125

Post by t1amat »

diggan wrote: post #108
kamranjon wrote: post #104

Earlier quoted context omitted.

A great thread with the type of info your looking for lives here: https://github.com/ggerganov/whisper.cpp/issues/89 But you can likely find similar threads for the llama.cpp benchmark here: https://github.com/ggerganov/llama.cpp/tree/master/examples/... These are good examples because the llama.cpp and whisper.cpp benchmarks take full advantage of the Apple hardware but also take full advantage of non-Apple hardware…

I guess I'm mostly lamenting about how unscientific these discussions are in general, on HN and elsewhere (besides specific GitHub repositories). Every community is filled with just anecdotal stories, or some numbers but missing to specify a bunch of settings + model + runtime details so people could at least compare it to something. Still, thanks for the links :)

In fairness it’s become even more difficult now than ever before.

* hardware spec

* inference engine

* specific model - differences to tokenizer will make models faster/slower with equivalent parameter count

* quantization used - and you need to be aware of hardware specific optimizations for particular quants

* kv cache settings

* input context size

* output token count

This is probably not a complete list either.

Re: Building a personal, private AI computer on a budget

#126

Post by ollybee »

The middle ground is to rent a GPU VPS as needed. You can get an H100 for $2/h. Not quite the same privacy as fully local offline, but better than a SASS API and good enough for me. Hopefully in a year or three it will truly be cost effective to run something useful locally and then I can switch.

Re: Building a personal, private AI computer on a budget

#128

Post by drcongo »

ekianjo wrote: post #88
kamranjon wrote: post #70

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

Mac Mini will be very slow for context ingestion compared to nvidia GPU, and the other issue is that they are not usable for Stable Diffusion... So if you just want to use LLMs, maybe, but if you have other interests in AI models, probably not the right answer.

I use a Mac Studio for Stable Diffusion, what's special about the Mac Mini that means it won't work?

Re: Building a personal, private AI computer on a budget

#130

Post by whywhywhywhy »

miniwark wrote: post #107

2 x Nvidia Tesla P40 card for €660 is not a thing i consider to be "on a budget". People can play with "small" or "medium" models less powerfull and cheaper cards. A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market (and they are around 300~350 new). In my opinion, 48Gb of VRAM is overkill to call it "on a budget", for me this setup is nice but it's for semi-profess…

> A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market

1080Ti might even be a better option, it also has a 12gb model and some reports say it even outperforms the 3060, in non-rtx I presume.

Post reply on HN