Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

211–220 of 236 posts

Re: Building a personal, private AI computer on a budget

#211

Post by GTP »

almostgotcaught wrote: post #207
yjftsjthsd-h wrote: post #205

Earlier quoted context omitted.

> Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. I thought the problem was that those cards have loads of RAM but lack really important compute capabilities such that they're kind of useless for actually running AI workloads on. Is that not the case?

> Is that not the case? it is - they're laughably slow and not even supported by latest CUDA > NVIDIA Driver support for Kepler is removed beginning with R495. CUDA Toolkit development support for Kepler continues through CUDA 11.x.

But Deepseek R1 doesn't use CUDA, so maybe for this specific case, it isn't a big deal?

Re: Building a personal, private AI computer on a budget

#212

Post by almostgotcaught »

GTP wrote: post #211
almostgotcaught wrote: post #207

Earlier quoted context omitted.

> Is that not the case? it is - they're laughably slow and not even supported by latest CUDA > NVIDIA Driver support for Kepler is removed beginning with R495. CUDA Toolkit development support for Kepler continues through CUDA 11.x.

But Deepseek R1 doesn't use CUDA, so maybe for this specific case, it isn't a big deal?

> it isn't a big deal?

friend you shouldn't make comments like this unless you understand the definitions of the words. Deepseek wrote some parts of their kernels using PTX. newsflash: PTX support for features is lockstep with CUDA support for the same features ie the fact that CUDA doesn't support it means you couldn't write the PTX to use those features either.

Re: Building a personal, private AI computer on a budget

#213

Post by IanCal »

deadbabe wrote: post #21
elorant wrote: post #20

Earlier quoted context omitted.

Runs is an overstatement though. With 4 tokens/second you can't use it on production.

Isn’t 4 tps good enough for local use by a single user, which is the point of a personal AI computer?

4 tokens per second is pretty slow. That's 5-10s for a comment the length of yours (and R1 specifically likes to output a lot of tokens). It's 10-20x slower than many top end models, which are available cheaply. Even high cost versions of R1 (at more than twice the price of sonnet) are $7/million tokens. For $2K you get 285 million tokens. You'd have to run the box at full whack for over two years (for 4tps) to hit that spending, and that ignores electricity prices. Sonnet 3.5 is half that price, and other R1 providers you could probably hit about a billion tokens for $2k. Gemini flash 2 is over 100 tokens per second and $2k gets you something like 5+B tokens (more really but I'm taking an easy estimate over the more expensive part).

If there are things you cannot send to a random party, you might want to look at hosted versions with agreements (if it's a code issue, if you're fine with github then azure is probably fine too).

Outside of that, if you really need to then sure, but these are the kinds of things that really benefit from being able to get high usage on GPUs for short periods of time.

Re: Building a personal, private AI computer on a budget

#214

Post by behohippy »

Eisenstein wrote: post #203
wkat4242 wrote: post #201

Earlier quoted context omitted.

Even at q8_0? I thought it wasn't bad just like the models itself. But very interested to hear. And q8_0 already halves the memory usage compared to fp16. One of the ollama Devs called the quality impact negligible at q8_0: https://smcleod.net/2024/12/bringing-k/v-context-quantisatio... But perhaps quantifying the KV cache does not scale as gracefully as the model itself?

It highly depends on the model and the context use. A model like command-r for instance is practically unaffected by it, but Qwen will go nuts. As well, tasks highly dependent on context like translation or evaluation will be more impacted than say, code generation or creative output.

Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too

Re: Building a personal, private AI computer on a budget

#215

Post by numpad0 »

magicalhippo wrote: post #39
numpad0 wrote: post #7

Earlier quoted context omitted.

Is it cheaper in $/GB than used Vega 56(HBM2 8GB) besides? There are mining boards with bunch of x1 slots that probably can run half a dozen of them for same 48GB.

AFAIK this doesn't really work for interactive use, as LLMs process data serially. So your request needs to pass through all of the cards for each token, one at a time. Thus a lot of PCIe traffic and hence latency. Better than nothing, but only really useful if you can batch requests so you can keep each GPU working all the time, rather than just one at a time.

Clearly I wasn't aware enough that DNN is by default like a mesh. Makes sense that it's going to be bottlenecked by the tightest link. Thanks...

Re: Building a personal, private AI computer on a budget

#216

Post by anon373839 »

idrathernot wrote: post #47
gregwebs wrote: post #24

The problem for me with making such an investment is that next month a better model will be released. It will either require more or less RAM than the current best model- making it either not runnable or expensive to run on an overbuilt machine. Using cloud infrastructure should help with this issue. It may cost much more per run but money can be saved if usage is intermittent. How are HN users handling this?

There is also an overlooked “tail risk” with cloud services that can end up costing you more than a a few entire on-premise rigs if you don’t correctly configure services or forget to shut down a high end vm instance. Yeah you can implement additional scripts and services as a fail-safe, but this adds another layer of complexity that isn’t always trivial (especially for a hobbyist). I’m not saying that dumping $10k i…

Yes. And somehow, those cloud providers just can’t seem to work out how to build a spend limit feature for customers who’d like to prevent that. It must be a really difficult engineering problem…

Re: Building a personal, private AI computer on a budget

#217

Post by Eisenstein »

behohippy wrote: post #214
Eisenstein wrote: post #203

Earlier quoted context omitted.

It highly depends on the model and the context use. A model like command-r for instance is practically unaffected by it, but Qwen will go nuts. As well, tasks highly dependent on context like translation or evaluation will be more impacted than say, code generation or creative output.

Qwen is a little fussy about the sampler settings, but it does run well quantized. If you were getting infinite repetition loops, try dropping the top_p a bit. I think qwen likes lower temps too

We are talking about dynamically quantizing KV cache, not the model weights.

Re: Building a personal, private AI computer on a budget

#220

Post by troyvit »

vladgur wrote: post #132
diggan wrote: post #86

Earlier quoted context omitted.

> likely could outperform this setup in terms of tokens per second I've heard arguments both for and against this, but they always lack concrete numbers. I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.

as someone who is paying $0.50 per kwh, id also like to include kw per 1000 tokens or something to give me a sense of cost of ownership these local systems

That would be an awesome thing across the industry -- even for the big commercial models -- for those who care not only about price but also carbon footprint.
Post reply on HN