Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

201–210 of 236 posts

Re: Building a personal, private AI computer on a budget

#201

Post by wkat4242 »

Eisenstein wrote: post #192
wkat4242 wrote: post #187

Earlier quoted context omitted.

As far as I understand it does if you quantify the K/V store as well (the context). And that's pretty standard now because it can increase maximum context size a lot.

It is available in most inference engines, but I wouldn't call it in standard use, as it can degrade quality tremendously.

Even at q8_0? I thought it wasn't bad just like the models itself. But very interested to hear.

And q8_0 already halves the memory usage compared to fp16.

One of the ollama Devs called the quality impact negligible at q8_0: https://smcleod.net/2024/12/bringing-k/v-context-quantisatio...

But perhaps quantifying the KV cache does not scale as gracefully as the model itself?

Re: Building a personal, private AI computer on a budget

#202

Post by vanillax »

3s wrote: post #63
vanillax wrote: post #60

Earlier quoted context omitted.

Huh? Toys? You can run DeepSeek 70b on 36GB ram Macbook pro.. You can run Phi4, Qwen2.5, or llama3.3. They work great for coding tasks

Yeah but as one of the replies points out the resulting tokens/second would be unusable in production environments

What? Literally use it at work to write code.

Re: Building a personal, private AI computer on a budget

#203

Post by Eisenstein »

wkat4242 wrote: post #201
Eisenstein wrote: post #192

Earlier quoted context omitted.

It is available in most inference engines, but I wouldn't call it in standard use, as it can degrade quality tremendously.

Even at q8_0? I thought it wasn't bad just like the models itself. But very interested to hear. And q8_0 already halves the memory usage compared to fp16. One of the ollama Devs called the quality impact negligible at q8_0: https://smcleod.net/2024/12/bringing-k/v-context-quantisatio... But perhaps quantifying the KV cache does not scale as gracefully as the model itself?

It highly depends on the model and the context use. A model like command-r for instance is practically unaffected by it, but Qwen will go nuts. As well, tasks highly dependent on context like translation or evaluation will be more impacted than say, code generation or creative output.

Re: Building a personal, private AI computer on a budget

#204

Post by mrbonner »

dandanua wrote: post #123
mrbonner wrote: post #77

Earlier quoted context omitted.

Wouldn't eGPU defeat the purpose of having fast memory bandwidth? Have you tried it with stable diffusion?

40Gbps of USB4 is plenty. I've tried this pytorch tests https://github.com/aime-team/pytorch-benchmarks/ and saw only 10% drop in performance. No drop in performance for LLM inference, if a model is already loaded to the VRAM.

Wow that makes sense now if you can load the entire model to vram. What eGPU dock and GPU setup you use if you don’t mind?

Re: Building a personal, private AI computer on a budget

#205

Post by yjftsjthsd-h »

axegon_ wrote: post #10

I did something similar but using a K80 and M40 I dug up from eBay for pennies. Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. That said, I had a decent-ish HP workstation laying around with 1200 watt power supply so I had where to put those two in. The one thing…

> Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing.

I thought the problem was that those cards have loads of RAM but lack really important compute capabilities such that they're kind of useless for actually running AI workloads on. Is that not the case?

Re: Building a personal, private AI computer on a budget

#206

Post by jt_b »

vanillax wrote: post #202
3s wrote: post #63

Earlier quoted context omitted.

Yeah but as one of the replies points out the resulting tokens/second would be unusable in production environments

What? Literally use it at work to write code.

I think they're talking about using it to power inference for self hosted user facing applications.

Re: Building a personal, private AI computer on a budget

#207

Post by almostgotcaught »

yjftsjthsd-h wrote: post #205
axegon_ wrote: post #10

I did something similar but using a K80 and M40 I dug up from eBay for pennies. Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. That said, I had a decent-ish HP workstation laying around with 1200 watt power supply so I had where to put those two in. The one thing…

> Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. I thought the problem was that those cards have loads of RAM but lack really important compute capabilities such that they're kind of useless for actually running AI workloads on. Is that not the case?

> Is that not the case?

it is - they're laughably slow and not even supported by latest CUDA

> NVIDIA Driver support for Kepler is removed beginning with R495. CUDA Toolkit development support for Kepler continues through CUDA 11.x.

Re: Building a personal, private AI computer on a budget

#208

Post by wkat4242 »

Eisenstein wrote: post #203
wkat4242 wrote: post #201

Earlier quoted context omitted.

Even at q8_0? I thought it wasn't bad just like the models itself. But very interested to hear. And q8_0 already halves the memory usage compared to fp16. One of the ollama Devs called the quality impact negligible at q8_0: https://smcleod.net/2024/12/bringing-k/v-context-quantisatio... But perhaps quantifying the KV cache does not scale as gracefully as the model itself?

It highly depends on the model and the context use. A model like command-r for instance is practically unaffected by it, but Qwen will go nuts. As well, tasks highly dependent on context like translation or evaluation will be more impacted than say, code generation or creative output.

Interesting. I didn't know that. I thought it was basically 'free' space saving. Would you know how llama3.1 fares by any chance?

Re: Building a personal, private AI computer on a budget

#209

Post by dandanua »

mrbonner wrote: post #204
dandanua wrote: post #123

Earlier quoted context omitted.

40Gbps of USB4 is plenty. I've tried this pytorch tests https://github.com/aime-team/pytorch-benchmarks/ and saw only 10% drop in performance. No drop in performance for LLM inference, if a model is already loaded to the VRAM.

Wow that makes sense now if you can load the entire model to vram. What eGPU dock and GPU setup you use if you don’t mind?

That ADT-link UT3G is the eGPU doc, you just need an ordinary PSU to power it and the GPU (I use an old one that I have). Though, to make everything work you might need time to fix quirks. E. g., I had to use "nvidia error 43 fixer" on Windows and to find a correct configuration on my Linux NixOS system (you need to load the Nvidia driver into the kernel in the boot process etc.). Here is how it looks https://imgur.com/a/qySDN4n
Post reply on HN