Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

171–180 of 236 posts

Re: Building a personal, private AI computer on a budget

#171

Post by navbaker »

diggan wrote: post #163
navbaker wrote: post #155

Earlier quoted context omitted.

If you hit the drop-down menu for the size of the model, then tap “view all”, you will see the size and hash of the model you have selected and can compare it to the full list below it that has the quantization specs in the name.

Still, I don't see a way (from the web library) to see the default quantization (from Ollama's POV) at all, is that possible somehow?

The model displayed in the drop-down when you access the web library is the default that will be pulled. Compare the size and hash to the more detailed model listing below it and you will see what quantization you have.

Example: the default model weights for Llama 3.3 70b, after hitting the “view all” have this hash and size listed next to it - a6eb4748fd29 • 43GB

Now scroll down through the list and you will find the one that matches that hash and size is “70b-instruct-q4_K_M”. That tells you that the default weights for Llama 3.3 70B from Ollama are 4-bit quantized (q4) while the “K_M” tells you a bit about what techniques were used during quantization to balance size and performance.

Re: Building a personal, private AI computer on a budget

#172

Post by unshavedyak »

diggan wrote: post #133
GTP wrote: post #124

Earlier quoted context omitted.

Yes, but we don't have enough people doing that to get quality data. Not many people are building this kind of setup, and even less are publishing their results. Additionally, if I just run a test a couple of time and then average the results, this is still far from a solid measurement.

> but we don't have enough people doing that to get quality data But how are we supposed to get enough people doing those things if everyone say "There isn't enough data right now for it to be useful"? We have to start somewhere

I don't think they're saying anything counter to that. The people who don't require the volume of data will run these. Ie the super early adopters.

Re: Building a personal, private AI computer on a budget

#173

Post by lolinder »

gmueckl wrote: post #138
cwoolfe wrote: post #136

As others have said, a high powered Mac could be used for the same purpose at a comparable price and lower power usage. Which makes me wonder: why doesn't Apple get into the enterprise AI chip game and compete with Nvidia? They could design their own ASIC for it with all their hardware & manufacturing knowledge. Maybe they already are.

The primary market for such a product would be businesses. And Apple isn't particularly good at selling to companies. The consumer product focus may just be too ingrained to be successful with such a move. A beefed up home pod with a local LLM-based assistant would be a more typical Apple product. But they'd probably need LLMs to become much, much more reliable to not ruin their reputation over this.

> And Apple isn't particularly good at selling to companies.

With a big glaring exception: developer laptops are overwhelmingly Apple's game right now. It seems like they should be able to piggyback off of that, given that the decision makers are going to be in the same branch of the customer company.

Re: Building a personal, private AI computer on a budget

#174

Post by unshavedyak »

xienze wrote: post #55
mechagodzilla wrote: post #35

Earlier quoted context omitted.

I have a similar setup running at about 1.5 tokens/second, and it's perfectly usable for the sorts of difficult tasks one needs a frontier model like this for - give it a prompt and come back an hour or two later. You interact with it like e-mailing a coworker. If I need an answer back in seconds, it's probably not a very complicated question, and a much smaller model will do.

I get where you’re coming from, but the problem with LLMs is that you very regularly need a lot of back-and-forth with them to tease out the information you’re looking for. A more apt analogy might be a coworker that you have to follow up with three or four times, at an hour per. Not so appealing anymore. Doubly so when you have to stand up $2k+ of hardware for the privilege. If I’m paying good money to host somethin…

Agreed. Furthermore, for some tasks like large context code assistant windows i want really fast responses. I've not found a UX i'm happy with yet but for anything i care about i'd want very fast token responses. Small blocks of code which instantly autocomplete, basically.

Re: Building a personal, private AI computer on a budget

#176

Post by Eisenstein »

whywhywhywhy wrote: post #130
miniwark wrote: post #107

2 x Nvidia Tesla P40 card for €660 is not a thing i consider to be "on a budget". People can play with "small" or "medium" models less powerfull and cheaper cards. A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market (and they are around 300~350 new). In my opinion, 48Gb of VRAM is overkill to call it "on a budget", for me this setup is nice but it's for semi-profess…

> A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market 1080Ti might even be a better option, it also has a 12gb model and some reports say it even outperforms the 3060, in non-rtx I presume.

CUDA compute version is a big deal. 1080ti is 6.1. 3060 is 8.6. It also has tensor cores.

Note that CUDA version numbers are confusing, the compute number is a different thing than the runtime/driver version.

Re: Building a personal, private AI computer on a budget

#177

Post by colonCapitalDee »

diggan wrote: post #133
GTP wrote: post #124

Earlier quoted context omitted.

Yes, but we don't have enough people doing that to get quality data. Not many people are building this kind of setup, and even less are publishing their results. Additionally, if I just run a test a couple of time and then average the results, this is still far from a solid measurement.

> but we don't have enough people doing that to get quality data But how are we supposed to get enough people doing those things if everyone say "There isn't enough data right now for it to be useful"? We have to start somewhere

We've already started, we just haven't finished yet

Re: Building a personal, private AI computer on a budget

#178

Post by Eisenstein »

birktj wrote: post #117

I was wondering if anyone here has experimented with running a cluster of SBC for LLM inference? Ex. the Radxa ROCK 5C has 32GB of memory and also a NPU and only costs about 300 euros. I'm not super up to date on the architecture on modern LLMs, but as far as I understand you should be able to split the layers between multiple nodes? It is not that much data the needs to be sent between them, right? I guess you won't…

> I was wondering if anyone here has experimented with running a cluster of SBC for LLM inference? Ex. the Radxa ROCK 5C has 32GB of memory and also a NPU and only costs about 300 euros.

Look into RPC. Llama.cpp supports it.

* https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacp...

> Last time I experimented it was possible to run some layers on the GPU and some on the CPU, but to me it seems more efficient to run everything on the GPU initially (but a few layers at a time so they fit in VRAM) and then switch to the CPU when doing the memory bound token generation.

Moving layers over the PCIe bus to do this is going to be slow, which seems to be the issue with that strategy. I think it the key is to use MoE and be smart about which layers go where. This project seems to be doing that with great results:

* https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...

Re: Building a personal, private AI computer on a budget

#179

Post by Melatonic »

whywhywhywhy wrote: post #130
miniwark wrote: post #107

2 x Nvidia Tesla P40 card for €660 is not a thing i consider to be "on a budget". People can play with "small" or "medium" models less powerfull and cheaper cards. A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market (and they are around 300~350 new). In my opinion, 48Gb of VRAM is overkill to call it "on a budget", for me this setup is nice but it's for semi-profess…

> A Nvidia Geforce RTX 3060 card with "only" 12Gb VRAM can be found around €200-250 on second hand market 1080Ti might even be a better option, it also has a 12gb model and some reports say it even outperforms the 3060, in non-rtx I presume.

Not sure what used prices are like these days but the Titan XP (similar to the 1080 ti) is even better

Re: Building a personal, private AI computer on a budget

#180

Post by MonkeyClub »

xienze wrote: post #55
mechagodzilla wrote: post #35

Earlier quoted context omitted.

I have a similar setup running at about 1.5 tokens/second, and it's perfectly usable for the sorts of difficult tasks one needs a frontier model like this for - give it a prompt and come back an hour or two later. You interact with it like e-mailing a coworker. If I need an answer back in seconds, it's probably not a very complicated question, and a much smaller model will do.

I get where you’re coming from, but the problem with LLMs is that you very regularly need a lot of back-and-forth with them to tease out the information you’re looking for. A more apt analogy might be a coworker that you have to follow up with three or four times, at an hour per. Not so appealing anymore. Doubly so when you have to stand up $2k+ of hardware for the privilege. If I’m paying good money to host somethin…

> If I’m paying good money to host something locally

The thing is, however, that at 2k one is not paying good money, one is paying near the least amount possible. TFA specifically is about building a machine on a budget, and as such cuts corners to save costs, e.g. by buying older cards.

Just because 2k is not a negligible amount in itself, that doesn't also automatically make it adequate for the purpose. Look for example at the 15k, 25k, and 40k price range tinyboxes:

https://tinygrad.org/#tinybox

It's like buying a 2k-worth used car, and expecting it to perform as well as a 40k one.

Post reply on HN