Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

81–90 of 236 posts

Re: Building a personal, private AI computer on a budget

#81

Post by CamperBob2 »

reacharavindh wrote: post #3

The thing is though.... the locally hosted models in such hardware are cute as toys, and sure do write funny jokes and importantly, perform private tasks that I would never consider passing to non-selfhosted models, but pale in comparison to the models accessible over APIs(Claude 3.5 Sonnet, OpenAI etc). If I could run deepseek-r1-678b locally, without breaking the bank, I would. But, for now, opex > capex at a consu…

The 1.58-bit DeepSeek R1 dynamic quant model from Unsloth is no joke. It just needs a lot of RAM and some patience.

Re: Building a personal, private AI computer on a budget

#82

Post by diggan »

redman25 wrote: post #54
diggan wrote: post #27

Earlier quoted context omitted.

How is the performance difference between using a dedicated GPU from Nvidia for example compared to whatever Apple does? So lets say we'd run a model on a Mac Mini M4 with 24GB RAM, how many tokens/s are you getting? Then if we run the exact same model but with a RTX 3090ti for example, how many tokens/s are you getting? Do these comparisons exist somewhere online already? I understand it's possible to run the model…

Not the exact same comparison but I have an M1 mac with 16gb ram and can get about 10 t/s with a 3B model. The same model on my 3060ti gets more than 100 t/s. Needless to say, ram isn't everything.

Could you say what exact model+quant you're using for that specific test + settings + runtime? Just so I could try to compare with other numbers I come across.

Re: Building a personal, private AI computer on a budget

#84

Post by CamperBob2 »

elorant wrote: post #20
walterbell wrote: post #9

Earlier quoted context omitted.

200+ comments, https://news.ycombinator.com/item?id=42897205 > This runs the 671B model in Q4 quantization at 3.5-4.25 TPS for $2K on a single socket Epyc server motherboard using 512GB of RAM.

Runs is an overstatement though. With 4 tokens/second you can't use it on production.

What I'd like to know is how well those dual-Epyc machines run the 1.58 bit dynamic quant model. It really does seem to be almost as good as the full Q8.

Re: Building a personal, private AI computer on a budget

#86

Post by diggan »

kamranjon wrote: post #70

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

> likely could outperform this setup in terms of tokens per second

I've heard arguments both for and against this, but they always lack concrete numbers.

I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.

Re: Building a personal, private AI computer on a budget

#87

Post by xienze »

JKCalhoun wrote: post #79
xienze wrote: post #56

Earlier quoted context omitted.

The price goes up dramatically once you go past 12GB though, that’s the problem.

Not on these server GPUs. I'm seeing 24GB M40 cards for $200, 24GB K80 cards for $40 on eBay.

Well OK, I should have been more specific that, even for server GPUs on eBay:

* Cheap

* Fast

* Decent amount of RAM

Pick two.

These old GPUs are as cheap as they are because they don’t perform well.

Re: Building a personal, private AI computer on a budget

#88

Post by ekianjo »

kamranjon wrote: post #70

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

Mac Mini will be very slow for context ingestion compared to nvidia GPU, and the other issue is that they are not usable for Stable Diffusion... So if you just want to use LLMs, maybe, but if you have other interests in AI models, probably not the right answer.

Re: Building a personal, private AI computer on a budget

#89

Post by ekianjo »

refibrillator wrote: post #64

Pay attention to IO bandwidth if you’re building a machine with multiple GPUs like this! In this setup the model is sharded between cards so data must be shuffled through a PCIe 3.0 x16 link which is limited to ~16 GB/s max. For reference that’s an order of magnitude lower than the ~350 GB/s memory bandwidth of the Tesla P40 cards being used. Author didn’t mention NVLink so I’m presuming it wasn’t used, but I believe…

> Author didn’t mention NVLink so I’m presuming it wasn’t used, but I believe these cards would support it.

How would you setup NVLink, if the cards support it?

Re: Building a personal, private AI computer on a budget

#90

Post by michaelt »

apples_oranges wrote: post #12

Does using 2x24GB VRAM mean that the model can be fully loaded into memory if it's between 24 and 48 GB in size? I somehow doubt it, at least ollama wouldn't work like that I think. But does anyone know?

For a great many LLMs, you can find someone on HuggingFace who has produced a set of different quantised versions, with approximate RAM requirements.

For example, if you want to run "CodeLlama 70B" from https://huggingface.co/TheBloke/CodeLlama-70B-Python-GGUF where's a table saying the "Q4_K_M" quantised version is a 41.42 GB download and runs in 43.92 GB of memory.

Post reply on HN