Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

111–120 of 236 posts

Re: Building a personal, private AI computer on a budget

#111

Post by cratermoon »

From the article: In the future, I fully expect to be able to have a frank and honest discussion about the Tiananmen events with an American AI agent, but the only one I can afford will have assumed the persona of Father Christmas who, while holding a can of Coca-Cola, will intersperse the recounting of the tragic events with a joyful "Ho ho ho... Didn't you know? The holidays are coming!"

How unfortunate that people are discounting the likelihood that American AI agents will avoid saying things their master think should not be said. Anyone want to take bets on when the big 3 (Open AI, Meta, and Google) will quietly remove anything to do with DEI, trans people, or global warming? They'll start out changing all mentions of "Gulf of Mexico" to "Gulf of America", but then what?

Re: Building a personal, private AI computer on a budget

#112

Post by diggan »

un_ess wrote: post #109
diggan wrote: post #86

Earlier quoted context omitted.

> likely could outperform this setup in terms of tokens per second I've heard arguments both for and against this, but they always lack concrete numbers. I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.

Per the screenshot, this is a DeepSeek running on a 192GB M2 Studio https://nitter.poast.org/ggerganov/status/188461277009384272... The same on Nvidia (various models) https://github.com/ggerganov/llama.cpp/issues/11474 [1] this is a the model: https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/De...

So Apple M2 Studio does ~15 tks/second and A100-SXM4-80GB does 9 tks/second?

I'm not sure I'm reading the results wrong or missing some vital context, but that sounds unlikely to me.

Re: Building a personal, private AI computer on a budget

#113

Post by diggan »

fkyoureadthedoc wrote: post #110
diggan wrote: post #106

Earlier quoted context omitted.

> gemma2:27b What quantization are you using? What's the runtime+version you run this with? And the rest of the settings? Edit: Turns out parent is using Q4 for their test. Doing the same test with LM Studio and a 3090ti + Ryzen 5950X (with 44 layers on GPU, 2 on CPU) I get ~15 tokens/second.

Fresh install from brew, ollama version is 0.5.7 Only settings I did were the ones shown in the blog post OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 Ran the model like ollama run gemma2:27b --verbose With the same prompt, "Can you write me a story about a tortoise and a hare, but one that involves a race to get the most tokens per second?"

When you run that, what quantization do you get? The library website of Ollama (https://ollama.com/library/gemma2:27b) isn't exactly a good use case in surfacing useful information like what the default quantization is.

Re: Building a personal, private AI computer on a budget

#114

Post by egorfine »

axegon_ wrote: post #10

I did something similar but using a K80 and M40 I dug up from eBay for pennies. Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. That said, I had a decent-ish HP workstation laying around with 1200 watt power supply so I had where to put those two in. The one thing…

> K80 - the drivers were one of the most painful tech things I've ever had to endure

Well, for a dedicated LLM box it might be feasible to suffer with drivers a bit, no? What was your experience like with the software side?

Re: Building a personal, private AI computer on a budget

#115

Post by cratermoon »

reacharavindh wrote: post #3

The thing is though.... the locally hosted models in such hardware are cute as toys, and sure do write funny jokes and importantly, perform private tasks that I would never consider passing to non-selfhosted models, but pale in comparison to the models accessible over APIs(Claude 3.5 Sonnet, OpenAI etc). If I could run deepseek-r1-678b locally, without breaking the bank, I would. But, for now, opex > capex at a consu…

This is not because the models are better. These services have unknown and opaque levels of shadow prompting[1] to tweak the behavior. The subject article even mentions "tweaking their outputs to the liking of whoever pays the most". The more I play with LLMs locally, the more I realize how much prompting going on under the covers is shaping the results from the big tech services.

1 https://www.techpolicy.press/shining-a-light-on-shadow-promp...

Re: Building a personal, private AI computer on a budget

#116

Post by [deleted] »

diggan wrote: post #86
kamranjon wrote: post #70

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

> likely could outperform this setup in terms of tokens per second I've heard arguments both for and against this, but they always lack concrete numbers. I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.

[deleted]

Re: Building a personal, private AI computer on a budget

#117

Post by birktj »

I was wondering if anyone here has experimented with running a cluster of SBC for LLM inference? Ex. the Radxa ROCK 5C has 32GB of memory and also a NPU and only costs about 300 euros. I'm not super up to date on the architecture on modern LLMs, but as far as I understand you should be able to split the layers between multiple nodes? It is not that much data the needs to be sent between them, right? I guess you won't get quite the same performance as a modern mac or nvidia GPU, but it could be quite acceptable and possibly a cheap way of getting a lot of memory.

On the other hand I am wondering about what is the state of the art in CPU + GPU inference. Prompt processing is both compute and memory constrained, but I think token generation afterwards is mostly memory bound. Are there any tools that support loading a few layers at a time into a GPU for initial prompt processing and then switches to CPU inference for token generation? Last time I experimented it was possible to run some layers on the GPU and some on the CPU, but to me it seems more efficient to run everything on the GPU initially (but a few layers at a time so they fit in VRAM) and then switch to the CPU when doing the memory bound token generation.

Re: Building a personal, private AI computer on a budget

#118

Post by oofbaroomf »

kamranjon wrote: post #70

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

The bottleneck for single batch inference is memory bandwidth. The M4 Pro has less memory bandwidth than the P40, so it would be slower. Also, the setup presented in the OP has system RAM, allowing you to run models than what fits in 48GB of VRAM (and with good speeds too if you offload with something like ktransformers).

Re: Building a personal, private AI computer on a budget

#120

Post by fkyoureadthedoc »

diggan wrote: post #113
fkyoureadthedoc wrote: post #110

Earlier quoted context omitted.

Fresh install from brew, ollama version is 0.5.7 Only settings I did were the ones shown in the blog post OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 Ran the model like ollama run gemma2:27b --verbose With the same prompt, "Can you write me a story about a tortoise and a hare, but one that involves a race to get the most tokens per second?"

When you run that, what quantization do you get? The library website of Ollama ( https://ollama.com/library/gemma2:27b ) isn't exactly a good use case in surfacing useful information like what the default quantization is.

not sure how to tell, but here's the full output from ollama serve https://pastes.io/ollama-run-gemma2-27b
Post reply on HN