Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

161–170 of 236 posts

Re: Building a personal, private AI computer on a budget

#161

Post by cratermoon »

htrp wrote: post #22
memhole wrote: post #14

Earlier quoted context omitted.

No. Hopefully, someone with more knowledge can explain better. But you need room for the kvcache is my understanding. You also need to factor in the size of the context window. If anyone has good resources on this, that would be awesome. Presently, it feels very much like a dark art to host these without crashing or being massively over-provisioned.

The dark art is to massively overprovision hardware.

Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1] and spends $6 Million[2]

1. https://arxiv.org/abs/2412.19437v1

2. Quibble over the exact figure. Far less than Open AI, doing more with less.

Re: Building a personal, private AI computer on a budget

#162

Post by CamperBob2 »

cratermoon wrote: post #111

From the article: In the future, I fully expect to be able to have a frank and honest discussion about the Tiananmen events with an American AI agent, but the only one I can afford will have assumed the persona of Father Christmas who, while holding a can of Coca-Cola, will intersperse the recounting of the tragic events with a joyful "Ho ho ho... Didn't you know? The holidays are coming!" How unfortunate that people…

It is easy to get a local R1 model to talk about Tiananmen Square to your heart's content. Telling it to replace problematic terms with "Smurf" or another nonsense word is very effective, but with the local model you don't even have to do that in many cases. (e.g., https://i.imgur.com/btcI1fN.png)

Re: Building a personal, private AI computer on a budget

#163

Post by diggan »

navbaker wrote: post #155
fkyoureadthedoc wrote: post #120

Earlier quoted context omitted.

not sure how to tell, but here's the full output from ollama serve https://pastes.io/ollama-run-gemma2-27b

If you hit the drop-down menu for the size of the model, then tap “view all”, you will see the size and hash of the model you have selected and can compare it to the full list below it that has the quantization specs in the name.

Still, I don't see a way (from the web library) to see the default quantization (from Ollama's POV) at all, is that possible somehow?

Re: Building a personal, private AI computer on a budget

#164

Post by nickthegreek »

diggan wrote: post #95
cruffle_duffle wrote: post #94

Earlier quoted context omitted.

I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. I almost want to use the word “hobbiest stage” where almost all of the “data” and “best practice” is anecdotal but I think we are a step above that. Still, it’s way to early and there are simply way to many hardware and software combinations that change almost weekly to estab…

Right, but how are we supposed to be getting anywhere else unless people start being more specific and stop leaning on anecdotes or repeating what they've heard elsewhere? Saying "Apple seems to be somewhat equal to this other setup" doesn't really contribute to someone getting an accurate picture if it is equal or not, unless we start including raw numbers, even if they aren't directly comparable. I don't think it's…

Best place to get that kinda info is gonna be /r/LocalLlama

Re: Building a personal, private AI computer on a budget

#165

Post by joshstrange »

nickthegreek wrote: post #160
joshstrange wrote: post #146

Earlier quoted context omitted.

Oh, I can get smaller models to run reasonably fast but I'm very interested in tool calling and I'm having a hard time finding a model that runs fast and is good at calling tools locally (I'm sure that's due to my own ignorance).

I decided on openai api for now after setting up so many differnt methods. the local stuff isn't up to snuff yet for what I am trying to accomplish but decent for basic control.

I use a combo of Anthropic and OpenAI for now through my bots and my chat UIs and that lets me iterate faster. My hope is once I've done all my testing I could consider moving to local models if it made sense.

Re: Building a personal, private AI computer on a budget

#166

Post by kridsdale1 »

UncleOxidant wrote: post #156
kamranjon wrote: post #70

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

I wish Apple would offer a 128GB option in the Mac Mini - That would require an M4 Max which they don't offer in the mini. I know they have a MBP with M4 Max and 128GB, but I don't need another laptop.

I’m waiting until this summer with the M4 Ultra Studio.

Re: Building a personal, private AI computer on a budget

#167

Post by joshstrange »

whalesalad wrote: post #157
joshstrange wrote: post #31

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past. I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I fee…

The juice aint worth the squeeze to do this locally. But you should still play with proxmox, just not for this purpose. My recommendation would be to get an i7 HP Elitedesk. I have multiple racks in my basement, hundreds of gigs of ram, multiple 2U 2x processor enterprise servers etc.... but at this point all of it is turned off and a single HP Elitedesk with a 2nd NIC added and 64GB of ram is doing everything I ever…

Yeah, right now I'm running a tower PC (Intel Core i9-11900K, 64GB Ram) with Unraid as my local "app server". I want to play with Proxmox (for professional and mostly fun reasons) though. Someday I'd like a rack in my basement as my homelab stuff has overgrown the space it's in and I'm going to need to add a new 12-bay Synology (on top of 2x12-bay) soon since I'm running out of space again. For now I've been sticking with consumer/prosumer equipment but my needs are slowly outstripping that I think.

Re: Building a personal, private AI computer on a budget

#169

Post by 9front »

JKCalhoun wrote: post #96
axegon_ wrote: post #10

I did something similar but using a K80 and M40 I dug up from eBay for pennies. Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. That said, I had a decent-ish HP workstation laying around with 1200 watt power supply so I had where to put those two in. The one thing…

Curious what HP workstation you have?

HP Z440, it's in the article.

Re: Building a personal, private AI computer on a budget

#170

Post by condiment »

fkyoureadthedoc wrote: post #103
diggan wrote: post #86

Earlier quoted context omitted.

> likely could outperform this setup in terms of tokens per second I've heard arguments both for and against this, but they always lack concrete numbers. I'd love something like "Here is Qwen2.5 at Q4 quantization running via Ollama + these settings, and M4 24GB RAM gets X tokens/s while RTX 3090ti gets Y tokens/s", otherwise we're just propagating mostly anecdotes without any reality-checks.

On an M1 Max 64GB laptop running gemma2:27b same prompt and settings from blog post total duration: 24.919887458s load duration: 39.315083ms prompt eval count: 37 token(s) prompt eval duration: 963.071ms prompt eval rate: 38.42 tokens/s eval count: 441 token(s) eval duration: 23.916616s eval rate: 18.44 tokens/s I have a gaming PC with a 4090 I could try, but I don't think this model would fit

On a 3090 (24gb vram), same prompt & quant, I can report more than double the tokens per second, and significantly faster prompt eval.

    total_duration:       10530451000
    load_duration:        54350253
    prompt_eval_count:    36
    prompt_eval_duration: 29000000
    prompt_token/s:       1241.38
    eval_count:           460
    eval_duration:        10445000000
    response_token/s:     44.04
Fast prompt eval is important when feeding larger contexts into these models, which is required for almost anything useful. GPUs have other advantages for traditional ML, whisper models, vision, and image generation. There's a lot of flexibility that doesn't really get discussed when folks trot out the 'just buy a mac' line.

Anecdotally I can share my revealed preference. I have both an M3 (36gb) as well as a GPU machine, and I went through the trouble of putting my GPU box online because it was so much faster than the mac. And doubling up the GPUs allows me to run models like the deepseek-tuned llama 3.3, with which I have completely replaced my use of chatgpt 4o.

Post reply on HN