Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

51–60 of 236 posts

Re: Building a personal, private AI computer on a budget

#51

Post by 3s »

gregwebs wrote: post #24

The problem for me with making such an investment is that next month a better model will be released. It will either require more or less RAM than the current best model- making it either not runnable or expensive to run on an overbuilt machine. Using cloud infrastructure should help with this issue. It may cost much more per run but money can be saved if usage is intermittent. How are HN users handling this?

Exactly! While I have llama running locally on RTX and it’s fun to tinker with, I can’t use it for my workflows and don’t want to invest 20k+ to run a decent model locally

> How are HN users handling this? I’m working on a startup for end-to-end confidential AI using secure enclaves in the cloud (think of it like extending a local+private setup to the cloud with verifiable security guarantees). Live demo with DeepSeek 70B: chat.tinfoil.sh

Re: Building a personal, private AI computer on a budget

#52

Post by datadrivenangel »

joshstrange wrote: post #31

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past. I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I fee…

You pay a premium to get the theoretical local privacy and reliability of hosting your own models.

But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.

Re: Building a personal, private AI computer on a budget

#53

Post by tempoponet »

gregwebs wrote: post #24

The problem for me with making such an investment is that next month a better model will be released. It will either require more or less RAM than the current best model- making it either not runnable or expensive to run on an overbuilt machine. Using cloud infrastructure should help with this issue. It may cost much more per run but money can be saved if usage is intermittent. How are HN users handling this?

Most of these new models release several variants, typically in the 8b, 30b, and 70b range for personal use. YMMV with each, but you usually use the models that fit your hardware, and the models keep getting better even in the same parameter range.

To your point about cloud models, these are really quite cheap these days, especially for inference. If you're just doing conversation or tool use, you're unlikely to spend more than the cost of a local server, and the price per token is a race to the bottom.

If you're doing training or processing a ton of documents for RAG setups, you can run these in batches locally overnight and let them take as long as they need, only paying for power. Then you can use cloud services on the resulting model or RAG for quick and cheap inference.

Re: Building a personal, private AI computer on a budget

#54

Post by redman25 »

diggan wrote: post #27
jmyeet wrote: post #17

The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac M…

How is the performance difference between using a dedicated GPU from Nvidia for example compared to whatever Apple does? So lets say we'd run a model on a Mac Mini M4 with 24GB RAM, how many tokens/s are you getting? Then if we run the exact same model but with a RTX 3090ti for example, how many tokens/s are you getting? Do these comparisons exist somewhere online already? I understand it's possible to run the model…

Not the exact same comparison but I have an M1 mac with 16gb ram and can get about 10 t/s with a 3B model. The same model on my 3060ti gets more than 100 t/s.

Needless to say, ram isn't everything.

Re: Building a personal, private AI computer on a budget

#55

Post by xienze »

mechagodzilla wrote: post #35
elorant wrote: post #20

Earlier quoted context omitted.

Runs is an overstatement though. With 4 tokens/second you can't use it on production.

I have a similar setup running at about 1.5 tokens/second, and it's perfectly usable for the sorts of difficult tasks one needs a frontier model like this for - give it a prompt and come back an hour or two later. You interact with it like e-mailing a coworker. If I need an answer back in seconds, it's probably not a very complicated question, and a much smaller model will do.

I get where you’re coming from, but the problem with LLMs is that you very regularly need a lot of back-and-forth with them to tease out the information you’re looking for. A more apt analogy might be a coworker that you have to follow up with three or four times, at an hour per. Not so appealing anymore. Doubly so when you have to stand up $2k+ of hardware for the privilege. If I’m paying good money to host something locally, I want decent performance.

Re: Building a personal, private AI computer on a budget

#56

Post by xienze »

walterbell wrote: post #49
gregwebs wrote: post #24

The problem for me with making such an investment is that next month a better model will be released. It will either require more or less RAM than the current best model- making it either not runnable or expensive to run on an overbuilt machine. Using cloud infrastructure should help with this issue. It may cost much more per run but money can be saved if usage is intermittent. How are HN users handling this?

> expensive to run on an overbuilt machine There's a healthy secondary market for GPUs.

The price goes up dramatically once you go past 12GB though, that’s the problem.

Re: Building a personal, private AI computer on a budget

#57

Post by redman25 »

DemetriousJones wrote: post #23
0xEF wrote: post #15

Earlier quoted context omitted.

How useful is this, though? In my modest experience, these tiny models aren't good for much more than tinkering with, definitely not something I'd integrate into my workflow since the output quality is pretty low. Again, though, my experience is limited. I imagine others know something I do not and would absolutely love to hear more from people who are running tiny models on low-end hardware for things like code assi…

I tried running the 8B model on my 8GB M2 Macbook Air through Ollama and it was awful. It took ages to do anything and the responses were bad at best.

Doesn't 8B need at least 16gb of ram? Otherwise, your swapping I would imagine...

Re: Building a personal, private AI computer on a budget

#58

Post by rsanek »

datadrivenangel wrote: post #52
joshstrange wrote: post #31

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past. I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I fee…

You pay a premium to get the theoretical local privacy and reliability of hosting your own models. But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.

With something like OpenRouter, you don't even have to manually integrate with multiple vendors

Re: Building a personal, private AI computer on a budget

#59

Post by joshstrange »

datadrivenangel wrote: post #52
joshstrange wrote: post #31

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past. I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I fee…

You pay a premium to get the theoretical local privacy and reliability of hosting your own models. But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.

Yeah, really I'd love for my Home Assistant to be able to use a local LLM/TTS/STT which I did get working but was way too slow. Also it would fun to just throw some problems/ideas at the wall without incurring (more) cost, that's a big part of it. But each time I run the numbers I would be better off using Anthropic/OpenAI/DeepSeek/other.

I think sooner or later I'll break down and buy a server for local inference even if the ROI is upside down because it would be a fun project. I also find that these thing fall in the "You don't know what you will do with it until you have it and it starts unlocking things in your mind"-category. I'm sure there are things I would have it grind on overnight just to test/play with an idea which is something I'd be less likely to do on a paid API.

Re: Building a personal, private AI computer on a budget

#60

Post by vanillax »

reacharavindh wrote: post #3

The thing is though.... the locally hosted models in such hardware are cute as toys, and sure do write funny jokes and importantly, perform private tasks that I would never consider passing to non-selfhosted models, but pale in comparison to the models accessible over APIs(Claude 3.5 Sonnet, OpenAI etc). If I could run deepseek-r1-678b locally, without breaking the bank, I would. But, for now, opex > capex at a consu…

Huh? Toys? You can run DeepSeek 70b on 36GB ram Macbook pro.. You can run Phi4, Qwen2.5, or llama3.3. They work great for coding tasks
Post reply on HN