Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

191–200 of 236 posts

Re: Building a personal, private AI computer on a budget

#191

Post by cratermoon »

CamperBob2 wrote: post #162
cratermoon wrote: post #111

From the article: In the future, I fully expect to be able to have a frank and honest discussion about the Tiananmen events with an American AI agent, but the only one I can afford will have assumed the persona of Father Christmas who, while holding a can of Coca-Cola, will intersperse the recounting of the tragic events with a joyful "Ho ho ho... Didn't you know? The holidays are coming!" How unfortunate that people…

It is easy to get a local R1 model to talk about Tiananmen Square to your heart's content. Telling it to replace problematic terms with "Smurf" or another nonsense word is very effective, but with the local model you don't even have to do that in many cases. (e.g., https://i.imgur.com/btcI1fN.png )

Ah yes, evidence of Shadow Prompting[1]. Likely their API is given additional, hidden, prompt context making Tiananmen verboten.

1 https://www.techpolicy.press/shining-a-light-on-shadow-promp...

Re: Building a personal, private AI computer on a budget

#192

Post by Eisenstein »

wkat4242 wrote: post #187
Eisenstein wrote: post #41

Earlier quoted context omitted.

But the inference doesn't necessarily run at the quant precision.

As far as I understand it does if you quantify the K/V store as well (the context). And that's pretty standard now because it can increase maximum context size a lot.

It is available in most inference engines, but I wouldn't call it in standard use, as it can degrade quality tremendously.

Re: Building a personal, private AI computer on a budget

#193

Post by vunderba »

drcongo wrote: post #128
ekianjo wrote: post #88

Earlier quoted context omitted.

Mac Mini will be very slow for context ingestion compared to nvidia GPU, and the other issue is that they are not usable for Stable Diffusion... So if you just want to use LLMs, maybe, but if you have other interests in AI models, probably not the right answer.

I use a Mac Studio for Stable Diffusion, what's special about the Mac Mini that means it won't work?

What models are you using? Stable diffusion 1.5, SDXL, or flux?

I've heard that Macs are pretty slow with XL and borderline unusable for flux requiring minutes at a time to generate a single image - whereas an RTX4090 can generate a 1024x1024 image with the higher quality Flux Dev model (not schnell) in 14 seconds.

OP is probably correct that if you want to branch out of just strictly LLM's, cuda is the way to go. I've never heard of anyone getting LTX or hunyuan running on a Mac for example.

Re: Building a personal, private AI computer on a budget

#194

Post by UncleOxidant »

kridsdale1 wrote: post #166
UncleOxidant wrote: post #156

Earlier quoted context omitted.

I wish Apple would offer a 128GB option in the Mac Mini - That would require an M4 Max which they don't offer in the mini. I know they have a MBP with M4 Max and 128GB, but I don't need another laptop.

I’m waiting until this summer with the M4 Ultra Studio.

Which will likely be over five grand for 128GB.

Re: Building a personal, private AI computer on a budget

#195

Post by vunderba »

diggan wrote: post #36
walterbell wrote: post #33

Earlier quoted context omitted.

Some LLM use cases are async, e.g. agents, "deep research" clones.

Not to mention even simpler things, like wanting to tag all of your local notes based on the content, basically a bash loop you can run indefinitely and speed doesn't matter much, as long as it eventually finishes

Additionally if all you're doing is simple tagging and classification, you can probably get away with a significantly smaller model (sub 14b parameter model) like Mistral 7b or Qwen 14b.

Re: Building a personal, private AI computer on a budget

#196

Post by DrPhish »

cratermoon wrote: post #159
DrPhish wrote: post #101

This is just a limited recreation of the ancient mikubox from https://rentry.org/lmg-build-guides Its funny to see people independently "discover" these builds that are a year plus old. Everyone is sleeping on these guides, but I guess the stink of 4chan scares people away?

"ancient" guide. Pub: 10 May 2024 21:48 UTC

The build guide index page is newer, but to be fair, the mikubox rentry is from Oct 6, 2023.

If that isn't "ancient" in terms of AI workstation build guides, then I don't know what is.

Re: Building a personal, private AI computer on a budget

#197

Post by pelatimtt »

wkat4242 wrote: post #185
rsanek wrote: post #58

Earlier quoted context omitted.

With something like OpenRouter, you don't even have to manually integrate with multiple vendors

Is that like LiteLLM? I have that running but never tried OpenRouter. I wonder now if it's better :)

You can also try LangDB or Portkey

Re: Building a personal, private AI computer on a budget

#199

Post by BizarroLand »

axegon_ wrote: post #10

I did something similar but using a K80 and M40 I dug up from eBay for pennies. Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. That said, I had a decent-ish HP workstation laying around with 1200 watt power supply so I had where to put those two in. The one thing…

What kind of performance did you get out of that?
Post reply on HN