Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a personal, private AI computer on a budget

ewintr.nl

Post reply on HN
Print view View original

151–160 of 236 posts

Re: Building a personal, private AI computer on a budget

#151

Post by lewisl9029 »

This article is coming out at an interesting time for me.

We probably have different definitions for "budget", but I just ordered a super janky eGPU setup for my very dated 8th gen Intel NUC, with a m2->pcie adapter, a PSU, and a refurb Intel A770 for about 350 all-in, not bad considering that's about the cost of a proper Thunderbolt eGPU enclosure alone.

The overall idea: A770 seems like a really good budget LLM GPU since it has more memory (16GB) and more memory bandwidth (512GB/s) than a 4070, but costs a tiny fraction. The m2-pcie adapter should give it a bit more bandwidth to the rest of the system than Thunderbolt as well, so hopefully it'll make for a decent gaming experience too.

If the eGPU part of the setup doesn't work out for some reason, I'll probably just bite the bullet and order the rest of the PC for a couple hundred more, and return the m2-pcie adapter (I got it off of Amazon instead of Aliexpress specifically so I could do this), looking to end up somewhere around 600 bux total. I think that's probably a more reasonable price of entry for something like this for most people.

Curious if anyone else has experience with the A770 for LLM? Been looking at Intel's https://github.com/intel/ipex-llm project and it looked pretty promising, that's what made me pull the trigger in the end. Am I making a huge mistake?

Re: Building a personal, private AI computer on a budget

#153

Post by UncleOxidant »

lewisl9029 wrote: post #151

This article is coming out at an interesting time for me. We probably have different definitions for "budget", but I just ordered a super janky eGPU setup for my very dated 8th gen Intel NUC, with a m2->pcie adapter, a PSU, and a refurb Intel A770 for about 350 all-in, not bad considering that's about the cost of a proper Thunderbolt eGPU enclosure alone. The overall idea: A770 seems like a really good budget LLM GPU…

> refurb Intel A770 for about 350

I'm seeing A770s for about $500 - $550. Where did you find a refurb one for $350 (or less since you're also including other parts of the system)

Re: Building a personal, private AI computer on a budget

#154

Post by whalesalad »

How many credits does this get you in any cloud or via openai/anthropic api credits? For ~$1700 I can accomplish way more without hardware locally. Don't get me wrong I enjoy tinkering and building projects like this - but it doesn't make financial sense here to me. Unless of course you live 100% off-grid and have Stallman level privacy concerns.

Of course I do want my own local GPU compute setup, but the juice just isn't worth the squeeze.

Re: Building a personal, private AI computer on a budget

#155

Post by navbaker »

fkyoureadthedoc wrote: post #120
diggan wrote: post #113

Earlier quoted context omitted.

When you run that, what quantization do you get? The library website of Ollama ( https://ollama.com/library/gemma2:27b ) isn't exactly a good use case in surfacing useful information like what the default quantization is.

not sure how to tell, but here's the full output from ollama serve https://pastes.io/ollama-run-gemma2-27b

If you hit the drop-down menu for the size of the model, then tap “view all”, you will see the size and hash of the model you have selected and can compare it to the full list below it that has the quantization specs in the name.

Re: Building a personal, private AI computer on a budget

#156

Post by UncleOxidant »

kamranjon wrote: post #70

For the same price ($1799) you could buy a Mac Mini with 48gb of unified memory and an m4 pro. It’d probably use less power and be much quieter to run and likely could outperform this setup in terms of tokens per second. I enjoyed the write up still, but I would probably just buy a Mac in this situation.

I wish Apple would offer a 128GB option in the Mac Mini - That would require an M4 Max which they don't offer in the mini. I know they have a MBP with M4 Max and 128GB, but I don't need another laptop.

Re: Building a personal, private AI computer on a budget

#157

Post by whalesalad »

joshstrange wrote: post #31

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past. I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I fee…

The juice aint worth the squeeze to do this locally.

But you should still play with proxmox, just not for this purpose. My recommendation would be to get an i7 HP Elitedesk. I have multiple racks in my basement, hundreds of gigs of ram, multiple 2U 2x processor enterprise servers etc.... but at this point all of it is turned off and a single HP Elitedesk with a 2nd NIC added and 64GB of ram is doing everything I ever needed and more.

Re: Building a personal, private AI computer on a budget

#158

Post by lewisl9029 »

UncleOxidant wrote: post #153
lewisl9029 wrote: post #151

This article is coming out at an interesting time for me. We probably have different definitions for "budget", but I just ordered a super janky eGPU setup for my very dated 8th gen Intel NUC, with a m2->pcie adapter, a PSU, and a refurb Intel A770 for about 350 all-in, not bad considering that's about the cost of a proper Thunderbolt eGPU enclosure alone. The overall idea: A770 seems like a really good budget LLM GPU…

> refurb Intel A770 for about 350 I'm seeing A770s for about $500 - $550. Where did you find a refurb one for $350 (or less since you're also including other parts of the system)

I got this one from Acer's Ebay for $220: https://www.ebay.com/itm/266390922629

It's out of stock now unfortunately, but it does seem to pop up again from time to time according to Slickdeals: https://slickdeals.net/newsearch.php?q=a770&pp=20&sort=newes...

I would probably just watch the listing and/or set up a deal alert on Slickdeals and wait. If you're in a hurry though, you can probably find a used one on Ebay for not too much more.

Re: Building a personal, private AI computer on a budget

#159

Post by cratermoon »

DrPhish wrote: post #101

This is just a limited recreation of the ancient mikubox from https://rentry.org/lmg-build-guides Its funny to see people independently "discover" these builds that are a year plus old. Everyone is sleeping on these guides, but I guess the stink of 4chan scares people away?

"ancient" guide. Pub: 10 May 2024 21:48 UTC

Re: Building a personal, private AI computer on a budget

#160

Post by nickthegreek »

joshstrange wrote: post #146
nickthegreek wrote: post #62

Earlier quoted context omitted.

You shouldn't be having slow response issues with LLM/TTS/STT for HA on a mbp m3 max 128gb. I'd either limit the entities exposed or choose a smaller model.

Oh, I can get smaller models to run reasonably fast but I'm very interested in tool calling and I'm having a hard time finding a model that runs fast and is good at calling tools locally (I'm sure that's due to my own ignorance).

I decided on openai api for now after setting up so many differnt methods. the local stuff isn't up to snuff yet for what I am trying to accomplish but decent for basic control.
Post reply on HN