Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I regret building this $3000 Pi AI cluster

jeffgeerling.com

Post reply on HN
Print view View original

231–240 of 377 posts

Re: I regret building this $3000 Pi AI cluster

#231

Post by ted_dunning »

nromiun wrote: post #83

There is a reason all the big supercomputers have started using GPUs in the last decade. They are much more efficient. If you want 32bit parallel performance just buy some consumer GPUs and hook them up. If you need 64bit buy some prosumer GPUs like the RTX 6000 Pro and you are done. Nobody is really building CPU clusters these days.

Well, El Capitan uses AMD CPUs (which have integrated GPU capabilities) and it is right on top of the rankings lately.

Frontier is right behind it with the same arrangement.

Having honest to god dedicated GPUs on their own data bus with their own memory isn't necessarily the fastest way to roll.

Re: I regret building this $3000 Pi AI cluster

#232

Post by chermi »

LTL_FTC wrote: post #171
mercutio2 wrote: post #122

Earlier quoted context omitted.

Your googling gave results that were likely accurate for California 4-5 years ago. My average cost per kWh is about 60 cents. Rates have gone up enormously because the cost of wildfires is falling on ratepayers, not the utility owners. Regulated monopolies are pretty great, aren’t they? Heads I win, tales you lose.

They have definitely increased but not all of California is like this. In the heart of Silicon Valley, Santa Clara, it's about $0.15/kWh. Having Data Centers nearby helps, I suppose.

I'm guessing the parent is talking about total bill (transmission, demand charges..) $.15/kwH is probably just the usage, and I am very skeptical that's accurate for residential.

Re: I regret building this $3000 Pi AI cluster

#233

Post by [deleted] »

mercutio2 wrote: post #125
randomgermanguy wrote: post #37

Earlier quoted context omitted.

Depends on how heavy one wants to go with the quants (for Q6-Q4 the AMD Ryzen AI MAX chips seem better/cheaper way to get started). Also the Mac Studio is a bit hampered by its low compute-power, meaning you really can't use a 100b+ dense model, only MoE feasibly without getting multi minute prompt-processing times (assuming 500+ tokens etc.)

Huh? My maxed out Mac Studio gets 60-100 tokens per second on 120B models, with latency on the order of 2 seconds. It was expensive, but slow it is not for small queries. Now, if I want to bump the context window to something huge, it does take 10-20 seconds to respond for agent tasks, but it’s only 2-3x slower than paid cloud models, in my experience. Still a little annoying, and the models aren’t as good, but the g…

[deleted]

Re: I regret building this $3000 Pi AI cluster

#234

Post by Kubuxu »

kllrnohj wrote: post #202
mattbillenstein wrote: post #189

Earlier quoted context omitted.

Perhaps, but I don't really get the dozens of TB of storage in the home use case a lot of the time either. Like if you have a large media library, you need to push maybe 10MB/s, you don't need 128GB of RAM to do that... It's mostly just hardware porn - perhaps there are a few legit use cases for the old hardware, but they are exceedingly rare in my estimate.

> Like if you have a large media library, you need to push maybe 10MB/s, For just streaming a 4k bluray you need more than 10MB/s, Ultra HD bluray tops out at 144 Mbit/s. Not to mention if that system is being hit by something else at the same time (backup jobs, etc...). Is the 128GB of RAM just hardware porn? Eh, maybe, probably. But if you want 8+ bays for a decent sized NAS then you're already quickly into price p…

For 8+ bays you just need a SAS HBA card and one free PCI-E slot. Not to mention that many motherboards will have 6+ SATA ports already.

If anything, 2nd hand AMD gaming rigs make more sense than old servers. I say that as someone with always off r720xd at home due to noise and heat. It was fun when I bought it during winter years ago, until summer came.

Re: I regret building this $3000 Pi AI cluster

#235

Post by [deleted] »

allanrbo wrote: post #78
moduspol wrote: post #29

Earlier quoted context omitted.

Also cost effective is to buy used rack mount servers from Amazon. They may be out of warranty but you get a lot more horsepower for your buck, and now your VMs don’t have to be small.

No, again, just run VMs on your desktop/laptop. The software doesn't know or care if it's a rack mounted machine.

[deleted]

Re: I regret building this $3000 Pi AI cluster

#236

Post by moduspol »

Aurornis wrote: post #31
moduspol wrote: post #29

Earlier quoted context omitted.

Also cost effective is to buy used rack mount servers from Amazon. They may be out of warranty but you get a lot more horsepower for your buck, and now your VMs don’t have to be small.

Putting a retired datacenter rack mount server in your house is a great way to learn how unbearably loud a real rack mount datacenter server is.

True! They aren't quiet. I keep mine in a well-ventilated room that doesn't typically have people in it.

Re: I regret building this $3000 Pi AI cluster

#238

Post by densh »

For anyone interested in playing with distributed systems, I'd really recommend getting a single machine with latest 16-core CPU from AMD and just running 8 virtual machines on it. 8 virtual machines, with 4 hyper threads pinned per machine, and 1/8 of total RAM per machine. Create a network between them virtually within your virtualization software of choice (such as Proxmox).

And suddenly you can start playing with distributed software, even though it's running on a single machine. For resiliency tests you can unplug one machine at a time with a single click. It will annihilate a Pi cluster in Perf/W as well, and you don't have to assemble a complex web of components to make it work. Just a single CPU, motherboard, m.2 SSD, and two sticks of RAM.

Naturally, using a high core count machine without virtualization will get you best overall Perf/W in most benchmarks. What's also important but often not highlighted in benchmarks in Idle W if you'd like to keep your cluster running, and only use it occasionally.

Re: I regret building this $3000 Pi AI cluster

#239

Post by EnPissant »

mercutio2 wrote: post #125
randomgermanguy wrote: post #37

Earlier quoted context omitted.

Depends on how heavy one wants to go with the quants (for Q6-Q4 the AMD Ryzen AI MAX chips seem better/cheaper way to get started). Also the Mac Studio is a bit hampered by its low compute-power, meaning you really can't use a 100b+ dense model, only MoE feasibly without getting multi minute prompt-processing times (assuming 500+ tokens etc.)

Huh? My maxed out Mac Studio gets 60-100 tokens per second on 120B models, with latency on the order of 2 seconds. It was expensive, but slow it is not for small queries. Now, if I want to bump the context window to something huge, it does take 10-20 seconds to respond for agent tasks, but it’s only 2-3x slower than paid cloud models, in my experience. Still a little annoying, and the models aren’t as good, but the g…

I think the Mac Studio is a poor fit for gpt-oss-120b.

On my 96 GB DDR5-6000 + RTX 5090 box, I see ~20s prefill latency for a 65k prompt and ~40 tok/s decode, even with most experts on the CPU.

A Mac Studio will decode faster than that, but prefill will be 10s of times slower due to much lower raw compute vs a high-end GPU. For long prompts that can make it effectively unusable. That’s what the parent was getting at. You will hit this long before 65k context.

If you have time, could you share numbers for something like:

llama-bench -m -ngl 999 -fa 1 --mmap 0 -p 65536 -b 4096 -ub 4096

Edit: The only Mac Studio pp65536 datapoint I’ve found is this Reddit thread:

https://old.reddit.com/r/LocalLLaMA/comments/1jq13ik/mac_stu ...

They report ~43.2 minutes prefill latency for a 65k prompt on a 2-bit DeepSeek quant. Gpt-oss-120b should be faster than that, but still very slow.

Re: I regret building this $3000 Pi AI cluster

#240

Post by Damogran6 »

mercutio2 wrote: post #122
rogerrogerr wrote: post #102

Earlier quoted context omitted.

100W over a month (rule of thumb 730 hours) is 73kWh. Which is $7.30 at my $0.10/kWh rate, or less than $25 at (what Google told me is) Cali’s average $0.30/kWh.

Your googling gave results that were likely accurate for California 4-5 years ago. My average cost per kWh is about 60 cents. Rates have gone up enormously because the cost of wildfires is falling on ratepayers, not the utility owners. Regulated monopolies are pretty great, aren’t they? Heads I win, tales you lose.

CORE energy in Colorado is charging $0.10819 per kWh _today_

https://core.coop/my-cooperative/rates-and-regulations/rate-...

Post reply on HN