Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I regret building this $3000 Pi AI cluster

jeffgeerling.com

Post reply on HN
Print view View original

191–200 of 377 posts

Re: I regret building this $3000 Pi AI cluster

#191

Post by numpad0 »

> goes round-robin style asking each node to perform its prompt processing, then token generation.

Yeah, this is a now-long-wide-known issue with LLM processing. This can be remediated so that all nodes split computation, but then you'll come back to classical supercomputing problem of node interconnect latency/bandwidth bottlenecks.

It looks to me that many such interconnect simulate Ethernet cards. I wonder if it can be recreated using the M.2 slot rather than using that slot for node-local data, and cost effectively so(like cheaper than bunch of 10GE cards and short DACs).

Re: I regret building this $3000 Pi AI cluster

#193

Post by TZubiri »

alias_neo wrote: post #138
TZubiri wrote: post #115

Earlier quoted context omitted.

Fun fact, a raspberry pi does not have a built in Real Time Clock with its own battery, so it relies on network clocks to keep the time. Another fun fact, the network module of the pi is actually connected to the USB bus, so there's some overhead as well as a throughput limitation. Fun fact, the Pi does not have a power button, relying on software to shut down cleanly. If you lose access to the machine, it's not poss…

> the raspberry pi is still an amazingly cost effective choice It's really not though. I've been a Pi user and fan since it was first announced, and I have dozens of them, so I'm not hating on RPi here; we did the maths some time back here on HN when something else Pi related came up. If you go for a Pi5 with say 8GB RAM, by the time you factor in an SSD + HAT + PSU + Case + Cooler (+ maybe a uSD), you're actually al…

What do you mean by Cooler? Raspberry pi doesn't need a fan.

Also the other peripherals you consider are irrelevant, since you would need them (or not), in other setups. You can use a pi without a PSU for example. And if you use an SSD, you have to consider that cost in whatever you compare it to.

>I went this route myself and have figuratively and literally shelved a bunch of Pis

>and I have dozens of them,

Reread my post? I meant specifically that Pis are great for the 1 to 2 range. with 3 pis you should change to something else. So I'm saying they are good at the 100$-200$ budget, but bad anywhere above that.

Re: I regret building this $3000 Pi AI cluster

#195

Post by ComputerGuru »

Aurornis wrote: post #31
moduspol wrote: post #29

Earlier quoted context omitted.

Also cost effective is to buy used rack mount servers from Amazon. They may be out of warranty but you get a lot more horsepower for your buck, and now your VMs don’t have to be small.

Putting a retired datacenter rack mount server in your house is a great way to learn how unbearably loud a real rack mount datacenter server is.

Only if it’s a 1U. 2U units idle at silent.

Re: I regret building this $3000 Pi AI cluster

#196

Post by wcchandler »

I was just exploring Pi’s and AI hats, so this post is appreciatively timely.

I’m finally at the point where I can dedicate time for building an AI with a specific use case in mind. I play competitive paintball and would like to utilize AI for a handful of things. Specifically hit detections in video streams. Pi’s were my natural choice simply because of low cost of entry and wide range of supported products to get a PoV running. I even thought about reaching out to Jeff and asking his input.

This post didn’t change my direction too much, but it did help level set some realistic expectations. So thanks for sharing.

Re: I regret building this $3000 Pi AI cluster

#197

Post by amatecha »

nine_k wrote: post #76
blobbers wrote: post #55

Earlier quoted context omitted.

The internet says 100W idle, so maybe more like $40-50 electricity, depending on where you live could be cheaper could be more expensive. Makes me wonder if I should unplug more stuff when on vacation.

I was surprised to find out that my apartment pulls 80-100W when everything is seemingly down during the night. A tiny light here and there, several displays in sleep mode, a desktop idling (mere 15W, but), a laptop charging, several phones charging, etc, the fridge switches on for a short moment. The many small amounts add up to something considerable.

Yeah it kinda puts it all into perspective when you think of how every home used to use 60-watt light bulbs all throughout. Most people just leave lights on all over their home all day, probably using hundreds of watts of electricity. Makes me realize my 35-65w laptop is pretty damn efficient haha

Re: I regret building this $3000 Pi AI cluster

#198

Post by ComputerGuru »

Guys, don’t take the claim so literally. He’s a successful tech poster. He makes good money showing off his purchases the good money complaining about how expensive they were.

But certainly don’t imitate his choices, his economics aren’t your economics!

Re: I regret building this $3000 Pi AI cluster

#199

Post by pinkgolem »

bunderbunder wrote: post #20
Aurornis wrote: post #17

I thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high performance compute. I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Maybe this is just YouTube-style headline writing spilling over to the blog for the clicks. If your goal is to play…

The cost effective way to do it is in the cloud. Because there's a very good chance you'll learn everything you intended to learn and then get bored with it long before your cloud compute bill reaches the price of a desktop with even fairly modest specs for this purpose.

For learning I feel much safer setting everything up locally, worst case I have to reinstall my system.

In the cloud, worst case I have a bill over 5-6 digits.

And I know my ADD, 2 is not super unlikely.

Post reply on HN