Can the LLMs be deployed split across multiple Pis? I thought it was not possible, may be I am not caught up.
I regret building this $3000 Pi AI cluster
181–190 of 377 posts
Re: I regret building this $3000 Pi AI cluster
#182I can think of SOME decent--if suboptimal--reasons to build an RPi cluster, but it never would have occurred to me to try running an LLM on one. Even if you could cluster hundreds or thousands of Pis, it's still just entirely the wrong architecture for running an AI model as they are currently built.
Re: I regret building this $3000 Pi AI cluster
#183Earlier quoted context omitted.
Depends entirely on the utilities board doing the regulation. That said, I'm of the opinion that power/water/internet should all be state/county/city ran. I don't want my utilities companies to have profit motives. My water company just got bought up by a huge water company conglomerate and, you guessed it, immediate rate increases.
Most utilities, even if ostensibly privately-owned, are profit-limited and rates must be approved by a regulatory board. Some are organized as non-profits (rural water and electric co-ops, etc.) This is in exchange for the local monopoly. If your local regulators approved the merger and higher rates, your complaint is with them as much as the utility company. Not saying that some regulators are not basically rubber s…
I agree. The issue really is that they are 3 layers removed from where I can make a change. They are all appointed and not elected which means I (and my neighbors) don't have any recourse beyond the general election. IIRC, they are appointed by the governor which makes it even harder to fix (might be the county commissioner, not 100% on how they got their position, just know it was an appointment).
I did (as did others), in fact, write in comments and complaints about the rate increases and buyout. That went unheard.
Re: I regret building this $3000 Pi AI cluster
#184It's a pretty rough headline, clearly the author had fun performing the test and constructing the thing. I would be pretty regretful of just the first sentence in the article, though: > I ordered a set of 10 Compute Blades in April 2023 (two years ago), and they just arrived a few weeks ago. That's rough.
[deleted]
Re: I regret building this $3000 Pi AI cluster
#185Earlier quoted context omitted.
The author, Jeff Geerling, is a very intelligent person. He has more experience with using niche hardware than almost anyone on earth. If he does something, there's usually a good a priori rationale for it.
Jeff is a good person/blogger and does interesting projects but more experience with niche hardware than literally anyone is a stretch. Like what about the people who maintain the alpha/sparc/parisc linux kernels? Or the designers behind idk tilera or tenstorrent hardware.
> more experience with niche hardware than literally anyone is a stretch.
This is why I said "almost anyone." If I changed your words, I could disagree with you as well.
Re: I regret building this $3000 Pi AI cluster
#186I thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high performance compute. I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Maybe this is just YouTube-style headline writing spilling over to the blog for the clicks. If your goal is to play…
I’d say it’s inconclusive. For traditional compute it wins on power and cost (it’ll always lose on space). The inference is noted to not be able to use the GPU due to llama.cpp’s vulkan backend AND that clustering software in llama.cpp is bad. I’d say it’s probably still going to be worse for AI but it’s inconclusive where it could be due to the software immaturity (ie not worth it today but could be with better soft…
But will there be a CM6 while you're waiting for the software to improve?
Re: I regret building this $3000 Pi AI cluster
#187Great write up! This is nice nerd catnip. And sharing a failed project teaches so much. Please, more should share the projects that absorbed tons of time yet never really delivered.
Re: I regret building this $3000 Pi AI cluster
#188I thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high performance compute. I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Maybe this is just YouTube-style headline writing spilling over to the blog for the clicks. If your goal is to play…
The author, Jeff Geerling, is a very intelligent person. He has more experience with using niche hardware than almost anyone on earth. If he does something, there's usually a good a priori rationale for it.
Is a Pi still considered "niche" hardware?
Re: I regret building this $3000 Pi AI cluster
#189Earlier quoted context omitted.
This is the correct analysis - there's a reason you see this stuff cheap or free. The homelab group on Reddit is full of people who don't understand any of this - they have full racks in their house that could be replaced with one high-end desktop.
> The homelab group on Reddit is full of people who don't understand any of this - they have full racks in their house that could be replaced with one high-end desktop. A lot of that group is making use of the IO capabilities of these systems to run lots of PCI-E devices & hard drives. There's not exactly a cost-effective modern equivalent for that. If there were cost-effective ways to do something like take a PCI-E…
Perhaps, but I don't really get the dozens of TB of storage in the home use case a lot of the time either.
Like if you have a large media library, you need to push maybe 10MB/s, you don't need 128GB of RAM to do that...
It's mostly just hardware porn - perhaps there are a few legit use cases for the old hardware, but they are exceedingly rare in my estimate.
Re: I regret building this $3000 Pi AI cluster
#190Earlier quoted context omitted.
That’s insane, drawing very little power from an always on server is a solved problem.
how many can run on batteries? it's nice to take it on road trips / into hotels. can't really imagine hauling a server around. we probably have different definitions of "very little power".
> it's nice to take it on road trips / into hotels.
> can't really imagine hauling a server around.
These two sentences contradict each other.