Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I regret building this $3000 Pi AI cluster

jeffgeerling.com

Post reply on HN
Print view View original

101–110 of 377 posts

Re: I regret building this $3000 Pi AI cluster

#101

Post by trenchpilgrim »

newsclues wrote: post #97
bunderbunder wrote: post #20

Earlier quoted context omitted.

The cost effective way to do it is in the cloud. Because there's a very good chance you'll learn everything you intended to learn and then get bored with it long before your cloud compute bill reaches the price of a desktop with even fairly modest specs for this purpose.

Text and reference books are free at the library. You don’t need hardware to learn. Sure it helps but you can learn from a book and pen and paper exercises.

I disagree. Most of what I've learned about systems comes from debugging the weird issues that only happen on real systems, especially real hardware. The book knowledge is like, 20-30% of it.

Re: I regret building this $3000 Pi AI cluster

#102

Post by rogerrogerr »

blobbers wrote: post #55
nine_k wrote: post #38

Earlier quoted context omitted.

It will probably consume $150 worth of electricity in less than a month, even sitting idle :-\

The internet says 100W idle, so maybe more like $40-50 electricity, depending on where you live could be cheaper could be more expensive. Makes me wonder if I should unplug more stuff when on vacation.

100W over a month (rule of thumb 730 hours) is 73kWh. Which is $7.30 at my $0.10/kWh rate, or less than $25 at (what Google told me is) Cali’s average $0.30/kWh.

Re: I regret building this $3000 Pi AI cluster

#103

Post by titanomachy »

blobbers wrote: post #55
nine_k wrote: post #38

Earlier quoted context omitted.

It will probably consume $150 worth of electricity in less than a month, even sitting idle :-\

The internet says 100W idle, so maybe more like $40-50 electricity, depending on where you live could be cheaper could be more expensive. Makes me wonder if I should unplug more stuff when on vacation.

100W continuous at 12¢/kWh (US average) is only ~$9 / month. Is your electricity 5x more expensive than the US average?

Re: I regret building this $3000 Pi AI cluster

#104

Post by teleforce »

glitchc wrote: post #25
Aurornis wrote: post #17

I thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high performance compute. I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Maybe this is just YouTube-style headline writing spilling over to the blog for the clicks. If your goal is to play…

I did some calculations on this. Procuring a Mac Studio with the latest Mx Ultra processor and maxing out the memory seems to be the most cost effective way to break into 100b+ parameter model space.

Not quite, as it stands now the most cost effective way is most likely framework desktop or similar system for example HP G1a laptop/PC [1],[2].

[1] The Framework Desktop is a beast:

https://news.ycombinator.com/item?id=44841262

[2] HP ZBook Ultra:

https://www.hp.com/us-en/workstations/zbook-ultra.html

Re: I regret building this $3000 Pi AI cluster

#105

Post by geerlingguy »

philipwhiuk wrote: post #88
noelwelsh wrote: post #26

I'd love to understand the economics here. $3000 purely for fun seems like a lot. $3000 for promotion of a channel? consulting? seems reasonable.

Jeff has a million YouTube subscribers, gets $2000 a month from Patreon and has 200 GitHub sponsors. The economics of spending $3,000 on a video probably work out fine.

It's definitely a stretch for my per-video budget, but I did want to have a 'maxed out' Pi cluster for future testing as well.

A lot of people (here, Reddit, elsewhere) speculate about how good/bad a certain platform or idea is. Since I have the means to actually test how good or bad something is, I try to justify the hardware costs for it.

Similar to testing various graphics cards on Pis, I've probably spent a good $10,000 on those projects over the past few years, but now I have a version of every major GPU from the past 3 generations to test on, not only on Pi, but other Arm platforms like Ampere and Snapdragon.

Which is fun, but also educational; I've learned a lot about inference, GPU memory access, cache coherency, the PCIe bus...

So a lot of intangibles, many of which never make it directly into a blog post or video. (Similar story with my time experiments).

Re: I regret building this $3000 Pi AI cluster

#106

Post by llm_nerd »

glitchc wrote: post #25
Aurornis wrote: post #17

I thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high performance compute. I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Maybe this is just YouTube-style headline writing spilling over to the blog for the clicks. If your goal is to play…

I did some calculations on this. Procuring a Mac Studio with the latest Mx Ultra processor and maxing out the memory seems to be the most cost effective way to break into 100b+ parameter model space.

The next generation M5 should bring the matmul functionality seen on the A19 Pro to the desktop SoC's GPU -- "tensor" cores, in essence -- and will dramatically improve the running of most AI models on those machine.

Right now the Macs are viable purely because you can get massive amounts of unified memory. Be pretty great when they have the massive matrix FMA performance to complement it.

Re: I regret building this $3000 Pi AI cluster

#107

Post by titanomachy »

trenchpilgrim wrote: post #101
newsclues wrote: post #97

Earlier quoted context omitted.

Text and reference books are free at the library. You don’t need hardware to learn. Sure it helps but you can learn from a book and pen and paper exercises.

I disagree. Most of what I've learned about systems comes from debugging the weird issues that only happen on real systems, especially real hardware. The book knowledge is like, 20-30% of it.

Agreed, I don't think I'd hire a datacenter engineer whose experience consisted of reading books and doing "pen and paper exercises".

Re: I regret building this $3000 Pi AI cluster

#108

Post by unregistereddev »

NitpickLawyer wrote: post #12
elzbardico wrote: post #2

Frankly, always thought about Pi Clusters as a nerd indulgence, something to play, not to do serious work.

It reminds me of the Beowulf clusters of the 90s-2000s, that were all the rage at some point, then slowly lost ground... I remember many friends tinkering with some variant of those, we had one in Uni, and there were even some linux distros dedicated to the concept.

Ditto! It reminded me of the time in college when I built a Beowulf cluster from recently-retired Pentium II desktops.

Was it fast? No. But that wasn't the point. I was learning about distributed computing.

Re: I regret building this $3000 Pi AI cluster

#109

Post by geerlingguy »

deadbabe wrote: post #87
theultdev wrote: post #79

Earlier quoted context omitted.

I use mine for a plex server. I don't need to transcode + I need something I can leave on that draws little power. I have a powerful rig, but the one time I get to turn it off is when I'd need the media server lol. There's a lot of scenarios where power usage comes into play. These clusters don't make much sense to me though.

That’s insane, drawing very little power from an always on server is a solved problem.

What's your idea of very little power, though?

I know for many who run SBCs (RK3588, Pi, etc.), very little is 1-2W idle, which is almost nothing (and doesn't even need a heatsink if you can stand some throttling from time to time).

Most of the Intel Mini PCs (which are about the same price, with a little more performance) idle at 4-6W, or more.

Re: I regret building this $3000 Pi AI cluster

#110

Post by kbenson »

Almondsetat wrote: post #33
bunderbunder wrote: post #20

Earlier quoted context omitted.

The cost effective way to do it is in the cloud. Because there's a very good chance you'll learn everything you intended to learn and then get bored with it long before your cloud compute bill reaches the price of a desktop with even fairly modest specs for this purpose.

I can get a Xeon E5-2690V4 with 28 threads and 64GB of RAM for about $150. If you need cores and memory to make a lot of VMs you can do it extremely cheaply

Source? That seems like something I would want to take advantage if at the moment...
Post reply on HN