Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I regret building this $3000 Pi AI cluster

jeffgeerling.com

Post reply on HN
Print view View original

361–370 of 377 posts

Re: I regret building this $3000 Pi AI cluster

#361

Post by a2128 »

ripdog wrote: post #219
a2128 wrote: post #82

Earlier quoted context omitted.

Why do people buy gaming PCs when it's much cheaper to use streaming platforms? I think the two cases share practically the same parallels in terms of reliability, availability, restrictions, flexibility, sovereignty, privacy, etc. But also when it comes to Vast/RunPod it can be annoying and genuinely become more expensive if you have to rent 2x the number of hours because you constantly have to upload and download d…

Because latency matters when gaming in a way which doesn't matter with AI inference? Plus cloud gaming is always limited in range of games, there are restrictions on how you can use the PC (like no modding and no swapping savegames in or out).

Yep those are exactly the same considerations. LLM providers will have inconsistent latency and throughput due to batching across many users, while training with cloud GPU servers can have inconsistent bandwidth and delay for uploading mass training data. LLM providers are always limited in how you can use them (often no LoRAs, finetuned models, prompt restrictions)

Re: I regret building this $3000 Pi AI cluster

#362

Post by nathan_douglas »

teaearlgraycold wrote: post #241

Having seen someone else build and tinker with a 7 node Pi cluster it seems like an absolute waste of time. 1Gb/s networking, PCIe 3.0 x1, slow RAM, slow CPU. And with all the hats and accessories needed it’s not that good of a deal. Getting some NUC-like machines makes a lot more sense to me. You’ll get 2.5Gb/s Ethernet at the least and way more FlOPS as well.

You might really dislike the ESP32 cluster I'm planning right now :P

Re: I regret building this $3000 Pi AI cluster

#363

Post by privatelypublic »

oasisaimlessly wrote: post #310
privatelypublic wrote: post #288

Earlier quoted context omitted.

You theoretically have a year before you even have to apply- but patents are expressly "first to file."

Public use of an idea still prevents someone else from patenting it.

Sounds like You're conflating sale of a product prior to filing with "prior art"

Re: I regret building this $3000 Pi AI cluster

#364

Post by horsawlarway »

0xbadcafebee wrote: post #170
horsawlarway wrote: post #71

Earlier quoted context omitted.

There are an absolutely stunning number of ways to lose a whole bunch of money very quickly if you're not careful renting compute. $3,000 is well under many "oopsie billsies" from cloud providers. And that's outside of the whole "I own it" side of the conversation, where things like latency, control, flexibility, & privacy are all compelling reasons to be willing to spend slightly more. I still run quite a number of…

You could also spill a can of Mountain Dew over the $8,000 AI rig next to you. Oopsies can happen anywhere... If it's for personal use, do whatever... there's nothing wrong with buying a $60,000 sports car if you get a lot of enjoyment out of driving it. (you could also lease if you want to trade up to the "faster model" next year) For business, renting (and managed hosting) makes more sense.

So you got downvoted already, but to clearly address this:

If I spill something on my own hardware, the max out-of-pocket amount I lose is the amount I spent on that hardware.

If I run up an AWS/GCP/Azure bill accidentally... the max out-of-pocket amount I lose is often literally unbounded. Are there some guardrails you can put around this? Sure. But they're often confusing, misleading, delayed, or riddled with "holes" which they don't catch.

Ex - the literal best AWS offers you is delayed "billing alarms" which need to be manually enabled and configured, and even then don't cover all the services you might incur billing charges for.

It's not that "Oopsies" can't happen locally - it's that even if they do, I have a clear understanding of the potential costs by default, and they're much less intangible than "I left a thing running overnight and I now I owe AWS a new car worth of cash".

The worst case for a misconfigured bit of software locally is that my machine stalls and my services go down (ex - overloaded). The worst case for a misconfigured bit of software in AWS is literal bankruptcy.

Think about that for a minute.

Re: I regret building this $3000 Pi AI cluster

#365

Post by robotguy »

geerlingguy wrote: post #301
milesvp wrote: post #298

Earlier quoted context omitted.

Seriously. Imagine the fractal nature of software problems (you do need firmware right?), and add that on top of a layer that has poorly documented errata and non standard peripheral variants. I'm currently making a board to solder onto an existing production run because we finally found the 1 sentence in a 1700page data sheet that explained why many of our MCUs are overheating. Had the datasheet simply labeled VREFP…

The stories hardware devs have about individual chip datasheets causing hours or weeks of pain... It's most fun when you can prove the vendor's datasheet is lying about some pin or some function, but they still don't update it after a decade or more. So everyone integrating the chip who hasn't before hits the exact same speed bump!

A junior EE can build the circuits from the examples in the datasheet. A mid-level EE can build configurations NOT shown in the datasheet. A Senior EE can tell you where the datasheet is wrong.

Re: I regret building this $3000 Pi AI cluster

#366

Post by alias_neo »

TZubiri wrote: post #358
alias_neo wrote: post #340

Earlier quoted context omitted.

> What do you mean by Cooler? Raspberry pi doesn't need a fan It's recommended for Pi 5, and if you're running a Pi 4, you should at least use a little heat sink, the 4 and 5 run pretty warm, and under any load they can throttle quite easily. I run mine in a rack, in the UK where it's not very warm compared to other parts of the world, and they get pretty warm even with cooling. > Also the other peripherals you consi…

Interesting. I may be outdated 1) raspberry pis competitors have gotten better, that nuc is very cheap. 2) the pi has gone in a different direction, increasing specs and price, the 3b+ or 4a had much lower specs, price, power consumption etc... In conclusion, if you can get an arm soc board with specs similar to the 3b+ or 4a (500mb to 2gb ram), then you can host a blog on linux for cheap. Should run you in the 50$ a…

Absolutely. I wouldn't suggest one shouldn't use a Pi if it fits their use case and budget, simply that once we get to a higher end Pi, it can be cost effective to simply buy a mini PC which will be more capable for not a lot more money.

The issue with competing ARM SBCs is the software support; Radxa makes some boards that are more powerful than Pis, but if you read the forums they've had hardware flaws in the designs, and they run old kernels and don't get updated, and of course there isn't the community behind it.

An x86 mini pc is a different beast to a Pi, but then I think a lot of people who were hosting software on a Pi weren't specifically looking for ARM architecture anyway, unless they were, in which case stick with a Pi.

Re: I regret building this $3000 Pi AI cluster

#367

Post by bokohut »

swiftcoder wrote: post #208
nine_k wrote: post #38

Earlier quoted context omitted.

It will probably consume $150 worth of electricity in less than a month, even sitting idle :-\

Obviously the solution is to pickup another hobby, and enter the DIY solar game at the same time as your home lab obsession :D

Interestingly enough it is often times a foundational change in one's 'normal' that inspires something 'new'.

In this case that 'new' is energy efficient software down to the individual lines of code and what their energy cost is on certain hardware. Academics are publishing about it in niche corners of the web and some entrepreneurs are doing it but of course none of this is cool now so we remain a mockery for our objectives. In time this too will become a real thing as many now are just beginning to feel the ever rising costs of energy which is only just starting to increase from decisions made years ago. The worst is yet to come as seen and heard directly from every single expert that has testified in the last years before the Energy and Commerce committee however only the outside-the-boxers among us watch such educational content to better prepare for tomorrow.

Electricity powers our world and nearly all take it for granted, time too will change this thinking.

:D

Re: I regret building this $3000 Pi AI cluster

#368

Post by globular-toast »

motorest wrote: post #347
globular-toast wrote: post #334

Earlier quoted context omitted.

The point was you don't need to wait for 8 Pis to become available when most people can get going straight away with what they already have. If you want to learn physical networking or really need to "see" things happening on physically separate machines just get a free old PC from gumtree or something.

> The point was you don't need to wait for 8 Pis to become available when most people can get going straight away with what they already have. You also don't need RPis to learn anything about programming, networking, electronics, etc. But people do it anyways. I really don't see what point anyone thinks they are making regarding pedogogy. RPis are synonymous with tinkering, regardless of how you cut it. Distributed s…

I think you misread my comment, maybe it's clearer if I say "(admittedly) the pi is meant for paedogogy (however) I feel like most of them are wasted".

Re: I regret building this $3000 Pi AI cluster

#369

Post by Computer0 »

jaredhallen wrote: post #321
greenavocado wrote: post #314

Earlier quoted context omitted.

N100s absolutely curb stomp Pis and have very good video hardware encoders/decoders to boot

A different thread on here the other day seemed to come to the consensus that if you're considering an N100, you may as well go with a low end Ryzen. And for much the same reasoning being used here. Seems to be a little bit of a slippery slope.

I have looked at it extensively and the Ryzen never is available at the pricing of an n100. Though it would offer a lot more performance.

Re: I regret building this $3000 Pi AI cluster

#370

Post by throwaway894345 »

theodric wrote: post #319
throwaway894345 wrote: post #261

Earlier quoted context omitted.

> Depends. At full load? At Irish power prices? Just the Pi, no peripherals, no NVMe? 5 units? €13/mo. Pretty sure most of us aren't running anywhere close to full load 24/7, but whoa, Irish power is expensive. In the central US I pay $0.14/KWh.

Yeah, it's brutal. Was €0.39 right after Mad Vlad kicked off his vanity conflict.

That’s rough. What’s your progress on renewables? Wind has made electricity really cheap in my state and I would think Ireland would be pretty windy (esp offshore)?
Post reply on HN