Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I regret building this $3000 Pi AI cluster

jeffgeerling.com

Post reply on HN
Print view View original

341–350 of 377 posts

Re: I regret building this $3000 Pi AI cluster

#341

Post by Havoc »

phito wrote: post #336
Havoc wrote: post #129

Have a bunch of Pis too, but realized I can use them to create a high availability control plane for a k8s cluster. Pi4s are entirely adequate for that

Could you please elaborate? I find this interesting as I'm building a k8s cluster and also have a bunch of RPIs laying around...

3x Pi 4s booting off ssd running Talos with virtual IP enabled

You know in k8s you've got worker nodes and control plane nodes? The control planes don't need much horsepower, but they're what you need to be online all to communicate with the cluster. Pis work just fine for that.

Re: I regret building this $3000 Pi AI cluster

#342

Post by phito »

Havoc wrote: post #341
phito wrote: post #336

Earlier quoted context omitted.

Could you please elaborate? I find this interesting as I'm building a k8s cluster and also have a bunch of RPIs laying around...

3x Pi 4s booting off ssd running Talos with virtual IP enabled You know in k8s you've got worker nodes and control plane nodes? The control planes don't need much horsepower, but they're what you need to be online all to communicate with the cluster. Pis work just fine for that.

Great, I will look into that. Thank you!

Re: I regret building this $3000 Pi AI cluster

#343

Post by flas9sd »

kllrnohj wrote: post #117
kbenson wrote: post #110

Earlier quoted context omitted.

Source? That seems like something I would want to take advantage if at the moment...

Note the E5-2690V4 is a 10 year old CPU, they are talking about used servers. You can find those on ebay or whatever as well as stores specializing in that. Depending on where you live, you might even find them free as they are often considered literal ewaste by the companies decommissioning them. It also means it performs like a 10 year old server CPU, so those 28 threads are not exactly worth a lot. The geekbench r…

I tend to use quite old hardware that is powered-off when not in use for its intended purpose and I coined "capability is its own quality".

For dedicated build boxes that crunch through lots of sources (whole distributions, AOSP) but do run seldomly, getting your hands on lots of Cores and RAM very cheaply can still trump buying newer CPUs with better perf/watt but higher cost.

Re: I regret building this $3000 Pi AI cluster

#344

Post by kllrnohj »

ssl-3 wrote: post #305
kllrnohj wrote: post #257

Earlier quoted context omitted.

> For 8+ bays you just need a SAS HBA card and one free PCI-E slot. Not to mention that many motherboards will have 6+ SATA ports already. And what case are you putting them into? What if you want it rack mounted? What about >1gig networking? What if I want a GPU in there to do whisper for home assistant? Used gaming rigs are great. But used servers also still have loads of value, too. Compute just isn't one of them.

> And what case are you putting them into? Maybe one of the Fractal Designs cases with a bunch of drive bays? > What if you want it rack mounted? Companies like Rosewill sell ATX cases that can scratch that itch. > What about >1gig networking? What about PCI Express card? Regular ATX computers are expandable. > What if I want a GPU in there to do whisper for home assistant? I mean... We started with a gaming rig, rig…

> Companies like Rosewill sell ATX cases that can scratch that itch.

Have you looked at what they cost? Those cases alone cost as much as a used server. Which comes with a case.

> What about PCI Express card? Regular ATX computers are expandable.

As mentioned higher up, they run out of lane count in a hurry. Especially when you're using things like used Connect-X cards

Re: I regret building this $3000 Pi AI cluster

#345

Post by user432678 »

densh wrote: post #238

For anyone interested in playing with distributed systems, I'd really recommend getting a single machine with latest 16-core CPU from AMD and just running 8 virtual machines on it. 8 virtual machines, with 4 hyper threads pinned per machine, and 1/8 of total RAM per machine. Create a network between them virtually within your virtualization software of choice (such as Proxmox). And suddenly you can start playing with…

Yes, but this is boring. Saying this as an owner of home server with ProxMox.

Re: I regret building this $3000 Pi AI cluster

#346

Post by fgbarben »

deater wrote: post #84

as someone who has built various raspberry pi clusters over the years (I even got an academic paper out of one) the big shame is that as far as I know it's still virtually impossible to use the fairly powerful GPUs they have for GPGPU work

sub hundred gigaflop counts as "fairly powerful" now?

Re: I regret building this $3000 Pi AI cluster

#347

Post by motorest »

globular-toast wrote: post #334
motorest wrote: post #322

Earlier quoted context omitted.

> I've been saying this for years. When the last Raspberry Pi shortage happened people were scrambling to get them for building these toy clusters and it's such a shame. The Pi was made for paedogogy but I feel like most of them are wasted. You're describing people using RPis to learn distributed systems, and you conclude that these RPis are wasted because RPis were made for paedogogy? > I run a K8s "cluster" on a si…

The point was you don't need to wait for 8 Pis to become available when most people can get going straight away with what they already have. If you want to learn physical networking or really need to "see" things happening on physically separate machines just get a free old PC from gumtree or something.

> The point was you don't need to wait for 8 Pis to become available when most people can get going straight away with what they already have.

You also don't need RPis to learn anything about programming, networking, electronics, etc.

But people do it anyways.

I really don't see what point anyone thinks they are making regarding pedogogy. RPis are synonymous with tinkering, regardless of how you cut it. Distributed systems too.

Re: I regret building this $3000 Pi AI cluster

#348

Post by fragmede »

throwaway894345 wrote: post #261
theodric wrote: post #245

Earlier quoted context omitted.

Depends. At full load? At Irish power prices? Just the Pi, no peripherals, no NVMe? 5 units? €13/mo. Handy: https://700c.dk/?powercalc My Pi CM4 NAS with a PCIe switch, SATA and USB3 controllers, 6 SATA SSDs, 2 VMs, 2 LXC containers, and a Nextcloud snap pretty much sits at 17 watts most of the time, hitting 20 when a lot is being asked of it, and 26-27W at absolute max with all I/O and CPU cores pegged. €3.85/mo if…

> Depends. At full load? At Irish power prices? Just the Pi, no peripherals, no NVMe? 5 units? €13/mo. Pretty sure most of us aren't running anywhere close to full load 24/7, but whoa, Irish power is expensive. In the central US I pay $0.14/KWh.

cries in west coast peak $0.71/KWh rate

Re: I regret building this $3000 Pi AI cluster

#349

Post by EnPissant »

int_19h wrote: post #335
EnPissant wrote: post #239

Earlier quoted context omitted.

I think the Mac Studio is a poor fit for gpt-oss-120b. On my 96 GB DDR5-6000 + RTX 5090 box, I see ~20s prefill latency for a 65k prompt and ~40 tok/s decode, even with most experts on the CPU. A Mac Studio will decode faster than that, but prefill will be 10s of times slower due to much lower raw compute vs a high-end GPU. For long prompts that can make it effectively unusable. That’s what the parent was getting at.…

This is Mac Studio M1 Ultra with 128Gb of RAM. > llama-bench -m ./gpt-oss-120b-MXFP4-00001-of-00002.gguf -ngl 999 -fa 1 --mmap 0 -p 65536 -b 4096 -ub 4096 | model | size | params | backend | threads | n_batch | n_ubatch | fa | mmap | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | -: | ---: | --------------: | -------------------: | | gpt-oss 120B…

Thanks. That’s better than I expected. It's only 8.3x worse than a 5090 + CPU: 167s latency.

Re: I regret building this $3000 Pi AI cluster

#350

Post by 0manrho »

anaganisk wrote: post #320
qmr wrote: post #282

Earlier quoted context omitted.

No need for so much CPU power, any old quad core would work.

Aren’t newer CPUs especially AMDs more energy efficient?

If the point is a multi-tasking sandbox, not heavy/sustained data-crunching, those old CPU's w/ boosting turned off or a mild underclock/undervolt (or an L spec which comes iwth that out of the box) really aren't any more power hungry than a newer Ryzen unless you intend on running whatever you buy at high load for long times. Yeah, on paper it still could be a double digit percentage difference, but in reality we're talking a difference of 10W or 20W if you're not running stuff above 50% load for sustained periods.

Again, lots of variables there and it really depends on how heavily you intend to use/rely on that sandbox as to what's the better play. Regional pricing also comes into it.

Post reply on HN