Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Build a 28-Core Raspberry Pi Cluster

ikarus.sg

Post reply on HN
Print view View original

11–20 of 158 posts

Re: Let's Build a 28-Core Raspberry Pi Cluster

#11

Post by chabad360 »

magicalhippo wrote: post #7

So I was toying with the idea of something similar. I'm a programmer and not a sysadmin, but I've dabbled with Docker and had a specific use-case. But so far I've not come further than this: how to deal with the host OS? Like, I don't want to log into N computers and manually run "sudo apt update" etc. I want them to run the same OS but with slightly different configurations. Network booting via PXE seems to be the b…

It seems to be primarily for initially provisioning servers, and doesn't really seem to be targeted at their upkeep. There is Rundeck but it's far too resource heavy to be useful for rPi.

Shameless plug: After a lot of research in this, I came to the conclusion that I'd just have to make it myself, hence Covey (https://github.com/chabad360/covey) a project that's written in Go, and currently in an early alpha stage of development, but it works. More details are in the readme, feel free to email me (its on my GH profile) if you want more details than the readme provides.

EDIT: seems like your looking for something that's primarily for provisioning anyways. Whoops, I seem to have missed the middle paragraph...

EDIT 2: PXE booting may not be the best option for a relatively low ram machine like the Pi (unless you have the storage somewhere else, but that's its own can of worms), even the minimal image required to run k3s would be at least 400mb (I think I'm underestimating) there are ways to deal with that (using a compressed squashfs). But none will beat running directly on the SD card. I'd suggest following the sibling comment, and using overlayfs on top of the main fs to ensure that the image doesn't get tarnished.

Re: Let's Build a 28-Core Raspberry Pi Cluster

#12

Post by jvilalta »

magicalhippo wrote: post #7

So I was toying with the idea of something similar. I'm a programmer and not a sysadmin, but I've dabbled with Docker and had a specific use-case. But so far I've not come further than this: how to deal with the host OS? Like, I don't want to log into N computers and manually run "sudo apt update" etc. I want them to run the same OS but with slightly different configurations. Network booting via PXE seems to be the b…

Ansible works great on PIs

Re: Let's Build a 28-Core Raspberry Pi Cluster

#13

Post by megous »

mdoms wrote: post #5

Am I understanding that he is storing web data on the SD cards of the Pis? Is that normal? SD storage is notoriously unreliable, especially in write-heavy applications.

Brand SD cards are quite reliable. If you configure your SW to avoid sources of needless constant write churn (eg. like postgresql stats collector), and use good power supply, they will last years.

Re: Let's Build a 28-Core Raspberry Pi Cluster

#14

Post by giobox »

magicalhippo wrote: post #7

So I was toying with the idea of something similar. I'm a programmer and not a sysadmin, but I've dabbled with Docker and had a specific use-case. But so far I've not come further than this: how to deal with the host OS? Like, I don't want to log into N computers and manually run "sudo apt update" etc. I want them to run the same OS but with slightly different configurations. Network booting via PXE seems to be the b…

If you plan to boot from sd cards (most common configuration), 95 percent of work will typically be the image for the first machine. You can then typically just flash the same image to however many pi sd cards you want. Beyond that, shell script/ansible/whatever can get the job done at boot time for whatever per node setup you might need - there’s tons of ways to accomplish this.

Re: Let's Build a 28-Core Raspberry Pi Cluster

#15

Post by Legogris »

magicalhippo wrote: post #7

So I was toying with the idea of something similar. I'm a programmer and not a sysadmin, but I've dabbled with Docker and had a specific use-case. But so far I've not come further than this: how to deal with the host OS? Like, I don't want to log into N computers and manually run "sudo apt update" etc. I want them to run the same OS but with slightly different configurations. Network booting via PXE seems to be the b…

For the common base image, I just build a normal Debian image, customize it (chroot) and flash it directly. For customization, maintenance and operations you want some sort of CM (Ansible, Salt, Chef or Puppet). There is also Packer, but it's not that easy to use for bare-metal images for some reason.

You could also do something like userdata where you have a script or playbook that runs on first boot and performs additional setup.

I've been meaning to try out PXE booting but am yet to.

Re: Let's Build a 28-Core Raspberry Pi Cluster

#16

Post by justinclift »

daneel_w wrote: post #4

Neat setup. An alternative would perhaps be a Pine64 Clusterboard with 7 SOPINE modules: https://store.pine64.org/product/clusterboard-with-7-sopine-...

Be careful of that. Those Clusterboards seem like a nifty idea, but looking through their forum they seem to have a tonne of problems:. :(

http://forum.pine64.org/forumdisplay.php?fid=91

That being said, I haven't personally used one. (But might get around to it one day out of curiosity.)

Re: Let's Build a 28-Core Raspberry Pi Cluster

#17

Post by sdab »

magicalhippo wrote: post #7

So I was toying with the idea of something similar. I'm a programmer and not a sysadmin, but I've dabbled with Docker and had a specific use-case. But so far I've not come further than this: how to deal with the host OS? Like, I don't want to log into N computers and manually run "sudo apt update" etc. I want them to run the same OS but with slightly different configurations. Network booting via PXE seems to be the b…

I am partial to the iPXE boot idea, you may find https://netboot.xyz useful.

As for how to prepare the images, you need a ramfs image with the root filesystem as a cpio file and a kernel as a vmlinuz file. You then serve these in a basic HTTP file server that your iPXE config points to.

Generating the vmlinuz file is pretty easy given a kernel. The cpio file is trickier. netboot.xyz recommends just grabbing a standard distribution. Otherwise this reduces to the problem of creating your own.

Re: Let's Build a 28-Core Raspberry Pi Cluster

#18

Post by Legogris »

daneel_w wrote: post #4

Neat setup. An alternative would perhaps be a Pine64 Clusterboard with 7 SOPINE modules: https://store.pine64.org/product/clusterboard-with-7-sopine-...

I've been a bit curious about this and other multi-board CoM carrier boards like Turing Pi[0]. Have you used it and what's your experience?

Like the author mentions, pre-4 Raspberry Pis gives sad throughput due to the shared bus; Rpi 4 would be much more suitable. There's also Odroid C4, Rockpi 4, Nanopi M4V2.

[0]: https://www.jeffgeerling.com/blog/2020/raspberry-pi-cluster-...

Re: Let's Build a 28-Core Raspberry Pi Cluster

#19

Post by chabad360 »

Legogris wrote: post #15
magicalhippo wrote: post #7

So I was toying with the idea of something similar. I'm a programmer and not a sysadmin, but I've dabbled with Docker and had a specific use-case. But so far I've not come further than this: how to deal with the host OS? Like, I don't want to log into N computers and manually run "sudo apt update" etc. I want them to run the same OS but with slightly different configurations. Network booting via PXE seems to be the b…

For the common base image, I just build a normal Debian image, customize it (chroot) and flash it directly. For customization, maintenance and operations you want some sort of CM (Ansible, Salt, Chef or Puppet). There is also Packer, but it's not that easy to use for bare-metal images for some reason. You could also do something like userdata where you have a script or playbook that runs on first boot and performs ad…

If you can, I'd suggest using unified kernel images, they get rid of the "whole PXE setup", you just send one file and you've booted.

Re: Let's Build a 28-Core Raspberry Pi Cluster

#20

Post by pengaru »

daneel_w wrote: post #4

Neat setup. An alternative would perhaps be a Pine64 Clusterboard with 7 SOPINE modules: https://store.pine64.org/product/clusterboard-with-7-sopine-...

I cannot recommend doing business with pine64, their customer service is nonexistent. If you cancel your order, or need to get ahold of someone @ sales@pine64.org for any exception, in my experience you may as well shout at the nearest wall.

- Someone out $165 spent on a cancelled pinephone order, no phone and no refund, complete radio silence.

Post reply on HN