51–60 of 65 posts
This is neat, but I'm really more interested to hear about potential use cases. I'm guessing this is mostly useful for ARM workloads? Maybe some situations with low power requirements? Personally, for my multi-node test clusters, I just run VMs on cheap x86 hardware.
What's the practical point of running containers on VM?
for this use case, i would rather use something like SimpleNUC[0] . It's basically just NUC machines with rack mount. It's a lot more powerful than Rpi, quite power-efficient (when compare to actual server rack) and dead silent. More importantly, this setup can handle some actual workload [0]: https://simplynuc.com/server-shelf-solution/
Earlier quoted context omitted.
I randomly came across a similar offering and thought about buying for fun: https://www.mininodes.com/product/5-node-raspberry-pi-3-com-... Any idea how the specs compare?
This is still based on the compute modules, so you're going to have the same tradeoffs. These are interesting ideas, but until the compute modules start having more on board RAM, you'd be much better off working with a few RPi 4's. They will be faster (1Gbps ethernet, more RAM), and cheaper since you only need a gigabit switch to connect them together, not a custom carrier board. If the compute modules start to get m…
3D printed trays for rockmount fiber cassette blanks and set it in a network rack.
Earlier quoted context omitted.
The other part of 'on your desk' is hearing damage. Server hardware vendors have traditionally not given two shits about their servers being north of 90 decibels, and I'm pretty sure I've witnessed a few that were pushing 100. That Raspberry Pi is probably going to absorb more noise than it makes.
Is there any reason you need a kubernetes cluster literally on your desk? Isn't the whole point of using these layers of abstraction over hardware that you pay someone else to manage it? The only times I can imagine needing hardware on my desk are when latency is super important or when I constantly need to manipulate the hardware (change hardware, play in bios, etc). In either case, I would not use k8s to run my sof…
That person is also needed in the future, and that person needs to start learning somewhere. A good start is messing around with it on a few Raspberry Pis
Earlier quoted context omitted.
To expand on my personal use-case, I don't want or need a whole stack of physical servers sitting around just for a test k8s environment. There are some things that you can only really test properly in a real multi-node environment rather than single-node solutions like minikube: failovers, shared storage, some networking particularities, etc. But there are reasons to run containers on VMs in production too. Hypervis…
Can you please elaborate a little bit more on the reason to run containers on VMs in production?
---
The default pod limit per node 110. This is by design and considered a reasonable upper limit for the kubelet to reliably monitor and manage everything without falling over into NotReady/PLEG status.
If your node has a ton of cpu and memory, then 110 pods will not come close to utilizing all the metal. You can go down the path of tuning and increasing the pod limit, but this is risky and often triggers other issues on components that are designed for more sane defaults.
It also means that if your node goes NotReady (not a hardware failure), it's now a much bigger deal because you have fewer total nodes and many more pods to re-schedule at once.
This is solved by splitting up these massive nodes into smaller nodes via virtualization.
It's also nice having an api-driven layer to manage and upgrade the vms versus shoehorning a bare-metal solution. I would argue it also encourages immutable infrastructure by making it much more accessible.
There are bare-metal solutions but it's often more complicated and slower than launching/destroying vms.
for this use case, i would rather use something like SimpleNUC[0] . It's basically just NUC machines with rack mount. It's a lot more powerful than Rpi, quite power-efficient (when compare to actual server rack) and dead silent. More importantly, this setup can handle some actual workload [0]: https://simplynuc.com/server-shelf-solution/
That takes it from $300 to $3000.
While Raspberry Pi's are awesome, and the power consumption is nothing to scoff at (when considering a cluster), you can accomplish this same thing for a lot less, and have quite a lot more compute power, by purchasing a used server or even something like an AMD 3600X... A single 3600X will grossly outperform this cluster (and cost less) with less headaches (you don't have N physical machines) by using KVM to deploy…
I have had the same setup for over 10 years, a PC in the garage running Ubuntu with lots of ram. I just upgrade it every so often with last-gen CPU and motherboard and swap-in the RAID controller.
I script all of my dev environments, if I need a K3S cluster it's literally 2 minutes (Bash, Python and some Ansible).
Want a set of RHEL machines? No problem, it's another script and 30 seconds.
For training you can't beat it. If I have to get out the "big guns" I have an old Dell tower server with 128GB of memory that I got cheap and has 12/24 threads. Sure it's a power-hog but I use it surgically.
Pis are great and you get a nice, limited blast-radius but the SD cards go wrong, they are slow at single-thread and they don't really have enough memory (Pi4 4GB being an exception). You can't beat x64 for compatibility either.
Earlier quoted context omitted.
To expand on my personal use-case, I don't want or need a whole stack of physical servers sitting around just for a test k8s environment. There are some things that you can only really test properly in a real multi-node environment rather than single-node solutions like minikube: failovers, shared storage, some networking particularities, etc. But there are reasons to run containers on VMs in production too. Hypervis…
Can you please elaborate a little bit more on the reason to run containers on VMs in production?
The main reason being on premise deployments. Our workload is primarily machine learning models, as such they are well suited to docker containers due to good support and being able to quickly build images, deploy and tear down.
Additionally, whilst the task would be well suited to a Kubernetes deployment which is ideal in the cloud, on prem we're likely running on a single box running multiple VM's. Hence we use containers within the VM.
It isn't the perfect solution and definitely has it's downsides, but it is good enough and better than the alternatives when considering everything possible.
Earlier quoted context omitted.
If you insist on using Pis (IMHO you're quickly at the point where a (potentially used) NUC or small office PC is the better choice), why would you go for extra expense and effort for PoE? Just connect a 5V PSU to the power pins? I guess it gives you individual power switching without any DIY, but other than that?
Simply for not having to deal with all the usb-c adaptors and cables. It's also pretty cool being able to turn the pis on or off by cycling the switch port from your network control plane (Unifi in my case) You could even do it dynamically via the api for a "bare-metal autoscaling" type workflow.
You can get one large 5V PSU and go directly to the pins on the Pi. No need to bother with individual USB adaptors...