Live data from Hacker News

MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

microk8s.io

111–120 of 138 posts

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#111
post #11

Earlier quoted context omitted.

Care to explain why? And what industry you're talking about? Kubernetes has made efforts for nodes to work with poor network connectivity. The node requirements aren't that big. A lot of use cases are relatively easy to containerise. And edge / IoT devices are getting more powerful as well. They aren't talking about consumer IoT here as well. It has become a meme that Kubernetes is complicated. But it solves a lot of…

IDK about OP, but when I think IoT, I think small single purpose devices: a ring door bell, a "smart" thermostat or fire alarm, a security camera. There at most a couple of processes running, what is there to orchestrate on an IoT doorbell? On the backend where you have services being fed, processing, and presenting all that data sure Kubernetes that part up, but that doesn't seem to need a special distro of Kubernet…

    when I think IoT, I think small single purpose devices: a ring door bell, a "smart" thermostat or fire alarm, a security camera. There at most a couple of processes running, what is there to orchestrate on an IoT doorbell?
Exactly!

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#112
post #38

If anybody seriously believe that Kubernetes is good on Edge and IoT, our industry is in deep trouble. I usually like Canonical, but this is next level bullshit.

Canonical is just one of several players in this area. There's also k3s and k0s, for example. I suppose it might depend on what you count as "edge", but we're using kubernetes to distribute a complex product to customers onprem. The product has multiple databases, services, transient processes, scheduled jobs, and machine learning. It needs to be able to run on a single machine or a cluster depending on customer requ…

on-prem is not edge IMO. Edge is something small, far away from datacenters, close to customers with limited compute and storage capacity. Did I get this wrong?

For example, SQLite advertise itself as "database on edge".

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#113

If anybody seriously believe that Kubernetes is good on Edge and IoT, our industry is in deep trouble. I usually like Canonical, but this is next level bullshit.

Why? In its smallest form k8s is a few go processes handling a proxy and some container scheduling/runtime logic. Sure if you run an entire control plane on the edge you're adding more complexity... but you don't have to do that, and control planes are complex beasts by their nature.

IoT devices doesn't even have the memory required for running a k8s worker process. They should run a low-cost, very small embedded system. Most of them doesn't even have a Linux kernel.

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#114

Earlier quoted context omitted.

k3s is brilliant. we run production clusters on it. The problem with k3s is that the architecture level libraries are a bit outdated. Early on, it was for a particular reason - ARM64 (raspberry pi) support. But today, like everyone is on ARM - even AWS. For example the network library is Flannel. Almost everyone switches to Calico for any real work stuff on k3s. it is not even a packaged alternative. Go-do-it-urself.…

> For example the network library is Flannel. Almost everyone switches to Calico for any real work stuff on k3s. What's the tradeoff? Why not flannel for Real Work™?

You could certainly use Flannel in production (Canal = Flannel + Calico) but I like the features that Calico provides, in particular:

- network policy enforcement

- intra-node traffic encryption with wireguard

- calico does not use VXLAN (sends routes via BGP and does some gateway trickery[0]), so it has slightly less overhead

[0]: https://stardomsolutions.blogspot.com/2019/06/flannel-vs-cal...

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#115
post #60

I tried a few incarnations of self-hosted k8s a few years ago, and the biggest problem I had was persistent storage. If you are using a cloud service they will integrate k8s into whatever persistent storage they offer, but if you are self-hosting you are left on your own, it seems most people end up using something like nfs or hostPath - but that ends up being a single point of failure. Have there been any developmen…

I'm a bit biased but Rook[0] or OpenEBS[1] are the best solutions that scale from hobbyist to enterprise IMO.

A few reasons:

- Rook is "just" managed Ceph[2], and Ceph is good enough for CERN[3]. But it does need raw disks (nothing saying these can't be loopback drives but there is a performance cost)

- OpenEBS has a lot of choices (Jiva is the simplest and is Longhorn[4] underneath, cStor is based on uZFS, Mayastor is their new thing with lots of interesting features like NVMe-oF, there's localpv-zfs which might be nice for your projects that want ZFS, regular host provisioning as well.

Another option which I rate slightly less is LINSTOR (via piraeus-operator or kube-linstor[6]). In my production environment I run Ceph -- it's almost certainly the best off the shelf option due to the features, support, and ecosystem around Ceph.

I've done some experiments with a reproducible repo (Hetzner dedicated hardware) attached as well[7]. I think the results might be somewhat scuffed but worth a look maybe anyways. I also have some older experiments comparing OpenEBS Jiva (AKA Longhorn) and HostPath [8].

[0]: https://github.com/rook/rook

[1]: https://openebs.io/

[2]: https://docs.ceph.com/

[3]: https://www.youtube.com/watch?v=OopRMUYiY5E

[4]: https://longhorn.io/docs

[5]: https://github.com/piraeusdatastore/piraeus-operator

[6]: https://github.com/kvaps/kube-linstor

[7]: https://vadosware.io/post/k8s-storage-provider-benchmarks-ro...

[8]: https://vadosware.io/post/comparing-openebs-and-hostpath/

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#116

If anybody seriously believe that Kubernetes is good on Edge and IoT, our industry is in deep trouble. I usually like Canonical, but this is next level bullshit.

Edge is a very overloaded term that just means a server not in a datacenter imo, it could be a thick edge with 8 esx, hundreds or cores and gigabytes of memory in a shop or it could be a little single node box in a closet at someone’s home or it could be a light switch.

Technically they’re all edge, but nobody thinks K8s can run on the latter, but it might work on the prior two.

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#117

Is Snap the only way to install this on linux? This is a non-starter for me.

Doesn't look like it. https://github.com/ubuntu/microk8s/issues/988#issuecomment-6...

You probably meant "it looks like it" or "it doesn't look like there's another (supported) way". Really sad, and it's a "classic" snap so you don't get any isolation benefits but performance penalties and lots of mounted snap filesystems.

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#118

Can it replace Docker on my local dev machine?

The classic "it depends" answer applies here, along two axes: container assembly, and container execution. If you mean "docker" the binary used to build container images, you don't even need that right now -- there are multiple projects that will build container images without involving docker or dockerd. If you mean "dockerd" the container management engine that one controls via "docker" to start and stop containers…

Do you think Docker will disappear and everybody will some kind of "single binary kubernetes" on their machine + A tool to build the images.

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#119
post #97

I use MicroK8S for local testing of Kubernetes clusters on my laptop and it works pretty well. I like that I can just run microk8s reset to clear out the state so I can redeploy everything without fear of some lingering configuration laying behind. I have yet to deploy it to an actual server though, though I would definitely be interested if it could do what EKS can do to some capacity (mainly creating EBS volumes an…

Are you serious? Microk8s reset takes ages. It's faster to uninstall the whole thing. You also can just delete whole workspaces, that should normally be enough.

Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT

#120
post #84

(Full disclaimer - I'm an engineer at Talos, but I believe it's pretty relevant here) If folks are interested in this kind of K8s deployment, they might also be interested at what we're doing at Talos ( https://talos.dev ). We have full support for all of these same environments (we have a great community of k8s-at-home folks running with Raspberry Pis) and a bunch of tooling to make bare metal easier with Cluster AP…

A few questions if I may. I administer a few thousand machines which are provisioned using Puppet, and I've had my eye on a more immutable style of administration integrating K8S. - How does Talos handle first getting on to the network? For example, some environments might require a static IP/gateway for example to first reach the Internet. Others might require DHCP. - How does Talos handle upgrades? Can it self upgr…

Hey, thanks for the questions. I'll try to answer them in-line:

- How does Talos handle first getting on to the network? For example, some environments might require a static IP/gateway for example to first reach the Internet. Others might require DHCP.

For networking in particular, you can configure interfaces directly at boot by using kernel args.

But that being said, Talos is entirely driven by a machine config file and there are several different ways of getting Talos off the ground, be it with ISO or any of our cloud images. Generally you can bring your own pre-defined machine configs to get everything configured from the start or you can boot the ISO and configure it via our interactive installer once the machine is online.

We also have folks that make heavy use of Cluster API and thus the config generation is all handled automatically based on the providers being used.

- How does Talos handle upgrades? Can it self upgrade once deployed?

Upgrades can be kicked off manually with `talosctl` or can be done automatically with our upgrade operator. We're currently in the process of revamping the upgrade operator to be smarter, however so it's in flux a bit. As with everything in Talos, upgrades are controllable by the API.

Kubernetes upgrades can also be performed across the cluster directly with `talosctl`. We’ve tried to bake in a lot of these common operations tasks directly into the system to make it easier for everyone.

- What hardware can Talos run on? Does it work well with virtualisation?

Pretty much anything ARM64 or AMD64 will work. We have folks that run in cloud, bare metal servers, Raspberry Pis, you name it. We publish images for all of these with each release.

Talos works very well with virtualization, whether that's in the cloud or with QEMU or VMWare. We've got folks running it everywhere.

- To what degree can Talos dynamically configure itself? What I mean by this is if a new disk is attached, can it partition it and start storing things on it?

Presently, the machine configuration allows you to specify additional disks to be used for non-Talos functions, including formatting and mounting them. However, this is currently an install-time function. We will be extending this in the future to allow for dynamic provisioning utilizing the new Common Operating System Interface (COSI) spec. This is a general specification which we are actively developing both internally and in collaboration with interested parties across the Kubernetes community. You can check that out here if you have interest: https://github.com/cosi-project/community

- How resilient is Talos to things like filesystem corruption?

Like any OS, filesystem corruption can indeed occur. We use standard Linux filesystems which have internal consistency checks, but ultimately, things can go wrong. An important design goal of Talos, however, is that it is designed for distributed systems and, as such, is designed to be thrown away and replaced easily when something goes awry. We also try to make it very easy to backup the things that matter from a Kubernetes perspective like etcd.

- What are the minimum hardware requirements?

Tiny. We run completely in RAM and Talos is less than 100MB. But keep in mind that you still have to run Kubernetes, so there's some overhead there as well. You’ll have container images which need to be downloaded, both for the internal Kubernetes components and for your own applications. We're roughly the same as whatever is required for something like K3s, but probably even a bit less since we don’t require a full Linux distro to get going.

Post reply on HN