Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Canonical introduces high-availability Micro-Kubernetes

zdnet.com

Post reply on HN
Print view View original

101–110 of 118 posts

Re: Canonical introduces high-availability Micro-Kubernetes

#101

Post by BenTheElder »

pmahoney wrote: post #12
koeng wrote: post #2

Anyone know how this compares to k3s? I’ve been using k3s for a while, and there have been a few bugs that made me a bit annoyed

This doesn't answer your question, but I'm piggy-backing with comments on k3s. For me, using k3s for development (not prod), the killer feature is running it in --docker mode where the node uses the local dockerd to run containers (vs. managing its own containerd instance). This allows building images locally with `docker build` and immediately using them in kubernetes pods _without_ first pushing to a (possibly loca…

[I work on kind amongst other things...]

If you want to turn your host into a node you can do this with --vm-driver=none in minikube or better yet just use `kubeadm init` directly. In KIND we point people to the latter -- the main thing we're doing is running inside a disposable container "node" of which you can have many.

Assuming you don't actually want to turn your host machine into a node managed by Kubernetes, you'll want to stick Kubernetes in a VM or container. If the rest of Kubernetes is in this container or VM, it doesn't make sense to be running containers out on the host, things like mounting volumes won't work, you need a consistent filesystem between kubelet and the container runtime.

With kind it's also important that we simulate multi-node and multi-cluster, which is not viable with a single container runtime instance.

Without actually running Kubernetes against the hosts's runtime you can't share storage. The way docker desktop does this is to run Kubernetes with docker as the node's container runtime while only supporting a single node/cluster in a VM and expose the same runtime for building.

For our test workloads it's important to have different clean clusters constantly for different tests / projects.

KIND and microk8s have made a bet on containerd, as kubernetes is actively moving away from dockershim towards CRI, so even if we exposed a node runtime you can't build with it.

It's indeed space-inefficient, but it's a tradeoff in isolation between projects etc. For multi-node you're going to wind up with multiple copies anyhow, and a lot of projects we work with wind up needing some multi-node testing.

Re: Canonical introduces high-availability Micro-Kubernetes

#102

Post by zaro »

jrockway wrote: post #7
candiddevmike wrote: post #6

This is very interesting, we're seeing a lot of Kubernetes "flavors" coming out that remove the etcd requirement. It's no secret that etcd is a key part of why Kubernetes is complex--etcd scaling/securing/recovery is really hard. I think Kubernetes would do well to make swapping out the storage backend possible without having all of these forks. Kubernetes is too tightly coupled to etcd, and for little benefit. I wou…

To some extent, the distributed consensus is kind of the important part. If you have a bunch of components that don't know what state they're supposed to be in, you don't really have a cluster. Looking at the marketing documentation (I didn't read the code), they just wrote their own consensus and datastore instead of using etcd. So the underlying fundamental computer science problems still exist, but nobody has been…

> In general, I'm not blown away with Canonical's track record.

Yeah, they really wanna be like Apple, but end up being more like Google, starting a lot of projects and abandoning them in couple of years.

Re: Canonical introduces high-availability Micro-Kubernetes

#103

Post by smarterclayton »

krimeo wrote: post #96
mroche wrote: post #90

Earlier quoted context omitted.

Red Hat is a subsidiary of IBM, not acquired and absorbed. It is very much its own thing within the company as a whole.

I really thought it was a merger. Is Red Hat still a seperate legal entity?

Yes, separate legal entity, combined financial entity (although I know little about the financial details).

Re: Canonical introduces high-availability Micro-Kubernetes

#105

Post by elderbarry »

jrockway wrote: post #7
candiddevmike wrote: post #6

This is very interesting, we're seeing a lot of Kubernetes "flavors" coming out that remove the etcd requirement. It's no secret that etcd is a key part of why Kubernetes is complex--etcd scaling/securing/recovery is really hard. I think Kubernetes would do well to make swapping out the storage backend possible without having all of these forks. Kubernetes is too tightly coupled to etcd, and for little benefit. I wou…

To some extent, the distributed consensus is kind of the important part. If you have a bunch of components that don't know what state they're supposed to be in, you don't really have a cluster. Looking at the marketing documentation (I didn't read the code), they just wrote their own consensus and datastore instead of using etcd. So the underlying fundamental computer science problems still exist, but nobody has been…

The database in question, dqlite, is raft-around-sqlite, and that’s just about the most widely used sql database anywhere, albeit pretty thin. Recovering files on disk won’t be an issue with dqlite. It would be interesting to see a jepsen analysis of dqlite to assess resilience in the face of trouble.

Re: Canonical introduces high-availability Micro-Kubernetes

#106

Post by elderbarry »

bayindirh wrote: post #19
jrockway wrote: post #7

Earlier quoted context omitted.

To some extent, the distributed consensus is kind of the important part. If you have a bunch of components that don't know what state they're supposed to be in, you don't really have a cluster. Looking at the marketing documentation (I didn't read the code), they just wrote their own consensus and datastore instead of using etcd. So the underlying fundamental computer science problems still exist, but nobody has been…

> I just don't trust Canonical much... They lost me when I've tried their ubuntu server when I was lazy one day and greeted with their Landscape advertisement in the MOTD display. Then I installed armbian's ubuntu version because Debian version was not ready and found out that MOTD was downloaded from web every time I log in. Add analytics (now opt-in), forcing snaps and their silent-ish efforts to monopolize the lan…

It’s easy enough to configure motd to pull your own notices for a company, and useful too. Nothing there seems particularly weird to me in this day and age, and Ubuntu has a much better track record of actual security maintenance over the long term.

Re: Canonical introduces high-availability Micro-Kubernetes

#107

Post by elderbarry »

emptysongglass wrote: post #86
tryauuum wrote: post #29

Earlier quoted context omitted.

we are using this thing — http://archive.ubuntu.com/ubuntu/ubuntu/dists/focal/main/ins... , it is loaded with PXE. It doesn't put a "customized image of some sort" on servers, it literally iterates through installer steps with the autoanswers provided by preseed file. You might say that it is customized since we wrote a preseed file. Well maybe, but it's not a "customized image". Also we didn't write any specific cod…

You may be in for a shock using Ubuntu Server 20.04 and later as it has switched to using cloud-init. Preseed files no longer work.

The new format does seem much nicer to my eyes but I suppose tastes vary. A pain to have to change but it looks like good work and a new release is usually required some pressed Hackett anyway.

Re: Canonical introduces high-availability Micro-Kubernetes

#108

Post by elderbarry »

koeng wrote: post #2

Anyone know how this compares to k3s? I’ve been using k3s for a while, and there have been a few bugs that made me a bit annoyed

Several commenters here say they have fewer bugs with MicroK8s, makes sense since it’s less of a modification than k3s. Slightly bigger since it keeps api and worker processes as separate binaries albeit in a single package.

Re: Canonical introduces high-availability Micro-Kubernetes

#109

Post by peterwwillis »

smarterclayton wrote: post #97
wahern wrote: post #68

Earlier quoted context omitted.

Distributed consensus doesn't require a leader. The same process used to elect a leader (i.e. change leader state) can simply be used to change state generally. See, e.g., There Is More Consensus in Egalitarian Parliaments: https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf In fact, for something like what etcd is used for in k8s, as the root and lynchpin of all state, but not direct application I/O (pods use the…

If you run etcd as a singleton you already get this behavior - consensus is shortcircuited and you only have to pay the cost to write to durable storage (which is already heavily batched). And you need durable writes so you can crash recover (kube has a gaping hole today in that a restore from an earlier time point breaks many controllers until a reconcile is performed). Note that Kubernetes requires a total ordering…

> I’ve never seen a production Kube system with HA etcd go down due to non-human error

I have, with OpenShift, several times. The worst one took hours to get it partially back online, days to fully recover. At least I got to have one free drink before I had to leave our holiday party to start fixing it. (Ok, technically it was non-prod; luckily prod was ECS... long story)

> I don’t believe single instance is going to give you better availability when single machine faults happen

Depends on your definition and context of availability. It is possibly the simplest thing to destroy any existing nodes (and in the event that a node can't be destroyed, null-route it and/or disable its network port) and bring up a new node. So single instance is fairly easy to recover. But with multiple nodes, for each function of each instance that is required to reach consensus, the likelihood of consensus failure increases; you actually need more nodes and variety in the cluster to resist consensus failure. Yet ironically, the more nodes you have, the higher the probability of failure. In the end, the real-world reliability of the system is based on additional factors besides the network model.

Re: Canonical introduces high-availability Micro-Kubernetes

#110

Post by e12e »

dhagz wrote: post #93
gavinray wrote: post #35

I am all for this trend. Microk8s and k3s are both a joy to develop with (though I have ran into a few bugs with k3s so tend to prefer Microk8s). How many folks are running self-managed k8s in production though, out of curiosity? It seems so economical to deploy k3s or use something like Rancher on dirt cheap VPS's from some place like Hetzner -- but what's the ops burden and failure risk like? Never tried it myself…

My employer has over 2000 clusters running k3s (I'm on the team that manages them), it's been...okay. We're on an old version though, so a lot of the k3s-specific issues we're running into are mostly fixed in more recent releases. The issues we run into more often are network or power related than k3s itself. Sometimes with the OS image the devices are running.

> My employer has over 2000 clusters running k3s

Oh wow.. Why so many clusters, and what kind of resources per cluster? Are you running 100 000s of physical servers?

Or are you just using k3s over three physical nodes as a way to achieve hardware redundancy and rolling hardware replacement?

Post reply on HN