Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Horrors of Upgrading Etcd Beneath Kubernetes

gravitational.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#31

Post by aberoham »

a2tech wrote: post #29

What problems exactly is etcd trying to solve?

Interviewer here -- "If you want to form a multi-node cluster, you pass it a little bit of configuration and off you go." .. "Raft [etcd] gets you past this old single-node database mantra, which isn’t really particularly old or even a wrong way to go. Raft gives you a new generation of systems where, if you have any type of hardware problem, any type of software instability problem, memory leaks, crashes, etc., another node is ready to take over that workload because you are reliably replicating your entire state to multiple servers."

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#32

Post by philips »

geertj wrote: post #18
philips wrote: post #3

Hey thanks for the article. I know etcd upgrades can look complex but upgrading distributed databases live is always going to be quite non-trivial. That said for many people taking some downtime on their Kube API server isn't the end of the world. The system, by design, can work OK for sometime in a degraded state: workloads keep running. A few things that I do want to try to clarify: 1) The strict documented upgrade…

> upgrading distributed databases live is always going to be quite non-trivial. I understand that the implementation of live upgrades for a distributed database will be complex but this post is about the user experience. Given enough resources, is there a reason that it can't be a single "upgrade now" command? Or maybe slightly more real-world, a 3 step process like: "stage update" -> "test update" -> "start update".

That is the process: https://github.com/coreos/etcd/blob/master/Documentation/upg...

  for i in members
    Replace etcd binary
    Restart etcd process

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#33

Post by smarterclayton »

otterley wrote: post #16
robszumski wrote: post #14

Earlier quoted context omitted.

I was around at CoreOS before Kubernetes existed. I don't recall exactly when etcd was chosen at the data store, but the Google team valued focus for this very important part of the system. etcd didn't have an embedded DNS server, etc. Of course, these things can be built on top of etcd easily. Upstream has taken advantage of this by swapping the DNS server used in Kubernetes twice, IIRC. Contrast this with Consul wh…

My understanding is that Google valued the fact that etcd was willing to support gRPC and Consul wasn't -- i.e., raw performance/latency was the gating factor. etcd was historically far less stable and less well documented than Consul, even though Consul had more functionality. etcd may have caught up in the last couple years, though.

At the time gRPC was not part of etcd - that only arrived in etcd 3.x.

The design of etcd 3.x was heavily influenced by the Kube usecase, but the original value of etcd was that

A) you could actually do an reasonably cheap HA story (vs Singleton DBs)

B) the clustering fundamentals were sound (zookeeper at the time was not able to do dynamic reconfiguration, although in practice this hasn’t been a big issue)

C) consul came with a lot of baggage that we wanted to do differently - not to knock consul, it just overlapped with alternate design decisions (like a large local agent instead of a set of lightweight agents)

D) etcd was the simplest possible option that also supported efficient watch

While I wasn’t part of the pre open sourcing discussions, I agreed with the initial rationale and I don’t regret the choice.

The etcd2 - 3 migration was more painful than it could be, but most of the challenges I think were excacerbated by us not pulling the bandaid off early and forcing a 2-3 migration for all users right after 1.6.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#35

Post by bonesss »

brianwawok wrote: post #6
SteveNuts wrote: post #4

Earlier quoted context omitted.

Where do you run your persistent apps?

Not the OP, but I have two kinds of persistent data. 1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s 2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.

> ... not-in-k8s VM

One of the cooler innovations we've seen, and I think we're going to see more of, is the ability to take a non-k8s VM and expose it to the cluster as-if it were just another pod. This would let you schedule and expose RDBMs and other specialized servers through kubernetes while keeping them on a standard VM.

I think that's the win/win approach to bridge the gap.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#36

Post by outworlder »

This article really hits home.

A K8s cluster can survive just about anything. Worker nodes destroyed, meh, scheduler will take care of bringing stuff up. Master nodes destroyed. Meh. It doesn't care.

ETCD issues though? Prepare for a whole lot of pain. They are very uncommon though. Upgrading is the most frequent operation.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#37

Post by outworlder »

djb_hackernews wrote: post #9

The clustering story for etcd is pretty lacking in general. The discovery mechanisms are not built for cattle type infrastructure or public clouds. ie it is difficult to bootstrap a cluster on a public cloud without first knowing the network interfaces your nodes will have or it requires you to already have an etcd cluster OR use SRV records. From my experience etcd makes it hard to use auto scaling groups for healin…

There are several ways of bootstraping ETCD. The one I use is the one you mention: since they are brought up with Terraform, always on a brand new VPC, we can calculate what the IP addresses will be on Terraform itself and fill the initial node list that way. We can destroy an ETCD node if need be, and recreate. Granted, it is nowhere near close to being as convenient as an ASG.

The alternate method, and the method we used before, is to use an existing cluster, as you mention. If cattle self-healing is that important, perhaps you could afford a small cluster only for bootstrapping? Load will be very low unless you are bootstrapping a node somewhere. There are costs involved in keeping those instances 24/7, but they may be acceptable in your environment(and the instances can be small). Then the only thing you need is to store the discovery token and inject it with cloud init or some other mechanism.

That said, I just finish a task to automate our ELK clusters. For Elasticsearch I can just point to a load balancer which contains the masters and be done with it. I wish I could do the same for ETCD.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#38

Post by outworlder »

bonesss wrote: post #35
brianwawok wrote: post #6

Earlier quoted context omitted.

Not the OP, but I have two kinds of persistent data. 1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s 2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.

> ... not-in-k8s VM One of the cooler innovations we've seen, and I think we're going to see more of, is the ability to take a non-k8s VM and expose it to the cluster as-if it were just another pod. This would let you schedule and expose RDBMs and other specialized servers through kubernetes while keeping them on a standard VM. I think that's the win/win approach to bridge the gap.

Indeed.

But I think that doesn't go far enough: I want to provide K8s with a YAML file and have Kubernetes itself go and create and provision a VM for me. I.e, I want to "kill" Terraform. I don't care about a 'fabric' network (although that could be convenient), just give me an IP that I can reach even it if is external to the cluster. VMs could be just one more resource that can be created or destroyed by k8s, just like network-based storage today.

I haven't found this exact use-case implemented yet. If noone else builds it in the coming months I'll probably start doing it.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#39

Post by outworlder »

zaat wrote: post #15
wereHamster wrote: post #13

Earlier quoted context omitted.

In k8s you can easily host stateful services. You have persistent disks that you can attach to containers, and you also have StatefulSets if you have a stateful service that you want to have automatically scaled ( https://kubernetes.io/docs/concepts/workloads/controllers/st... ). You can use both to run a database (postgres) for example.

I thought that this was the case too, but OP had a link ( https://gravitational.com/blog/running-postgresql-on-kuberne... ) to previous post that got me worried.

From TFA

> Kubernetes is not aware of the deployment details of Postgres. A naive deployment could lead to complete data loss.

That sounds ominous, but is actually a tautology.

You have the exact same challenges anywhere else, but since K8s makes some operations so easy to do, you need to be careful. RDBMs are specially tricky because most of them expect a single "master" which holds special status. And it so happens to hold all your data too (as do your replicas, provided they are up to date).

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#40

Post by etaioinshrdlu »

threeseed wrote: post #20
mirceal wrote: post #17

This may be an unpopular opinion, but I’m not a big fan of containers and K8S. If your app needs a container to run properly, it’s already a mess. While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. Even then, the complexity it adds may not be worth it. Did I mention that the tech is not mature enough to just run on autopilot…

This is such nonsense not to mention hilariously hypocritical since you criticise "chasing the latest hotness" but then advocate for a Serverless architecture. Anyway go and try and build a typical application with Lambda/SAM. It is a nightmare of complexity and all you are doing is moving your logic to AWS where you pay 100x the cost of just running it yourself in a container. And the idea that Kubernetes isn't matu…

No project with 759 confirmed bugs on Github should be considered mature, in my opinion.
Post reply on HN