Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Horrors of Upgrading Etcd Beneath Kubernetes

gravitational.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#21

Post by taeric »

peterwwillis wrote: post #7

I don't know about you, but my application is tested on a single platform/stack with a specific set of operations. When the operation of the thing I'm running on changes, my application has changed. It just can't be expected to run the same way. Upgrade means your app is going to work differently. Not only is the app now different, but the upgrade itself is going to be dangerous. The idea that you can just "upgrade a…

Picking a dependency for a system that does not have a live method of updating, is a terrible idea for a software system.

Physical systems, like a car, have obvious limitations on what can be modified when. Similarly, software will have some limitations on what happens when you are updating. But accepting "upgrades can't be done easily" for software is putting much more limitations on the software than makes sense.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#22

Post by threeseed »

shoo wrote: post #19
mirceal wrote: post #17

This may be an unpopular opinion, but I’m not a big fan of containers and K8S. If your app needs a container to run properly, it’s already a mess. While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. Even then, the complexity it adds may not be worth it. Did I mention that the tech is not mature enough to just run on autopilot…

> If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Yes there is lock-in. Yes, if not properly arhitected it can be a mess. I've just rolled off a project (line of business web app + cluster of workers for background job processing), this more or less describes what we have running. Some of the architecture is a bit wrong (it…

> it's not obvious to me what the business case would be to containerise everything

Containers allow you to move apps trivially between environments and guarantee that they will just work. It allows you to isolate dependencies between apps e.g. Python 2 versus Python 3. It allows you to move apps between cloud providers or between on premise and cloud. With platforms like Kubernetes it allows you to easily scale and self heal when nodes die.

And compared to rewriting your app in Lambda which is expensive and complex it is simple to build a container as almost every language has automated tooling.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#23

Post by mirceal »

threeseed wrote: post #20
mirceal wrote: post #17

This may be an unpopular opinion, but I’m not a big fan of containers and K8S. If your app needs a container to run properly, it’s already a mess. While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. Even then, the complexity it adds may not be worth it. Did I mention that the tech is not mature enough to just run on autopilot…

This is such nonsense not to mention hilariously hypocritical since you criticise "chasing the latest hotness" but then advocate for a Serverless architecture. Anyway go and try and build a typical application with Lambda/SAM. It is a nightmare of complexity and all you are doing is moving your logic to AWS where you pay 100x the cost of just running it yourself in a container. And the idea that Kubernetes isn't matu…

Ha. EKS is a joke. I have written services powered by lambda. As a matter of fact I’ve used AWS/GCP/Azure. I mentioned lambda as an example of a fully managed service - it was not to “endorse” serverless.

Also if you look at a container long enough and squint you will see the words serverless emerge.

I also had the “pleasure” of working with K8S. If anything, this looks like a pretty good play from Google to get you to eventually run your workloads in GCP.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#24

Post by mirceal »

threeseed wrote: post #22
shoo wrote: post #19

Earlier quoted context omitted.

> If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Yes there is lock-in. Yes, if not properly arhitected it can be a mess. I've just rolled off a project (line of business web app + cluster of workers for background job processing), this more or less describes what we have running. Some of the architecture is a bit wrong (it…

> it's not obvious to me what the business case would be to containerise everything Containers allow you to move apps trivially between environments and guarantee that they will just work. It allows you to isolate dependencies between apps e.g. Python 2 versus Python 3. It allows you to move apps between cloud providers or between on premise and cloud. With platforms like Kubernetes it allows you to easily scale and…

Not sure I agree. If you need a container to do the things you mention you’re already in a pretty bad shape.

Not to mention that a lot of people don’t understand what a container is.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#26

Post by piva00 »

geertj wrote: post #18
philips wrote: post #3

Hey thanks for the article. I know etcd upgrades can look complex but upgrading distributed databases live is always going to be quite non-trivial. That said for many people taking some downtime on their Kube API server isn't the end of the world. The system, by design, can work OK for sometime in a degraded state: workloads keep running. A few things that I do want to try to clarify: 1) The strict documented upgrade…

> upgrading distributed databases live is always going to be quite non-trivial. I understand that the implementation of live upgrades for a distributed database will be complex but this post is about the user experience. Given enough resources, is there a reason that it can't be a single "upgrade now" command? Or maybe slightly more real-world, a 3 step process like: "stage update" -> "test update" -> "start update".

I agree with you, tooling is still lacking a lot in UX on k8s world.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#27

Post by piva00 »

otterley wrote: post #16
robszumski wrote: post #14

Earlier quoted context omitted.

I was around at CoreOS before Kubernetes existed. I don't recall exactly when etcd was chosen at the data store, but the Google team valued focus for this very important part of the system. etcd didn't have an embedded DNS server, etc. Of course, these things can be built on top of etcd easily. Upstream has taken advantage of this by swapping the DNS server used in Kubernetes twice, IIRC. Contrast this with Consul wh…

My understanding is that Google valued the fact that etcd was willing to support gRPC and Consul wasn't -- i.e., raw performance/latency was the gating factor. etcd was historically far less stable and less well documented than Consul, even though Consul had more functionality. etcd may have caught up in the last couple years, though.

My impression is that etcd works more in a lower-level data store abstraction than Consul, exactly why it's not so feature-rich but is used as building block. Consul packs more out-the-box if that's what you need.

Both are atill much better to operate than ZooKeeper.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#28

Post by ec109685 »

jacques_chester wrote: post #25

Etcd misbehaving during upgrades or when a VM was replaced was a massive source of bugs for Cloud Foundry. There is no longer an etcd anywhere in Cloud Foundry.

Aren’t they introducing Kubernetes as part of Cloud Foundry: https://techcrunch.com/2018/04/20/kubernetes-and-cloud-found...

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#30

Post by fen4o »

From my experience, running etcd in cluster mode simply creates too many problems. It can scale vertically very well and if you run etcd (and other Kubernetes control plane components) on top of Kubernetes you can get away with running only a single instance.
Post reply on HN