Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Horrors of Upgrading Etcd Beneath Kubernetes

gravitational.com

Post reply on HN
Print view View original

11–20 of 79 posts

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#11

Post by zbentley »

brianwawok wrote: post #6
SteveNuts wrote: post #4

Earlier quoted context omitted.

Where do you run your persistent apps?

Not the OP, but I have two kinds of persistent data. 1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s 2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.

That's a bit distressing. Most everywhere I've worked, the infrastructure that matters to the business has fallen into roughly two categories:

Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing application code.

Category 2: stateful workloads. Less than 10% of hosts/containers. 75% or more of operations headaches and time. Issues that happen here have less visibility, fewer short-term fixes ("just add an index and turn off the bad queries" only works so many times before you're out of low-hanging fruit), and require more expertise to solve in a way that doesn't require the application/clients to change.

If k8s and other next-gen technologies are only easing the first category, that makes me sad. It's like we have this sedan (off-the-shelf web technologies) that we have to take off-roading and it falls apart all the time. I don't want a better air conditioning system and more cushions in my seats; I want the vehicle to not break.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#12

Post by viraptor »

peterwwillis wrote: post #7

I don't know about you, but my application is tested on a single platform/stack with a specific set of operations. When the operation of the thing I'm running on changes, my application has changed. It just can't be expected to run the same way. Upgrade means your app is going to work differently. Not only is the app now different, but the upgrade itself is going to be dangerous. The idea that you can just "upgrade a…

It really depends on what your scenario/usage is. Sometimes it's a terrible idea: you wouldn't do a live code update on a server handling your blog. Sometimes it's just what you're aiming for: you have expensive hardware plugged into physical lines, there's not enough capacity to migrate data flow uninterrupted, you have to update in place without clients having more than X ms delay. Real life is virtually always the first case... But if you really need it, tech like Java hotswap or elang hot code swap is there.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#13

Post by wereHamster »

zbentley wrote: post #11
brianwawok wrote: post #6

Earlier quoted context omitted.

Not the OP, but I have two kinds of persistent data. 1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s 2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.

That's a bit distressing. Most everywhere I've worked, the infrastructure that matters to the business has fallen into roughly two categories: Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing appli…

In k8s you can easily host stateful services. You have persistent disks that you can attach to containers, and you also have StatefulSets if you have a stateful service that you want to have automatically scaled (https://kubernetes.io/docs/concepts/workloads/controllers/st...). You can use both to run a database (postgres) for example.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#14

Post by robszumski »

nvarsj wrote: post #10
djb_hackernews wrote: post #9

The clustering story for etcd is pretty lacking in general. The discovery mechanisms are not built for cattle type infrastructure or public clouds. ie it is difficult to bootstrap a cluster on a public cloud without first knowing the network interfaces your nodes will have or it requires you to already have an etcd cluster OR use SRV records. From my experience etcd makes it hard to use auto scaling groups for healin…

> From my experience consul seems to have a better clustering story but I'd be curious why etcd won out over other technologies as the k8s datastore of choice. That'd be some interesting history. That choice had a big impact in making etcd relevant, I think. As far as I know, etcd was chosen before kubernetes ever went public, pre-2014? So it must have been really bleeding edge at the time. I don't think consul was e…

I was around at CoreOS before Kubernetes existed. I don't recall exactly when etcd was chosen at the data store, but the Google team valued focus for this very important part of the system.

etcd didn't have an embedded DNS server, etc. Of course, these things can be built on top of etcd easily. Upstream has taken advantage of this by swapping the DNS server used in Kubernetes twice, IIRC.

Contrast this with Consul which contains a DNS server and is now moving into service mesh territory. This isn't a fault of Consul at all, just a desire to be a full solution vs a building block.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#15

Post by zaat »

wereHamster wrote: post #13
zbentley wrote: post #11

Earlier quoted context omitted.

That's a bit distressing. Most everywhere I've worked, the infrastructure that matters to the business has fallen into roughly two categories: Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing appli…

In k8s you can easily host stateful services. You have persistent disks that you can attach to containers, and you also have StatefulSets if you have a stateful service that you want to have automatically scaled ( https://kubernetes.io/docs/concepts/workloads/controllers/st... ). You can use both to run a database (postgres) for example.

I thought that this was the case too, but OP had a link (https://gravitational.com/blog/running-postgresql-on-kuberne...) to previous post that got me worried.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#16

Post by otterley »

robszumski wrote: post #14
nvarsj wrote: post #10

Earlier quoted context omitted.

> From my experience consul seems to have a better clustering story but I'd be curious why etcd won out over other technologies as the k8s datastore of choice. That'd be some interesting history. That choice had a big impact in making etcd relevant, I think. As far as I know, etcd was chosen before kubernetes ever went public, pre-2014? So it must have been really bleeding edge at the time. I don't think consul was e…

I was around at CoreOS before Kubernetes existed. I don't recall exactly when etcd was chosen at the data store, but the Google team valued focus for this very important part of the system. etcd didn't have an embedded DNS server, etc. Of course, these things can be built on top of etcd easily. Upstream has taken advantage of this by swapping the DNS server used in Kubernetes twice, IIRC. Contrast this with Consul wh…

My understanding is that Google valued the fact that etcd was willing to support gRPC and Consul wasn't -- i.e., raw performance/latency was the gating factor. etcd was historically far less stable and less well documented than Consul, even though Consul had more functionality. etcd may have caught up in the last couple years, though.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#17

Post by mirceal »

This may be an unpopular opinion, but I’m not a big fan of containers and K8S.

If your app needs a container to run properly, it’s already a mess.

While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. Even then, the complexity it adds may not be worth it. Did I mention that the tech is not mature enough to just run on autopilot and now instead of worrying about the “devops” for your app/service you are playing catch-up with upgrading your K8s cluster?

If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Yes there is lock-in. Yes, if not properly arhitected it can be a mess.

I wish we would live in a world where people pick simple over complex and think long term vs chasing the latest hotness.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#18

Post by geertj »

philips wrote: post #3

Hey thanks for the article. I know etcd upgrades can look complex but upgrading distributed databases live is always going to be quite non-trivial. That said for many people taking some downtime on their Kube API server isn't the end of the world. The system, by design, can work OK for sometime in a degraded state: workloads keep running. A few things that I do want to try to clarify: 1) The strict documented upgrade…

> upgrading distributed databases live is always going to be quite non-trivial.

I understand that the implementation of live upgrades for a distributed database will be complex but this post is about the user experience. Given enough resources, is there a reason that it can't be a single "upgrade now" command? Or maybe slightly more real-world, a 3 step process like: "stage update" -> "test update" -> "start update".

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#19

Post by shoo »

mirceal wrote: post #17

This may be an unpopular opinion, but I’m not a big fan of containers and K8S. If your app needs a container to run properly, it’s already a mess. While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. Even then, the complexity it adds may not be worth it. Did I mention that the tech is not mature enough to just run on autopilot…

> If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense and allow you to focus on your app. Yes there is lock-in. Yes, if not properly arhitected it can be a mess.

I've just rolled off a project (line of business web app + cluster of workers for background job processing), this more or less describes what we have running.

Some of the architecture is a bit wrong (it wasn't designed for AWS but shifted there after it had been running for a year or so) but the system works well enough to deliver value to the business.

As a developer I hate state, things that aren't properly isolated, ill-defined system boundaries -- but it's not obvious to me what the business case would be to containerise everything.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#20

Post by threeseed »

mirceal wrote: post #17

This may be an unpopular opinion, but I’m not a big fan of containers and K8S. If your app needs a container to run properly, it’s already a mess. While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. Even then, the complexity it adds may not be worth it. Did I mention that the tech is not mature enough to just run on autopilot…

This is such nonsense not to mention hilariously hypocritical since you criticise "chasing the latest hotness" but then advocate for a Serverless architecture.

Anyway go and try and build a typical application with Lambda/SAM. It is a nightmare of complexity and all you are doing is moving your logic to AWS where you pay 100x the cost of just running it yourself in a container.

And the idea that Kubernetes isn't mature is pretty laughable. It's used everyday by Netflix, eBay, Apple, Microsoft, IBM, Lyft, Uber, Square, Google, Pinterest, Stripe, Airbnb, Yahoo, Salesforce etc. And with AWS you have EKS which allows you to run containers in a HA and managed way.

Post reply on HN