Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Horrors of Upgrading Etcd Beneath Kubernetes

gravitational.com

Post reply on HN
Print view View original

71–79 of 79 posts

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#71

Post by thraxil »

zbentley wrote: post #11
brianwawok wrote: post #6

Earlier quoted context omitted.

Not the OP, but I have two kinds of persistent data. 1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s 2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.

That's a bit distressing. Most everywhere I've worked, the infrastructure that matters to the business has fallen into roughly two categories: Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing appli…

K8s and similar represent a current view on how systems should be designed and run from the ground up (largely based on the same observations you have made that the stateless workloads can be run with drastically less operational problems). If your architecture lines up with this approach (eg, following the 12-factor approach), there really are a lot of advantages. But no, legacy architected applications are just not going to benefit in the same way. I assume that there must be consultants and "thought leaders" out there who are pushing k8s and containerization as silver bullets and that's unfortunate.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#72

Post by king_nothing »

Lol. CoreOS and Hashicorp products often throw “cloud” and “discoverability” around but lack crucial features for ops supportability found in solutions that came before. Zookeeper, Cassandra, Couchbase didn’t evolve in a development vacuum chamber. New != better.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#73

Post by paulddraper »

mirceal wrote: post #17

This may be an unpopular opinion, but I’m not a big fan of containers and K8S. If your app needs a container to run properly, it’s already a mess. While what K8s has done for containers is freaking impressive, to me it does not make a lot of sense unless you run your own bare metal servers. Even then, the complexity it adds may not be worth it. Did I mention that the tech is not mature enough to just run on autopilot…

> If your app needs a container to run properly, it’s already a mess.

> If you’re in the cloud, VMs + autoscalling or fully managed services (eg S3, lambda, etc) make more sense

So running with container isolation is a mess, but serverless isolation is admirable?

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#74

Post by jacques_chester »

ec109685 wrote: post #28
jacques_chester wrote: post #25

Etcd misbehaving during upgrades or when a VM was replaced was a massive source of bugs for Cloud Foundry. There is no longer an etcd anywhere in Cloud Foundry.

Aren’t they introducing Kubernetes as part of Cloud Foundry: https://techcrunch.com/2018/04/20/kubernetes-and-cloud-found...

Sort of. Kinda. It's complicated. But insofar as Kubernetes becomes the container orchestrator, I imagine we will encounter some or all of those problems again.

Cloud Foundry's current orchestrator, Diego, is of similar vintage to Kubernetes. It now relies entirely on relational databases for tracking cluster state. Ditto other subsystems (eg, Loggregator). It scales just fine. MySQL, while not my personal favourite, has proved more reliable in practice than etcd. Some folks use PostgreSQL. Also more reliable in practice.

Paying customers care more about being more reliable in practice than being more reliable in theory.

I've ha-ha-only-seriously suggested we throw engineering support behind non-etcd cluster state. For example: https://github.com/rancher/k8s-sql

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#75

Post by amorousf00p »

I'm old school. I look at containers as jails and all the work to isolate applications in containers as of indifferent value given a flat plane process scope with MAC and application resource controls in well designed applications.

That is I default to good design and testing rather than boilerplate orchestration and external control planes.

All containers have done (popularly) in my opinion is add complexity and insecurity to the OS environment and encouraged bad behavior in terms of software development and systems administration.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#76

Post by mdekkers »

zbentley wrote: post #11
brianwawok wrote: post #6

Earlier quoted context omitted.

Not the OP, but I have two kinds of persistent data. 1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s 2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.

That's a bit distressing. Most everywhere I've worked, the infrastructure that matters to the business has fallen into roughly two categories: Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing appli…

require more expertise to solve in a way that doesn't require the application/clients to change.

Surprise! IT is hard and requires experts. Either hire one, or become one, but either way, the idea that "nobody in my enterprise/company/team need to understand the details of how stuff works" is crazy. Imagine a car-mechanic shop where nobody knows how an engine works: "we plugged in the computer tool, and it said something is wrong with your engine. I guess you need a new one"

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#77

Post by peterwwillis »

taeric wrote: post #21
peterwwillis wrote: post #7

I don't know about you, but my application is tested on a single platform/stack with a specific set of operations. When the operation of the thing I'm running on changes, my application has changed. It just can't be expected to run the same way. Upgrade means your app is going to work differently. Not only is the app now different, but the upgrade itself is going to be dangerous. The idea that you can just "upgrade a…

Picking a dependency for a system that does not have a live method of updating, is a terrible idea for a software system. Physical systems, like a car, have obvious limitations on what can be modified when. Similarly, software will have some limitations on what happens when you are updating. But accepting "upgrades can't be done easily" for software is putting much more limitations on the software than makes sense.

With the exception of like, binary patching of executables that use versioned symbols or some craziness like that, virtually all software cannot be upgraded while it is running and expected not to produce errors.

I mean, if you use a plug-in style system, you can program it to block operations while a module is reloaded or something. But most software is not designed this way. Especially with ancient monolithic models like Go programs.

Upgrades just can't be done easily in a complex system. You can do them without concern for their consequences, but that doesn't mean they're safe or reliable methods.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#78

Post by brianwawok »

zbentley wrote: post #11
brianwawok wrote: post #6

Earlier quoted context omitted.

Not the OP, but I have two kinds of persistent data. 1) Images / files / etc. It all lives in cloud storage ("s3"), outside of K8s 2) RDBMS data. You can just run as hosted sql (say CloudSQL) or a not-in-k8s VM. I have found no compelling reason to move my RDBMS into my k8s cluster.

That's a bit distressing. Most everywhere I've worked, the infrastructure that matters to the business has fallen into roughly two categories: Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing appli…

k8s solves state pretty well lately, but that doesn't mean you need to move everything into your cluster.

I use k8s for bin packing and easy rolling deployments. Neither of which matter for my DB (i.e. no such thing as an easy rolling postgres deployment (maybe citris?)), and I am not going to put anything else on my database server... so no bins to pack.

If you are doing one or the other, then sure worry about it from a k8s sense. But don't throw stuff in k8s for no reason. It makes stuff like rolling upgrades of your cluster way more scary.

Re: The Horrors of Upgrading Etcd Beneath Kubernetes

#79

Post by zbentley »

mdekkers wrote: post #76
zbentley wrote: post #11

Earlier quoted context omitted.

That's a bit distressing. Most everywhere I've worked, the infrastructure that matters to the business has fallen into roughly two categories: Category 1: stateless-"ish" workloads. More than 90% of hosts/containers used . . . less than 25% of operations headaches and time. Issues that happen here are solvable with narrow solutions: add caches, scale out, do very targeted, transparent fixes to poorly-performing appli…

require more expertise to solve in a way that doesn't require the application/clients to change. Surprise! IT is hard and requires experts. Either hire one, or become one, but either way, the idea that "nobody in my enterprise/company/team need to understand the details of how stuff works" is crazy. Imagine a car-mechanic shop where nobody knows how an engine works: "we plugged in the computer tool, and it said somet…

Yes, it is. But tooling can improve that. For example: having worked on DB2 and Postgres, and I'd choose Postgres for any new work barring extremely unique constraints--not because it means "I don't have to understand the details of how stuff works", but because it exposes how stuff works in a more intelligible, powerful, and useful way. Tools improve. Sometimes they repeat past mistakes--I'd pick Postgres over Hadoop/HBase too, until considerable duress. But sometimes? Sometimes tools learn from past mistakes--use those tools.

Not everyone complaining about the deficiencies of current-gen tools is doing so out of ignorance or laziness about their stack.

Post reply on HN