Kubernetes Failure Stories • Hacker News Board

Re: Kubernetes Failure Stories

#91

Post by jordanbeiber » Sun, Jan 20, 2019, 9:37 PM UTC

lmm wrote: post #87

jordanbeiber wrote: post #56

Earlier quoted context omitted.

If you achieve this scale of teams suggested, it means you have a lot of implicit gains here as well, right? Such as higher quality, higher "velocity", separation of concern and hopefully a clear sense of ownership. These are some of the things in my opinion that allows you to scale.

> If you achieve this scale of teams suggested, it means you have a lot of implicit gains here as well, right? > Such as higher quality, higher "velocity", separation of concern and hopefully a clear sense of ownership. No, it means you pay a huge overhead. Quality and velocity both drop as your day-to-day development requires a lot more setup and faff to do anything, and counterintuitively so does separation of conc…

Which is why you’ll need the input and help from experienced sysadmins that like to do development, which actually is ”automation”.

This done right reduces overhead and gives you an edge in repeatability and quality. Both are required to scale.

Re: Kubernetes Failure Stories

#92

Post by stonewhite » Sun, Jan 20, 2019, 9:42 PM UTC

I managed multiple mesos+marathon clusters on production a little over 1.5 years, and when I switched over to the K8s the only thing that felt like an improvement was the kubectl cli.

I really liked/missed the beauty of simplicity in marathon that everything was a task, the load balancer, autoscaler, app servers everything. I think it failed because provisioning was not easy, lack of first-class integrations with cloud vendors and horrible horrible documentation.

Kind of sad to see it lost the hype battle, and since then even Mesosphere had to come up with a K8s offering.

Re: Kubernetes Failure Stories

#93

Post by Tehchops » Sun, Jan 20, 2019, 9:43 PM UTC

romeisendcoming wrote: post #65

dvnguyen wrote: post #21

Having used Docker Compose/Swarm for last two years, I remember having problems with them twice. One of which was an MTU setting which I didn't really understand why, but overall I was relatively happy with them. Since Kubernetes seems to have won, I decided to learn it but got some disappointments. The first disappointment is setting up a local development environment. I failed to get minikube running on a Macbook A…

Or you could just learn how to manage infrastructure the old fashioned way, which was never broken for small business and mid-sized enterprise environments. The only time you need the complexity and overhead of something like kubernetes is when you are truly large or when you have caught the in-fashion disease. It's quite simple for a 20 year SA to stand up a highly integrated environment with modular monitoring, dir…

>Why don't you hire one of these for the job instead of recipe/containering yourself into 'doesn't work, I dunno' posts.

Because every.single.one of these "integrated environments" I've ever come across was an objective mess, poorly documented and littered with tech-debt.

It was clear the "20-year SA" had forced 20-year old administration abstractions and ideas on top of modern infrastructure and application concerns. It was cheaper/better/easier to throw it out and rebuild on something like k8s than to make any attempting at "scaling" the existing solution.

You're simply trading the "in-fashion" disease for the "I'm a 20-year Linuxbeard I know best and no one tells me different" disease.

Re: Kubernetes Failure Stories

#94

Post by nisa » Sun, Jan 20, 2019, 9:47 PM UTC

The k8s hype feels like the Hadoop hype from a few years ago. Both solve problems that most don't have and there is a lot of complexity - some due to the nature of the problem, some because everything is new and moving.

Of course it's 2019 and you have to migrate Hadoop to run on k8s now :)

My impression is that if you are a small shop and have the money, use k8s on google and be happy, but don't attempt to set it up for yourself.

If you only have a few dedicated boxes somewhere just use Docker Swarm and something like Portainer.

Re: Kubernetes Failure Stories

#95

Post by jordanbeiber » Sun, Jan 20, 2019, 9:49 PM UTC

9dev wrote: post #89

jordanbeiber wrote: post #37

Earlier quoted context omitted.

Microservices requires proper, structured systems management. Most seem to think "devops" and microservices are ways to ignore this, when in fact the complete opposite applies. Make sure you have a true service delivery and service management pipeline in place where it is real easy for dev and ops to deploy and decommission services. Service metadata is key in my experience. Not too much though, just enough (such as…

But why!?

Good question! :)

My thoughts and experiences:

Because once you are up to speed the business and devs work more in concert, which lets things flow fast, especially if changes are small and many, rather than fewer and larger.

As devs grow secure in the infrastructure (”it kinda just works” from their end) as well as the ability to roll back within seconds, deploying things to production becomes no big deal.

The ”separation of concerns” and containerization enable some of this, but only part is tech — I find a great deal is rooted in the culture and people of the same mindset working together.

Re: Kubernetes Failure Stories

#96

Post by lugg » Sun, Jan 20, 2019, 9:52 PM UTC

nisa wrote: post #94

The k8s hype feels like the Hadoop hype from a few years ago. Both solve problems that most don't have and there is a lot of complexity - some due to the nature of the problem, some because everything is new and moving. Of course it's 2019 and you have to migrate Hadoop to run on k8s now :) My impression is that if you are a small shop and have the money, use k8s on google and be happy, but don't attempt to set it up…

Docker swarm is really nice. I wish it had more traction. I fear it's going to be dropped and leave me holding a bag full of bugs.

Re: Kubernetes Failure Stories

#97

Post by sonnyblarney » Sun, Jan 20, 2019, 9:56 PM UTC

m0zg wrote: post #90

trhway wrote: post #64

Earlier quoted context omitted.

An open source having complexity of an enterprise monster - this is what generates the half million plus salaries. An old enterprise software trick. Simplification of it would serve no interests of anybody in the position to do the simplification.

I'm going to voice a contrarian viewpoint here and say that "half million plus" salaries are actually a good thing. Rising tide lifts all boats and since a lot of technies live in areas with exorbitant cost of living, that money re-enters the economy at a rapid clip anyway. But such salaries are only good if commensurate value is being delivered for the money. Which in a large IT shop it might be, but as a small busi…

"a lot of techies live in areas with exorbitant cost of living,"

And what do you think is the primary driver of said cost of living?

I believe there is a ton of arbitrary complexity in the system, and though it's not specifically created, it definitely grows if it's not checked and entities with power have no reason to do that.

Google, Oracle, MS, Governments, Banks - have very little incentive to clear the weeds, usually just the opposite.

Wherever there is steady profit, there are layers of cruft.

Re: Kubernetes Failure Stories

#98

Post by InTheArena » Sun, Jan 20, 2019, 9:57 PM UTC

clofresh wrote: post #60

DavidWoof wrote: post #58

Earlier quoted context omitted.

I've had small-ish docker swarms in production for a couple of years as well, and I really don't understand why it doesn't seem to be popular at all. I feel like I need to move to K8S just because swarm seems to be going away, but I'm really not seeing the technical advantages at all. If someone could point me to an article explaining why k8s is so much better than swarm, I'd really appreciate it. Are the big advanta…

I'm also constantly surprised at how unpopular docker swarm is given that everyone already uses docker itself. Why do you think swarm is going away though? I love the idea of just using my docker compose file as my deployment config.

The second version of Swarm (the first version built into the stack) was really unstable. It got a bad reputation very quickly in one area that it could not afford to.

That said, I was at a conference, talking with a CIO of a K8s related company, back during the initial days of that startup ecosystem. I was told off the record that Google and Redhat were offering an "marketing budget" for companies that would come on board the Kubernetes ecosystem. The person rightly stated that K8s was going to roll everyone else because of it.

Can't validate that - it's hearsay, but I definitely think that it was a big factor

Re: Kubernetes Failure Stories

#99

Post by andyidsinga » Sun, Jan 20, 2019, 9:58 PM UTC

dehrmann wrote: post #10

tnolet wrote: post #3

I'd be interested in a related "microservices failure stories". Must be a big overlap with this.

I have two. One was caused by data inconsistency between services and regions. One is more hypothetical: the microservices had gotten to the point that no one knew how to start the system if all services are down, and it's possible that services have circular dependencies to the point that it would be incredibly hard to do a cold start.

> One is more hypothetical: the microservices had gotten to the point that no one knew how to start the system if all services are down,

this is a good one and have been thinking about this myself. Even with smallish projects that might have 10s of container based applications / services. In my case I end up with what is essentially a 3 tier architecture with each tier being a group of containers/machines with their own rules for startup/shutdown.

Re: Kubernetes Failure Stories

#100

Post by m0zg » Sun, Jan 20, 2019, 10:02 PM UTC

sonnyblarney wrote: post #97

m0zg wrote: post #90

Earlier quoted context omitted.

I'm going to voice a contrarian viewpoint here and say that "half million plus" salaries are actually a good thing. Rising tide lifts all boats and since a lot of technies live in areas with exorbitant cost of living, that money re-enters the economy at a rapid clip anyway. But such salaries are only good if commensurate value is being delivered for the money. Which in a large IT shop it might be, but as a small busi…

"a lot of techies live in areas with exorbitant cost of living," And what do you think is the primary driver of said cost of living? I believe there is a ton of arbitrary complexity in the system, and though it's not specifically created, it definitely grows if it's not checked and entities with power have no reason to do that. Google, Oracle, MS, Governments, Banks - have very little incentive to clear the weeds, us…

>> And what do you think is the primary driver of said cost of living?

Mostly NIMBY-driven refusal to build more housing and transportation infrastructure. It's not like the US is lacking for land. There's no reason a dilapidated teardown-ready shack should cost $2M, no matter where it is.