Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

srcco.de

Post reply on HN
Print view View original

211–220 of 242 posts

Re: Kubernetes Failure Stories

#211

Post by JeremyNT »

manigandham wrote: post #110

I don't understand all the negative comments here, K8S solves many problems regardless of scale. You get a single platform that can run namespaced applications using simple declarative files with consolidated logging, monitoring, load-balancing, and failover built-in. What company would not want this?

From a developers' perspective, k8s feels like a holy Grail. Having fully embraced it with my latest rails app, I can say confidently that I've never had a more straightforward and enjoyable experience than K8s. It's absolutely the correct abstraction layer for me; it gives me all the power I could ask for in as concise a definition as I could possibly expect.

I think a lot of the complaints against K8s are from the ops side of things. In my org, I don't actually run or upgrade the K8s cluster myself, so those pain points aren't mine to bear. When you're running your own k8s, the operational complexity of managing the cluster itself is not trivial and the change in mindset for traditional sysadmin types is a substantial hurdle.

My own take: K8s (or something very much like it) is absolutely the future, but the operational challenges of migrating to it at this time should not be ignored if you want to run it yourself and have existing ops experience. This will only get easier over time as tooling improves and sysadmins start seeing that this is the future they have to embrace.

Re: Kubernetes Failure Stories

#212

Post by bogomipz »

manigandham wrote: post #110

I don't understand all the negative comments here, K8S solves many problems regardless of scale. You get a single platform that can run namespaced applications using simple declarative files with consolidated logging, monitoring, load-balancing, and failover built-in. What company would not want this?

Logging and monitoring are not built in to K8S, at least not something you would rely on for operational purposes.

I believe most people use an EFK/ELK stack for centralized logging and Prometheus for Monitoring.

Re: Kubernetes Failure Stories

#213

Post by romeisendcoming »

Tehchops wrote: post #93
romeisendcoming wrote: post #65

Earlier quoted context omitted.

Or you could just learn how to manage infrastructure the old fashioned way, which was never broken for small business and mid-sized enterprise environments. The only time you need the complexity and overhead of something like kubernetes is when you are truly large or when you have caught the in-fashion disease. It's quite simple for a 20 year SA to stand up a highly integrated environment with modular monitoring, dir…

>Why don't you hire one of these for the job instead of recipe/containering yourself into 'doesn't work, I dunno' posts. Because every.single.one of these "integrated environments" I've ever come across was an objective mess, poorly documented and littered with tech-debt. It was clear the "20-year SA" had forced 20-year old administration abstractions and ideas on top of modern infrastructure and application concerns…

There is some of that, certainly, but you can misapply any solution and perform poorly using any tool or personality.

For your information I find the new breed and k8s FUD to be more of a reaction rather than an educated, seamless and practical systems approach.

Re: Kubernetes Failure Stories

#214

Post by romeisendcoming »

manigandham wrote: post #150
romeisendcoming wrote: post #65

Earlier quoted context omitted.

Or you could just learn how to manage infrastructure the old fashioned way, which was never broken for small business and mid-sized enterprise environments. The only time you need the complexity and overhead of something like kubernetes is when you are truly large or when you have caught the in-fashion disease. It's quite simple for a 20 year SA to stand up a highly integrated environment with modular monitoring, dir…

Why not hire an SA? Because K8S is free and runs well from small to large and is available on every cloud where the IT infrastructure already is. Why is it better to spend money to rebuild a fraction of K8S with a patchwork of infrastructure put together by a single person?

Sales talk. This doesn't reflect reality, sadly.

Re: Kubernetes Failure Stories

#215

Post by romeisendcoming »

jordanbeiber wrote: post #88
romeisendcoming wrote: post #85

Earlier quoted context omitted.

Who said anything about manual processes? That's not what the modern SA does...it's mostly designing repeatable processes, creating recipes and integration in my experience. The real problem with the K8s and devops world is no understanding of why there is no magic pill in 'codifying' a bad system.

I did. As I see it all to often mainly at the larger places. Modern SA is about knowing that your job to help bring business value. Most of the time this is down to automation.

Did you consider that when there is a manual process it is in place to bring combined attention to what surely is (by 2019) a critical section: that needs consensus not provided by some monitoring hook. Sure, everything is about automation and it has been since 1999 in my experience.

Re: Kubernetes Failure Stories

#216

Post by ukoki »

manigandham wrote: post #199
ukoki wrote: post #188

Earlier quoted context omitted.

"Every sufficiently large K8s deployment contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a PaaS" K8s may well be the best foundation upon which to build a PaaS, but I think building a PaaS should be met with the same eye-rolls as writing your own crypto. Use Heroku or GAE or Elastic Beanstalk or IBM BlueMix or Azure PaaS or Cloud Foundry or Openshift.

It's just YAML files, there's nothing to build.

It's just YAML until you need:

* multi-tenancy

* auto-scaling

* security auditing

* automated os/container patching

* multi-tenant self-service route/ingress management

* multi-tenant self-service logging, monitoring and alerting

* multi-tenant self-service databases

and so on

Re: Kubernetes Failure Stories

#217

Post by dsumenkovic »

bvm wrote: post #207
dgruesso wrote: post #198

Earlier quoted context omitted.

Hi bvm, GitLab PM here. Sorry to hear your dev cluster broke. Would like to offer any help we can provide and if possible learn more about the failure so we can take corrective action to avoid this in the future. Thanks.

Hi drugesso - Thanks for getting back to me. I actually signed up to premium to get support for this issue. Got one email back, replied and then never heard back :( It would be really great if there is a human I could speak to at GitLab about this. I've put my email in my profile.

Hello Tom, thanks for reaching out about this. I've forwarded your request to the team internally, please let us know when everything is sorted out :-).

Re: Kubernetes Failure Stories

#218

Post by manigandham »

romeisendcoming wrote: post #214
manigandham wrote: post #150

Earlier quoted context omitted.

Why not hire an SA? Because K8S is free and runs well from small to large and is available on every cloud where the IT infrastructure already is. Why is it better to spend money to rebuild a fraction of K8S with a patchwork of infrastructure put together by a single person?

Sales talk. This doesn't reflect reality, sadly.

What's sales talk? What's not reality?

K8S can replace quite a lot of sysadmin responsibilities and is currently doing so at thousands of companies.

Re: Kubernetes Failure Stories

#219

Post by NikolaeVarius »

sgt101 wrote: post #77
dehrmann wrote: post #10

Earlier quoted context omitted.

I have two. One was caused by data inconsistency between services and regions. One is more hypothetical: the microservices had gotten to the point that no one knew how to start the system if all services are down, and it's possible that services have circular dependencies to the point that it would be incredibly hard to do a cold start.

I think that if there is a genuine circular dependency then the services won't start ever. But I think it is possible to introduce services that assume other services are up and have an apparent dependency circularity. The trick is to have all your services resilient to it's start requirements not being met - basically the service has to back off and wait if information it needs isn't yet in the environment... and th…

That's fine, until the environment doesn't come up because a hundred services all queried the source of truth simultaneously and caused load that prevented any service from getting its config

Re: Kubernetes Failure Stories

#220

Post by manigandham »

jacques_chester wrote: post #205
manigandham wrote: post #199

Earlier quoted context omitted.

It's just YAML files, there's nothing to build.

You can go entirely with YAML in a PaaS. Unless you feel you want it. Just push code. That's it. That's all you need to do.

How is that different though? You have to setup the PaaS the first time, the same as you write the YAML file the first time.

After that you push code regardless of destination.

Post reply on HN