Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

srcco.de

Post reply on HN
Print view View original

201–210 of 242 posts

Re: Kubernetes Failure Stories

#201

Post by bpicolo »

tpxl wrote: post #189
bpicolo wrote: post #101

Earlier quoted context omitted.

> app developers or platform operators Definitely not the former. The YAML-based configuration is not a pleasant app deployment experience. Companies end up needing to do some sort of auto-generation for it to make it sane for app devs. App developers want experiences similar to heroku. They want to git push and have applications safely roll out without downtime or configuration.

My team has configured gitlab with post-commit hooks so that pushes to dev branch get compiled/packaged/deployed to the dev environment with a simple git push. Also, I don't find YAMLs bad for deployment.

There's no question that you can build higher-level deployment strategies on top of the system (or any other system). It's just not baked in.

Also, YAML in general is not a bad thing, but kubernetes presents you with a ton of boilerplate and a complicated DSL. There's a lot of rough edges (annotations don't get validated, for example).

Re: Kubernetes Failure Stories

#202

Post by geggam »

dlor wrote: post #136
cookiecaper wrote: post #120

Earlier quoted context omitted.

k8s is meant to be hard to use. You're supposed to rent space on a k8s cluster from Google. Google has been pumping millions into marketing k8s as a mechanism to improve GCP adoption and establish a foothold in the cloud provider space.

I'm not exactly sure what point you're trying to make here. k8s is not meant to be a paas, but no one is trying to make k8s harder to use. I work at Google on a large team of engineers dedicated to making it as easy as possible to use.

The fact it takes a team of highly skilled engineers from the top of the talent pool trying to make it easy should tell you perhaps the design is wrong ?

Re: Kubernetes Failure Stories

#203

Post by 013a »

pgwhalen wrote: post #116
manigandham wrote: post #110

I don't understand all the negative comments here, K8S solves many problems regardless of scale. You get a single platform that can run namespaced applications using simple declarative files with consolidated logging, monitoring, load-balancing, and failover built-in. What company would not want this?

I very much agree that kubernetes is useful in an environment that doesn’t need to scale, but do tell how it enables consolidated logging and monitoring, since my medium/small shop is spending quite some time setting up our own infrastructure for it.

Installing a managed log ingestor is stupidly easy in Kubernetes. For example, on GCP here's the guide to getting it done [1]. Two kubectl commands, and you get centralized logging across hundreds of nodes in your cluster and thousands of containers within them. Most other platforms (like Datadog) have similar setups.

Infrastructure level monitoring is also very easy. For example, if you're on Datadog, you flip KUBERNETES=true as an environment variable in the datadog agent, and you'll instantly get events for stopped containers, with stopped reason (OOM, evictions, etc), which you can configure granular alerting on.

Let's say you're in a service-oriented environment and you want detailed network-level metrics between services (request latency, status codes, etc). No problem, two commands and you have Istio [2]. Istio has Jaeger built-in for distributed tracing, with an in-cluster dashboard, or you can export the OpenTracing spans to any service that supports OpenTracing. You can also export these metrics to Datadog or most other metrics services you use.

[1] https://kubernetes.io/docs/tasks/debug-application-cluster/l...

[2] https://istio.io/docs/setup/kubernetes/quick-start/

Re: Kubernetes Failure Stories

#204

Post by thetechlead »

m0zg wrote: post #51

It's not for everyone and it has significant maintenance overhead if you want to keep it up to date _and_ can't re-create the cluster with a new version every time. This is something most people at Google are completely insulated from in the case of Borg, because SRE's make infrastructure "just work". I wish there was something drastically simpler. I don't need three dozen persistent volume providers, or the ability…

we used to maintain our own k8s cluster and it's a pain in the ass given we have no dedicated ops. the cluster crashed every one or two month and we never tried making it up to date.

I suggest every startup use a hosted k8s solution, which takes care of most things like authentication, networking, monitoring, updating, etc.

also keep away from templating system such as jsonnet which is a huge overkill. you will end up writing a lot code you will hate to read later. instead write your own yaml builder in CI, together with parts that do docker image building, and code that deploys the microservices

imo Google did a really smart move with open sourcing k8s, as a latecomer of cloud provider. now infrastructure become so insignificant since everything runs on docker and pods.

Re: Kubernetes Failure Stories

#205

Post by jacques_chester »

manigandham wrote: post #199
ukoki wrote: post #188

Earlier quoted context omitted.

"Every sufficiently large K8s deployment contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a PaaS" K8s may well be the best foundation upon which to build a PaaS, but I think building a PaaS should be met with the same eye-rolls as writing your own crypto. Use Heroku or GAE or Elastic Beanstalk or IBM BlueMix or Azure PaaS or Cloud Foundry or Openshift.

It's just YAML files, there's nothing to build.

You can go entirely with YAML in a PaaS. Unless you feel you want it.

Just push code. That's it. That's all you need to do.

Re: Kubernetes Failure Stories

#206

Post by abledon »

FridgeSeal wrote: post #53
cygned wrote: post #49

I am a developer and I find k8s frustrating. To me, its documentation is confusing and scattered among too many places (best example: overlay networks). I have read multiple books and gazillions of articles and yet I have the feeling that I am lacking the bigger picture. I was able to set it up successfully a couple of times, with more or less time required. Last time, I gave up after four days because I realized tha…

Have you used other google products? I find their documentation routinely incomprehensible and difficult.

I have a theory that the type of people who make it past the google interview are smart people who are bad at teaching. Like they get all the concepts, algos etc.. but when it comes to distilling it into an Explain-Like-Im-5 tutorial, it just goes to hell very quickly.

What they need to do is hire some people who are great teachers, explainers etc.. Avoid people who rely on already attained technical knowledge, design patterns, algos etc.. to pattern match on new tech to instantly grok it. The 'noob' people who question the engineers who designed the tools and ask a ton of dumb questions about how it works so they can then translate it into everyday tutorial paragraphs.

Re: Kubernetes Failure Stories

#207

Post by bvm »

dgruesso wrote: post #198
bvm wrote: post #195

Earlier quoted context omitted.

Lots of magic for me, I've broken a (dev) k8s cluster by installing Istio via Gitlab k8s integration. The overhead appeared to be non-negligible, but I noped-out-of-there pretty quick, so I don't have the data to back that up.

Hi bvm, GitLab PM here. Sorry to hear your dev cluster broke. Would like to offer any help we can provide and if possible learn more about the failure so we can take corrective action to avoid this in the future. Thanks.

Hi drugesso - Thanks for getting back to me. I actually signed up to premium to get support for this issue. Got one email back, replied and then never heard back :(

It would be really great if there is a human I could speak to at GitLab about this. I've put my email in my profile.

Re: Kubernetes Failure Stories

#208

Post by ravedave5 »

awinter-py wrote: post #8

Beyond strictly runtime failures, 2018 feels like the year that most of my friends tried kube but not everybody stayed on. The adoption failures are mostly networking issues specific to their cloud. Performance and box limits vary widely depending on cloud vendor and I still don't quite understand the performance penalty of the different overlay networks / adapters.

A team at my work has spent a stupid amount of time trying to nail down networking issues with hand rolled k8 in AWS. HAd to move away from using node ports to fix it. Total pain in the ass.

Re: Kubernetes Failure Stories

#209

Post by ldng »

m0zg wrote: post #51

It's not for everyone and it has significant maintenance overhead if you want to keep it up to date _and_ can't re-create the cluster with a new version every time. This is something most people at Google are completely insulated from in the case of Borg, because SRE's make infrastructure "just work". I wish there was something drastically simpler. I don't need three dozen persistent volume providers, or the ability…

There was Rancher 1.6 with Cattle was the sweet spot for us. Rancher 2 went full kubernetes. Probably makes sense for their customers. We're looking for a replacement in that sweet spot.

Re: Kubernetes Failure Stories

#210

Post by nvarsj »

zerogvt wrote: post #183
m0zg wrote: post #51

It's not for everyone and it has significant maintenance overhead if you want to keep it up to date _and_ can't re-create the cluster with a new version every time. This is something most people at Google are completely insulated from in the case of Borg, because SRE's make infrastructure "just work". I wish there was something drastically simpler. I don't need three dozen persistent volume providers, or the ability…

I think Istio ( https://istio.io ) is a nice effort to create both an abstraction on top of k8s and to package a set of commonly needed functionality out of the box. Unsure of its production status or overhead though. Also I'd only go with a managed k8s solution and I'm not sure I'd consider k8s for older or non-microservice/containerized architectures. In the later case though I don't think there's anything better o…

I have pretty mixed feelings about Istio. It's trying to solve a lot of fundamental problems by introducing yet another layer of stuff. It's basically the middleware box all over again.
Post reply on HN