Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

191–200 of 203 posts

Re: Kubernetes Failure Stories

#191

Post by tutfbhuf »

h43k3r wrote: post #188
pojzon wrote: post #82

Earlier quoted context omitted.

Thats a very good question Im also interested in knowing the answer. Currently I have seen that every company I had a pleasure to work with had to settle with „eventual” consistency of replicated masters. This is ofcourse totally unacceptable in case of some industries. Thats why I would like to know whether someone found a holy grail of multi-region database design and is willing to share it for free!

Lookup Google's Spanner. It's a distributed transactional SQL database with whatever consistency levels you are interested in Disclaimer- Google employee who is heavily using Spanner

Yes, but Google Spanner is not open source, so it's not so easy to actually look at it. But what is known to public is that Spanner relies on perfectly synchronized atomic clocks within their datacenters, called TrueTime. This solution is very special and does not work for other databases in general.

Re: Kubernetes Failure Stories

#192

Post by chakspak »

FridgeSeal wrote: post #183
chakspak wrote: post #176

Earlier quoted context omitted.

My company has deployed Kubeflow for production model training. Early on, we used their big deployment, but we got frustrated trying to manage it with kfctl, so we started using kustomize directly and deploying only what we need, like KFP. So no istio for us! YMMV with your use case.

Oh this is perfect to hear! Yeah my plan is to use it the same way we use kubernetes: everything via kubernetes configs/justo use is friends and nobody using CLI’s, which I’m convinced are the text version of “click ops” hahaha. So Istio isn’t strictly required? Can I ask which components you deploy? At minimum I’m planning on just the deployment and serving components and just use straight Polyaxon for training.

Yeah, there are quite a few applications that actively discourage the use of anything other than their CLI, which I find profoundly bizarre. Istio had a big warning in their docs that their Helm charts were deprecated, even though istioctl uses them under the hood. Funny that Kubeflow has its own Istio deployment, and they make you use kfctl to deploy it, haha.

It's not strictly required, but might be for you, since A/B deployment actually is a thing Istio does. We only use Kubeflow Pipelines currently; looking into Katib.

My feeling about Kubeflow is that it's a package of a lot of things that exist, with nice things on top, and an easy way to deploy those things. Only, it ends up not being that easy, and not easy at all to configure, and various features of the underlying tools are hard to get to or completely unavailable.

If you deployed your own Istio, you'd understand it end-to-end and could solve problems with it when it goes south, and you'd even understand what exactly you're using it for and why. The biggest problem I have with kfctl is that it basically asks for system:masters privilege to install everything and then you get to figure out what it did on your own. I don't want Istio, Cert-Manager, Knative, Argo, etc. deployments that I don't understand and can't easily configure because they're buried 5 levels deep in Kustomize overlays. These are all things I can install from elsewhere and with more documentation.

Kubeflow Pipelines is still a little mysterious, but the footprint is much smaller and not as invasive.

Never used Polyaxon. Gonna be Googling that!

Re: Kubernetes Failure Stories

#193

Post by Karunamon »

xorcist wrote: post #133
bacongobbler wrote: post #121

Earlier quoted context omitted.

Sounds more like a people problem than a tech problem.

Everything is a people problem at the end of the day.

This is a thought-terminating cliché.

The existence of right tools for the job implies the existence of wrong tools for the job. The engineer in GP's story used the wrong tool for the job. That is a people problem. Had he used the right tool for the job, the problem would not have existed. GP wrote their CI stuff in shell to solve a tech problem, not a people problem.

Re: Kubernetes Failure Stories

#194

Post by ivansavz »

squibbles wrote: post #89
ivansavz wrote: post #83

Earlier quoted context omitted.

What about single-host container (i.e. regular docker host, not k8s) with data partition mounted from host?

For me, personally -- I cannot think of a sufficient justification to put a production database in a container. A good database server is designed for performance, reliability, scalability, security, etc., without containers. Putting a production database inside a container introduces a world of unnecessary edge cases and complexity.

Thanks for your feedback. I might run the DB on the host then, and just use containers for the app server. I'm not at the scale to warrant a separate host for the DB.

Re: Kubernetes Failure Stories

#195

Post by neop1x »

squibbles wrote: post #89
ivansavz wrote: post #83

Earlier quoted context omitted.

What about single-host container (i.e. regular docker host, not k8s) with data partition mounted from host?

For me, personally -- I cannot think of a sufficient justification to put a production database in a container. A good database server is designed for performance, reliability, scalability, security, etc., without containers. Putting a production database inside a container introduces a world of unnecessary edge cases and complexity.

Depends on requirements. Someone needs one big, highly-optimized DB instance. Someone else needs high-availability 3+ instance cluster. Having a cluster of containers brings performance penalty but if your app is read-heavy, you can read from all instances and multiply read throughput...

Re: Kubernetes Failure Stories

#196

Post by neop1x »

mamon wrote: post #175
squibbles wrote: post #80

Definitely a good one: "Experiences with running PostgreSQL on Kubernetes - Gravitational - blog post 2018" For anyone who thinks running a database in a container environment is a neat idea, think again. I am guilty of using containers for temporary test databases, but the thought of running production databases in containers sends shivers down my spine.

Some companies make a business out of it: https://www.crunchydata.com/products/crunchy-postgresql-oper...

Percona XtraDB Operator (mysql-like) https://www.percona.com/doc/kubernetes-operator-for-pxc/inde...

Re: Kubernetes Failure Stories

#197

Post by neop1x »

cbushko wrote: post #25

The Istio one hits home. It is the single scariest thing to work with in our kubernetes clusters. We've caused several outages by changing the smallest things.

I recommend staying away from Istio unless you have experienced team of Isto ops and very good reasons for using it. I tried it a year ago and it was not pleasant at all. It looks nice (Kiali) when it works but it's a nightmare once it stops working. Using TLS complicates debugging. It lacked basic functuionality like simple Basic Auth (just recently implemented in wasm) and they deprecated original plugin architecture. At one point it started logging out certificate problems for me, I spent hours on it trying to figure out what happens. Then tried to upgrade to newer Istio and it magically fixed it so I think they shipped a broken build but comparing commits resulted in so many changes that I was unable to figure out what happened. It's unnecessary complicated layer which almost no one really needs! Maybe in the future it will be a standard but maybe not, I strongly don't recommend it unless you know what you are doing and have work/troubleshoot capacity.

Re: Kubernetes Failure Stories

#198

Post by neop1x »

legohead wrote: post #79

Surprised I don't see something about "context deadline exceeded" in there. This error has been plaguing me from Kubernetes 1.5 to 1.18, and nobody has a solid answer for it other than maybe it's Docker related. Solution? Restart your node(s).

It is just a timeout. Golang uses contexts which can have timeout set. Then you can do a request (HTTP/GRPC) under that context and if it times out before getting a response, this error is returned. To find the real reason you are supposed to know the architecture and monitor metrics. It can be network problem or kernel problem (resource exhaustion).

Re: Kubernetes Failure Stories

#199

Post by jen20 »

corty wrote: post #145
teach wrote: post #50

Earlier quoted context omitted.

I guess it depends on your definition of "at scale" but IMO with Hashicorp's stuff and maybe something like Backstage you get all the benefits k8s but in a package that's much simpler to reason about and manage.

By building a neat box with few buttons on the outside and still the same or more complexity on the inside. If something goes wrong, you still have to deal with all the inside parts. The only hope is that the few outside buttons only trigger happy paths without complications. "At scale" this seems unlikely, simplifications usually only work in smaller, simpler use cases.

Nomad very definitely does not beget the same gordion knot of complexity of Kubernetes, while working at substantially larger cluster sizes than Kubernernetes is capable of.

Re: Kubernetes Failure Stories

#200

Post by pid_0 »

_asummers wrote: post #95
pid_0 wrote: post #93

I will die on the hill that almost nobody actually needs kubernetes. You are not Netflix or Google.

Feel free, but it's a bad hill. I can write my own config management, secret management, volume mounting, deployments, replica scaling, hardware affinity, antiaffinity for resilience, rollover kicking out e.g. staging if there's not enough room to schedule production workloads, and the list goes on. I can also figure out how to install and upgrade each piece of software in my universe, across a few different language…

There are many options that are not kubernetes and not VMs. Cloud native offerings are extremely simple and powerful. ECS does almost every single thing kubernetes does with less headaches.
Post reply on HN