Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes Failure Stories

k8s.af

Post reply on HN
Print view View original

181–190 of 203 posts

Re: Kubernetes Failure Stories

#181

Post by hinkley »

jacques_chester wrote: post #178
klohto wrote: post #173

Earlier quoted context omitted.

What? Take some damn responsibility for your decision and stop blaming them on the tools. You solely decided to migrate to K8s while Jenkins was working fine and you solely couldn’t manage a cluster with 6 nodes. That’s your responsibility that the tool you have chosen isn’t fit for your purpose.

> You solely decided to migrate to K8s A note: this person is different from the GP poster.

It hardly matters.

Every other industry has tools with finer tolerances than anything they produce. Except this one.

Then we blame each other for the quality of our output. It's madness of the first degree.

Re: Kubernetes Failure Stories

#182

Post by jacques_chester »

hinkley wrote: post #181
jacques_chester wrote: post #178

Earlier quoted context omitted.

> You solely decided to migrate to K8s A note: this person is different from the GP poster.

It hardly matters. Every other industry has tools with finer tolerances than anything they produce. Except this one. Then we blame each other for the quality of our output. It's madness of the first degree.

> Every other industry has tools with finer tolerances than anything they produce. Except this one.

I agree with this, actually, but I wanted to point out that there were some crossed wires.

Re: Kubernetes Failure Stories

#183

Post by FridgeSeal »

chakspak wrote: post #176
FridgeSeal wrote: post #131

Earlier quoted context omitted.

I don’t use it yet, but I’m probably going to have to because my team is looking at using Kubeflow and it leverages Istio to do things like traffic-splitting and stuff so you can A/B test models without needing to handle that at your model code level. KNative can also do really cool things like per-request routing, scale-to-zero deployments that it can bring back up when it gets a request, it’s pretty rad.

My company has deployed Kubeflow for production model training. Early on, we used their big deployment, but we got frustrated trying to manage it with kfctl, so we started using kustomize directly and deploying only what we need, like KFP. So no istio for us! YMMV with your use case.

Oh this is perfect to hear!

Yeah my plan is to use it the same way we use kubernetes: everything via kubernetes configs/justo use is friends and nobody using CLI’s, which I’m convinced are the text version of “click ops” hahaha.

So Istio isn’t strictly required? Can I ask which components you deploy? At minimum I’m planning on just the deployment and serving components and just use straight Polyaxon for training.

Re: Kubernetes Failure Stories

#184

Post by FridgeSeal »

hodgesrm wrote: post #165
FridgeSeal wrote: post #130

Earlier quoted context omitted.

It that I advocate running your own Postgres setup in your own cluster instead of just renting a managed version, but I’ve run a few databases on K8s and found it pretty fine: useful for when your hosting provider doesn’t support the database you want to run (Clickhouse managed AWS service when?) or for application-specific KV-stores: EBS volumes and PVC’s are great, solid performance, kubernetes takes care of the ne…

> (Clickhouse managed AWS service when?) Now! Altinity runs Altinity.Cloud now in AWS. Feel free to drop by. There are also services in other clouds. Yandex runs one in their cloud and there are at least 3 in China. ClickHouse has a big and active community of providers. Disclaimer: I work for Altinity.

> Now! Altinity runs Altinity.Cloud now in AWS. Feel free to drop by.

Fantastic to hear! This is so exciting.

Re: Kubernetes Failure Stories

#185

Post by trever123 »

avereveard wrote: post #48
tutfbhuf wrote: post #4

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…

What are the best practices for transactional data storage in multi cluster environments, just ditch databases and go for distributed, raft based, nosql variants?

CockroachDB works very well for this

Re: Kubernetes Failure Stories

#186

Post by TruthWillHurt »

thinkharderdev wrote: post #151
TruthWillHurt wrote: post #142

Here's mine - We were running on Cloud Foundry, had one DevOps person that mostly dealt with Jenkins, payed for 32-64GB RAM. Decided to move to K8s (Azure AKS), Three months later we have 4-6 DevOps people dealing with networking, cross-az replication, cluster size and autoscaling, And we're paying thousands of $$$ pm for a minimum of 6 64GB VMs. FAIL

Just curious, but if your old setup was working, why did you decide to move to Azure AKS?

Corporate decided to stop trying to compete with cloud vendors and shut down our in-house Cloud Foundry hosting. Also Microsoft sales folks worked client decision makers pretty hard.

Re: Kubernetes Failure Stories

#187

Post by TruthWillHurt »

klohto wrote: post #153
TruthWillHurt wrote: post #142

Here's mine - We were running on Cloud Foundry, had one DevOps person that mostly dealt with Jenkins, payed for 32-64GB RAM. Decided to move to K8s (Azure AKS), Three months later we have 4-6 DevOps people dealing with networking, cross-az replication, cluster size and autoscaling, And we're paying thousands of $$$ pm for a minimum of 6 64GB VMs. FAIL

How is that K8s fault?

Complicated, requires a lot of configuration, immature cloud managed service. one might say it's a step back from a PaaS offer like CF.

Re: Kubernetes Failure Stories

#188

Post by h43k3r »

pojzon wrote: post #82
avereveard wrote: post #48

Earlier quoted context omitted.

What are the best practices for transactional data storage in multi cluster environments, just ditch databases and go for distributed, raft based, nosql variants?

Thats a very good question Im also interested in knowing the answer. Currently I have seen that every company I had a pleasure to work with had to settle with „eventual” consistency of replicated masters. This is ofcourse totally unacceptable in case of some industries. Thats why I would like to know whether someone found a holy grail of multi-region database design and is willing to share it for free!

Lookup Google's Spanner. It's a distributed transactional SQL database with whatever consistency levels you are interested in

Disclaimer- Google employee who is heavily using Spanner

Re: Kubernetes Failure Stories

#189

Post by paulryanrogers »

ex_amazon_sde wrote: post #166
paulryanrogers wrote: post #85

Earlier quoted context omitted.

You mean the infra behind non-AWS services like retail Amazon.com? Or the infra underlying AWS itself?

The first one, but I'm trying not to divulge too many details on the architecture of Amazon - as you can imagine.

Fascinating because I would have thought the whole idea is AWS is dog-fooded by retail. Yet in truth AWS is itself too expensive and / or complex for Amazon itself! (Or at least the original part of it.)

Re: Kubernetes Failure Stories

#190

Post by fukmbas »

TruthWillHurt wrote: post #142

Here's mine - We were running on Cloud Foundry, had one DevOps person that mostly dealt with Jenkins, payed for 32-64GB RAM. Decided to move to K8s (Azure AKS), Three months later we have 4-6 DevOps people dealing with networking, cross-az replication, cluster size and autoscaling, And we're paying thousands of $$$ pm for a minimum of 6 64GB VMs. FAIL

Yeah, Azure is way more expensive than on-prem. We'll never stop all of the MBAs from running IT into the ground
Post reply on HN