I don't get k8s when you pay for EC2/Lambda. Why add another layer of complexity?
Kubernetes Failure Stories
171–180 of 203 posts
Re: Kubernetes Failure Stories
#172The Istio one hits home. It is the single scariest thing to work with in our kubernetes clusters. We've caused several outages by changing the smallest things.
Shouldn't even small changes be tested in a staging cluster before promoting to the production cluster?
We never test, in the rare situations we test, we first test it on production.
:)
Re: Kubernetes Failure Stories
#173Earlier quoted context omitted.
How is that K8s fault?
By that reasoning, how is anything ever anybody's fault? Caveat emptor, my product might not be fit for purpose, and it's your own damned fault if you think it is. Consumer protection? What's that?
What? Take some damn responsibility for your decision and stop blaming them on the tools. You solely decided to migrate to K8s while Jenkins was working fine and you solely couldn’t manage a cluster with 6 nodes. That’s your responsibility that the tool you have chosen isn’t fit for your purpose.
Re: Kubernetes Failure Stories
#174Earlier quoted context omitted.
By that reasoning, how is anything ever anybody's fault? Caveat emptor, my product might not be fit for purpose, and it's your own damned fault if you think it is. Consumer protection? What's that?
What? Take some damn responsibility for your decision and stop blaming them on the tools. You solely decided to migrate to K8s while Jenkins was working fine and you solely couldn’t manage a cluster with 6 nodes. That’s your responsibility that the tool you have chosen isn’t fit for your purpose.
In every crowd you will find a set of loud victim blamers.
Take some responsibility for your tools instead of getting defensive worrying that if someone actually yells at one tool author they're coming for you next.
I work my butt off to write good tools and half the time when someone comes to me about a problem, they're still right.
We have a giant Dancing Bear problem in our industry. We sit around hand wringing about all of the bad things people do out there but we never stop to think that it's environmental. Children with fucked up parents become fucked up adults. You surround them with chaos and you get more chaos. You surround them with pain and you get more pain. If you put developers in an environment where garbage tools are not only tolerated but are aggressively defended, what have they been inspired to produce? What can you expect them to produce?
More garbage.
So we go 'round and 'round victim blaming and never owning our part in this cycle of violence.
Re: Kubernetes Failure Stories
#175Definitely a good one: "Experiences with running PostgreSQL on Kubernetes - Gravitational - blog post 2018" For anyone who thinks running a database in a container environment is a neat idea, think again. I am guilty of using containers for temporary test databases, but the thought of running production databases in containers sends shivers down my spine.
Some companies make a business out of it: https://www.crunchydata.com/products/crunchy-postgresql-oper...
Re: Kubernetes Failure Stories
#176Earlier quoted context omitted.
What do you use Istio for? 90% of the usecases I hear about could be solved with simpler tooling.
I don’t use it yet, but I’m probably going to have to because my team is looking at using Kubeflow and it leverages Istio to do things like traffic-splitting and stuff so you can A/B test models without needing to handle that at your model code level. KNative can also do really cool things like per-request routing, scale-to-zero deployments that it can bring back up when it gets a request, it’s pretty rad.
My company has deployed Kubeflow for production model training. Early on, we used their big deployment, but we got frustrated trying to manage it with kfctl, so we started using kustomize directly and deploying only what we need, like KFP. So no istio for us! YMMV with your use case.
Re: Kubernetes Failure Stories
#177Earlier quoted context omitted.
By that reasoning, how is anything ever anybody's fault? Caveat emptor, my product might not be fit for purpose, and it's your own damned fault if you think it is. Consumer protection? What's that?
What? Take some damn responsibility for your decision and stop blaming them on the tools. You solely decided to migrate to K8s while Jenkins was working fine and you solely couldn’t manage a cluster with 6 nodes. That’s your responsibility that the tool you have chosen isn’t fit for your purpose.
At any given moment, half of developers have less that 5 years of experience.
I'm not just taking responsibility for my actions, I have to take responsibility for theirs, too. It's expected, but it would be the right thing to do even if it's not. Glitchy, dangerous tools make it difficult to grow other people into senior positions, where they can take responsibility for their own decisions. Every bad tool I use slows the process of creating a peer down.
Re: Kubernetes Failure Stories
#178Earlier quoted context omitted.
By that reasoning, how is anything ever anybody's fault? Caveat emptor, my product might not be fit for purpose, and it's your own damned fault if you think it is. Consumer protection? What's that?
What? Take some damn responsibility for your decision and stop blaming them on the tools. You solely decided to migrate to K8s while Jenkins was working fine and you solely couldn’t manage a cluster with 6 nodes. That’s your responsibility that the tool you have chosen isn’t fit for your purpose.
> You solely decided to migrate to K8s
A note: this person is different from the GP poster.
Re: Kubernetes Failure Stories
#179This is a compilation of gotcha-discovery-reports, distributed across the surface area of K8s (which, since K8s is huge, covers many nooks and crannies). This is not a compilation of "K8s sucks, here are 10 reasons why", which is what my kneejerk expectation was. (maybe I am too cynical...). Overall, this is a fantastic index to some very interesting resources. There's an old saying that you learn more from failure t…
At the same time your initial reaction must at the very least say something about how k8 is different from other technologies where you won't have this reaction.
Eh, I'd say most popular technologies suffer from the same problem.
There's always an accretion disc of "X is garbage/panacea, here's 10 reasons why" content around ubiquitous tools.
In fact, I think I can count on on hand, the number industry-standard tools that don't have these wakes. Can you think of any?
Re: Kubernetes Failure Stories
#180Earlier quoted context omitted.
What? Take some damn responsibility for your decision and stop blaming them on the tools. You solely decided to migrate to K8s while Jenkins was working fine and you solely couldn’t manage a cluster with 6 nodes. That’s your responsibility that the tool you have chosen isn’t fit for your purpose.
In every crowd you will find a set of loud victim blamers. Take some responsibility for your tools instead of getting defensive worrying that if someone actually yells at one tool author they're coming for you next. I work my butt off to write good tools and half the time when someone comes to me about a problem, they're still right. We have a giant Dancing Bear problem in our industry. We sit around hand wringing ab…
> In every crowd you will find a set of loud victim blamers.
I think the line has to be drawn somewhere on whether someone is a victim, or are complicit in their own suffering. If someone steps in doggy-doo on a walk, that's unfortunate and they might actually be a victim. However, if someone says "Me and my team have been stepping on doggy-doo everyday for the past 5 months", they are no longer victims, unless someone is coercing them into doing that against their will; and I will have some questions to ask their (technical) leadership.