Meta. Is this a common usage of .af domains?
I only came here to comment that the domain name was fantastic for this usage.
Donated by Joe Beda himself.
Kubernetes Failure Stories
41–50 of 203 posts
Re: Kubernetes Failure Stories
#42The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…
The tech churn cycle is getting more and more insane. It's the same process repeating endlessly.
1. Identify one problem you want to fix and ignore everything else.
2. Make a tool to manage the problem while still ignoring everything else.
3. Hype the tool up and shove it in every niche and domain possible.
4. Observer how "everything else" bites you in the ass.
5. Identify the worst problem from #4, use it to start the whole process again.
Microservices will save us! Oh no, they make things complicated. Well, I will solve that with containers! On no, managing containers is a pain. Container orchestration! Oh no, our clusters fail.
Meanwhile, the complexity of our technology goes up and its reliability in practice goes down. Plus, the concepts we operate with are less and less tethered to reality, which makes even talking about certain issues really hard.
Re: Kubernetes Failure Stories
#43Earlier quoted context omitted.
I only came here to comment that the domain name was fantastic for this usage.
Maybe I should mutter[1] "That's Kubernetes as fuck" next time I see another shenanigans involving multiple overcomplicated layers with confusing documentation interacting with each other in a way nobody can figure out. [1] Just to myself, of course.
Is Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down.
That's basically what a Deployment in Kubernetes would do; create a Pod to provision compute resources and an IP, pull a container to copy your code into the environment, run a health check to see if that all went OK, and then update the Service's Endpoints to start sending it traffic. Then it would drain the old Pod in a reverse of starting a new one, perhaps running your configured cleanup steps along the way.
The difference between the example and what Kubernetes does is that Kubernetes is a computer program that can do this thousands of times for you without really making you think about getting the details right every time. Doesn't seem overly complicated to me. A lot of stuff? Sure. But it's pretty close to the right level of complexity. (Some may argue that this is even too simple. What about blue/green testing? What about canaries? When do my database migrations run? Kubernetes has no answer, and if it did, it certainly wouldn't make it simpler. Perhaps the real problem is that Kubernetes isn't complicated enough!)
Anyway, "that's Kubernetes as fuck" is kind of a good meme. I'll give you that. But a deep understanding usually makes memes less amusing.
Re: Kubernetes Failure Stories
#44Earlier quoted context omitted.
Maybe I should mutter[1] "That's Kubernetes as fuck" next time I see another shenanigans involving multiple overcomplicated layers with confusing documentation interacting with each other in a way nobody can figure out. [1] Just to myself, of course.
Is Kubernetes really overcomplicated, though? Say you wanted to do a release of a new version of your app. You'd probably boot up a VM for the new version, provision it an IP, copy your code to it, and check if it's healthy. Then you'd edit your load balancer configuration to send traffic to that new IP, and drain traffic from the old IP. Then you'd shut the old instance down. That's basically what a Deployment in Ku…
> Is Kubernetes really overcomplicated, though?
I imagine that a million voices read that and silently, immediately, think "yes, it is." and then giggle without commenting.
Re: Kubernetes Failure Stories
#45I picked two or three random stories, and learned a lot just from that, be it technology I didn't know about or just some trick for debugging. Recommended.
Re: Kubernetes Failure Stories
#46Meta. Is this a common usage of .af domains?
There's also contained.af [0], Jessie Frazelle's educational container-based CTF.
Re: Kubernetes Failure Stories
#47Can we do one for micro-services next?
I have one for a long time now: microservices.fail
Re: Kubernetes Failure Stories
#48The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance. For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to…
What are the best practices for transactional data storage in multi cluster environments, just ditch databases and go for distributed, raft based, nosql variants?
Re: Kubernetes Failure Stories
#49Earlier quoted context omitted.
I only came here to comment that the domain name was fantastic for this usage.
Donated by Joe Beda himself.
Heh - yeah. Henning does all the hard work of curating these. But I had the domain sitting around and figured that it was a good use of it. Sharing and learning through stories like this is how systems get better.
Re: Kubernetes Failure Stories
#50Earlier quoted context omitted.
"All happy families are alike; each unhappy family is unhappy in its own way." https://en.wikipedia.org/wiki/Anna_Karenina_principle
Not to knock on Tolstoy but there are many ways in which unhappy families can be grouped together. You have the alcoholic parents, the living-vicariously-through-their-children folks, the abusive parents, etc etc etc. To tie it back to Kubernetes, you have the scheduler-does-not-schedule lessons, the ingress-not-consistent lessons, the autoscaler-killed-all-my-pods lessons, the moving-stateful-services-with-60TB-stor…
I guess it depends on your definition of "at scale" but IMO with Hashicorp's stuff and maybe something like Backstage you get all the benefits k8s but in a package that's much simpler to reason about and manage.