Isn't this a KIAM bug? The default configuration of any piece of software should not cause pathological cases in other pieces of software that are commonly used with it. Maybe I'm just a bleeding heart, but I think good software delights its users; the deployment and configuration story is a part of this.
Blending complex systems made my latency 10x higher
31–40 of 86 posts
Re: Blending complex systems made my latency 10x higher
#32Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.
I think the point is that this was a real problem that happened because of combining k8s and aws, which is a pretty common scenario. And it underscores that the bug was hard to find - I'm not sure how many people on my team would be comfortable looking deeply at both GC and wireshark. It required asking "Why" a few more levels deep than bugs usually require, and I think a lot of developers would get stumped after the first couple of levels. So it's another piece of data just counseling that a proper k8s integration is not as easy as people might expect.
I also get the sense that that team has a better than average allocation of resources. Some teams I've been on, this type of problem would be the responsibility of one person within an afternoon, or with impatient product people and managers checking for status after that.
Re: Blending complex systems made my latency 10x higher
#33"We are blending complex systems that had never interacted together before with the expectation that they collaborate forming a single, larger system."
I suggest more people read the first few chapters of Specifying Systems by Lamport. Maybe the rest is good also, but that's as far as I got.
It works through a trivial system (display clock) and combines it with another trivial system (display weather).
Nothing Earth-shattering, but it really stuck with me. Thinking about it at that level gave me a new appreciation for what combining two systems means.
Re: Blending complex systems made my latency 10x higher
#34And this is why I find DevOps work more interesting than programming. System integration is just endlessly challenging. I always enjoy reading a well-documented integration debugging session!
In my opinion DevOps shouldn't exist at all.
Okay. So based on 20+ years of experience, I can say that most developers have no interest in automating deployment, configuration, monitoring, performance, logging, etc... Who should do this work?
Re: Blending complex systems made my latency 10x higher
#35Earlier quoted context omitted.
Specific to k8s would mean it only affects k8s, which is false. It affects a large group of systems, of which k8s is one. Better (more accurate) title would have been: "Merging complicated software systems made my latency 10x higher"
But with that title the author wouldn't have gotten those juicy clicks
I think that’s exactly OP’s point. We need to stop reinforcing this kind of behavior.
Re: Blending complex systems made my latency 10x higher
#36Earlier quoted context omitted.
In my opinion DevOps shouldn't exist at all.
Okay. So based on 20+ years of experience, I can say that most developers have no interest in automating deployment, configuration, monitoring, performance, logging, etc... Who should do this work?
This might be achieved with software tooling, though we're a while away from having a great solution. Lots of stuff hasn't been automated yet, and we tend to be afraid of some of trying it, but I like to think we'll be able to automate that someday.
Re: Blending complex systems made my latency 10x higher
#37Earlier quoted context omitted.
In other words, switching to Kubernetes _did_ make their latency higher — otherwise they wouldn't have needed to reconfigure anything, would they? If you want to help k8s, try starting a pull-request to make the defaults better rather than playing spin-doctor telling people that well-documented problems don't exist.
It wasn’t. It was KIAM and the Java SDK, neither of which are part of K8S.
KIAM is not part of the core Kubernetes system but it was a necessary component to avoid introducing a security regression as part of the switch.
Again, my point was that rather than trying to do PR damage-control it would be better to work to improve things so this doesn't understand. Someone went to the trouble of posting a detailed examination of a real problem with a fix and some references to upstream improvements. That's a lot more useful than trying to draw a line between one of two tools commonly used to meet security requirements when operating Kubernetes in one of the most popular cloud environments.
Re: Blending complex systems made my latency 10x higher
#38Agreed about poor title, but: > DNS resolution is indeed a bit slower in our containers (the explanation is interesting, I will leave that for another post). I would like to see this expanded upon or hear if anyone else has suffered similar.
Exactly! On our OpenShift production cluster we ran into ndots problems with DNS and slow DNS resolution overall. This blog post was very helpful in understanding the issue and ways to fix it, https://pracucci.com/kubernetes-dns-resolution-ndots-options...
Re: Blending complex systems made my latency 10x higher
#39And this is why I find DevOps work more interesting than programming. System integration is just endlessly challenging. I always enjoy reading a well-documented integration debugging session!
In my opinion DevOps shouldn't exist at all.
If you tackle the problem from an efficiency angle it makes sense. Just like how people advocate for keeping things DRY why should every developer worry about deployment pipelines, logging/monitoring/alerting systems, etc. It makes sense to have a dedicated team to worry about those issue so others don't.
Re: Blending complex systems made my latency 10x higher
#40Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.
I think the point is that this was a real problem that happened because of combining k8s and aws, which is a pretty common scenario. And it underscores that the bug was hard to find - I'm not sure how many people on my team would be comfortable looking deeply at both GC and wireshark. It required asking "Why" a few more levels deep than bugs usually require, and I think a lot of developers would get stumped after the…
This is exactly what happens when you abstract away anything, in this case, infrastructure. Most of the time people focus on the value-added by the abstraction. This time somebody had to face the additional burden that was introduced by it, making it harder to track down the bug.