Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Blending complex systems made my latency 10x higher

srvaroa.github.io

Post reply on HN
Print view View original

71–80 of 86 posts

Re: Blending complex systems made my latency 10x higher

#71

Post by kazen44 »

outworlder wrote: post #63
goobynight wrote: post #42

Earlier quoted context omitted.

I think the best model is to have your "devops/sre guy" embedded right on the team. I market myself in interviews as a software engineer (on the same team as the other SWEs) that just happens to focus on what actually getting this product into production and keeping it going looks like. I got my start in this by working at a startup where we were responsible for everything just by the nature of the limited staff. I t…

> I think the best model is to have your "devops/sre guy" embedded right on the team. And here lies the problem. We had "Ops" before. People would throw stuff over the wall from Dev to Ops, and now it became "Ops" problem. Operations would stonewall releases – because releases bring problems, and they want to keep things stable. Developers would push for releases, because that's what they are paid to do. To solve thi…

> And, because they are developers at heart, they will do things to make their job easier and more predictable, like infrastructure as code. And heavy automation.

To be fair, most ops people i know worth their salt have been automating since forever. Doing operations at scale and still staying sane is impossible without automation.

The largests issue in ops is that reality in production is usually far more messy then development. Some minor issue which can be easily fixed in dev by something simple (take a reboot of a service for example) can be a major pain in production because of several factors. Usually related to interdependencies or customer impact.

Also, production (especially on the networking/hardware side) is usually hard because of things like hardware failure, physics itself or simply human error. These gems for instance[1][2].

[1] https://www.cisco.com/c/en/us/support/docs/field-notices/636...

[2]https://www.ibiblio.org/harris/500milemail.html

Re: Blending complex systems made my latency 10x higher

#72

Post by jaaron »

bavell wrote: post #13

Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.

I really like the title and I don't think it's link-bait-y.

Why?

Because too many engineers I've worked with would bump into this situation and this would be their answer. They wouldn't take the time to debug the situation deeply enough and they'd blame k8s, or blame the network, or blame...

In my experience, the most common issues with complex distributed systems are much more likely to be due to misconfiguration because of a limited understanding of the systems involved than such issues are to be caused by core, underlying bugs. And I believe that's why some engineers shy away from otherwise valuable frameworks and platforms: they have a natural and understandable bias to solve problems via engineering (writing code) than via messing with configuration parameters.

Re: Blending complex systems made my latency 10x higher

#73

Post by beat »

lostcolony wrote: post #64
tunesmith wrote: post #59

Earlier quoted context omitted.

People overlook that there's a common systematic belief that looks like this: 1) Developers don't code on prod 2) Prod needs to be protected 3) Therefore, developers need to be restricted from prod One of my teams went through the process of "let's do DevOps!" with the intent of giving developers the ability of pushing something all the way through to prod on AWS. Months later, this resulted in having a poorly-suppor…

There's a common systematic belief of that because that's the sort of thing a lot of actual compliance regulations de facto require (i.e., they demand controls around software deploys, and putting enforcing that in the same hands as those wanting to deploy it, i.e., devs, will fail an audit). Source: My employer is currently undergoing SOX compliance

And there's a good reason that separation of duties is in every compliance standard...

Re: Blending complex systems made my latency 10x higher

#74

Post by kodablah »

smarterclayton wrote: post #48
stuff4ben wrote: post #38

Earlier quoted context omitted.

Exactly! On our OpenShift production cluster we ran into ndots problems with DNS and slow DNS resolution overall. This blog post was very helpful in understanding the issue and ways to fix it, https://pracucci.com/kubernetes-dns-resolution-ndots-options...

Yeah, Tim Hockin and I still regret not designing the DNS name search process in Kube better. If we had, we would have avoided the need for 5 and could have kept 90% of the usability win of “name.namespace.svc” and “name” being resolvable to services without having to go to 5. And now we can’t change it by default without breaking existing apps. Forwards compatibility is painful.

Pardon my lack of Kubernetes knowledge, but any regrets supporting the hierarchical lookup where they don't have to qualify their dns requests (and maybe could have used some other way to find their "same namespace")?

Re: Blending complex systems made my latency 10x higher

#75

Post by nielsole »

threeseed wrote: post #67
pojzon wrote: post #56

Earlier quoted context omitted.

We found EKS to be really disappointing in comparison to Self hosted solution. Not only you simply cannot tweak extremely important kubelet configuration, you also cannot run real HA. Most of AWS implementation around EKS was simply terrible and outclassed by community driven projects. For me personally EKS is the same failed service as Elasticsearch Service. Good for low to medium size workloads but terrible for any…

Why can't you run "real HA" with EKS ? It is after all running the control plane in multiple AZs.

Also why can't you modify kubelet params? You are completely in charge of the nodes and can configure them freely.

Re: Blending complex systems made my latency 10x higher

#76

Post by manigandham »

acdha wrote: post #61
manigandham wrote: post #49

Earlier quoted context omitted.

It’s not PR. It’s being accurate that this has nothing to do with Kubernetes.

I would not describe a service which is only ever used on Kubernetes workers and is only necessary for code running on Kubernetes as having nothing to do with Kubernetes. The fact that you and the OP are so emotionally driven to find a way to dismiss it is what makes it sound like PR — why not just acknowledge there's a real problem which is being fixed and be glad someone documented it well enough to save other peop…

You're being strangely aggressive about this. Nobody cares about PR here, nor is anyone denying anything. We're just being accurate.

KIAM and the Java SDK have bad timeout overlap. That's the problem, has nothing to do with Kubernetes, and looks like it's well documented and now resolved.

Re: Blending complex systems made my latency 10x higher

#77

Post by MuffinFlavored »

threeseed wrote: post #66
MuffinFlavored wrote: post #60

Earlier quoted context omitted.

> this isn't specific to k8s at all. I don't know. k8s is pretty complicated. How many small/medium apps need more than this nginx/Terraform/Docker example? This would be a lot more difficult to set up in k8s (pods, ingress, etc.) Terraform: resource "docker_container" "app" { count = 2 name = "app-${count.index}" hostname = "app-${count.index}" image = "app:${var.app_version}" restart = "always" env = [ ... ] ports…

I don't know understand the point of this comment. Is there some epidemic of developers setting up complex, multi-node Kubernetes clusters just to run a small web app ?

How many developers need to be running multi-node complex Kubernetes clusters?

Re: Blending complex systems made my latency 10x higher

#78

Post by _y5hn »

outworlder wrote: post #63
goobynight wrote: post #42

Earlier quoted context omitted.

I think the best model is to have your "devops/sre guy" embedded right on the team. I market myself in interviews as a software engineer (on the same team as the other SWEs) that just happens to focus on what actually getting this product into production and keeping it going looks like. I got my start in this by working at a startup where we were responsible for everything just by the nature of the limited staff. I t…

> I think the best model is to have your "devops/sre guy" embedded right on the team. And here lies the problem. We had "Ops" before. People would throw stuff over the wall from Dev to Ops, and now it became "Ops" problem. Operations would stonewall releases – because releases bring problems, and they want to keep things stable. Developers would push for releases, because that's what they are paid to do. To solve thi…

DevOps is more about culture and collaboration. The tooling becomes necessary to truly make the processes go faster and better, ie. automated testing and CI/CD.

https://www.gartner.com/en/information-technology/glossary/d...

It's common to mistake DevOps meaning devs wearing ops "hats".

Re: Blending complex systems made my latency 10x higher

#79

Post by acdha »

manigandham wrote: post #76
acdha wrote: post #61

Earlier quoted context omitted.

I would not describe a service which is only ever used on Kubernetes workers and is only necessary for code running on Kubernetes as having nothing to do with Kubernetes. The fact that you and the OP are so emotionally driven to find a way to dismiss it is what makes it sound like PR — why not just acknowledge there's a real problem which is being fixed and be glad someone documented it well enough to save other peop…

You're being strangely aggressive about this. Nobody cares about PR here, nor is anyone denying anything. We're just being accurate. KIAM and the Java SDK have bad timeout overlap. That's the problem, has nothing to do with Kubernetes, and looks like it's well documented and now resolved.

This was a well-written post about a problem a lot of people would encounter with a common Kubernetes deployment. Instead of talking about that, most of the comments were people complaining about the title.

Re: Blending complex systems made my latency 10x higher

#80

Post by acdha »

threeseed wrote: post #69
acdha wrote: post #37

Earlier quoted context omitted.

KIAM is not part of the core Kubernetes system but it was a necessary component to avoid introducing a security regression as part of the switch. Again, my point was that rather than trying to do PR damage-control it would be better to work to improve things so this doesn't understand. Someone went to the trouble of posting a detailed examination of a real problem with a fix and some references to upstream improvemen…

There are many alternatives to kiam within the K8S space. So would disagree that is specifically is a necessary component.

It’s not necessary to use kiam - as I mentioned that multiple times - but you need something like that if you’re trying to maintain security coming from a good EC2/ECS deployment. Since it’s one of the more popular options, and Java is not uncommon, it seemed reasonable to consider this a likely pain point for many users.
Post reply on HN