Live data from Hacker News

Blending complex systems made my latency 10x higher

srvaroa.github.io

61–70 of 86 posts

Re: Blending complex systems made my latency 10x higher

#61
post #37

Earlier quoted context omitted.

KIAM is not part of the core Kubernetes system but it was a necessary component to avoid introducing a security regression as part of the switch. Again, my point was that rather than trying to do PR damage-control it would be better to work to improve things so this doesn't understand. Someone went to the trouble of posting a detailed examination of a real problem with a fix and some references to upstream improvemen…

It’s not PR. It’s being accurate that this has nothing to do with Kubernetes.

I would not describe a service which is only ever used on Kubernetes workers and is only necessary for code running on Kubernetes as having nothing to do with Kubernetes. The fact that you and the OP are so emotionally driven to find a way to dismiss it is what makes it sound like PR — why not just acknowledge there's a real problem which is being fixed and be glad someone documented it well enough to save other people time? Nobody is saying that Kubernetes is bad, or that you shouldn't use it, only that there's an interaction which Java users should know about if they're running on AWS.

Re: Blending complex systems made my latency 10x higher

#62
post #16

"Once this change was applied, requests started being served without involving the AWS Metadata service and returned to an even lower latency than in EC2." Title should be: My configuration made my latency 10x higher.

But that wouldn’t potentially kick off a circle jerk though...

Re: Blending complex systems made my latency 10x higher

#63

Earlier quoted context omitted.

In my opinion DevOps shouldn't exist at all.

I think the best model is to have your "devops/sre guy" embedded right on the team. I market myself in interviews as a software engineer (on the same team as the other SWEs) that just happens to focus on what actually getting this product into production and keeping it going looks like. I got my start in this by working at a startup where we were responsible for everything just by the nature of the limited staff. I t…

> I think the best model is to have your "devops/sre guy" embedded right on the team.

And here lies the problem.

We had "Ops" before. People would throw stuff over the wall from Dev to Ops, and now it became "Ops" problem. Operations would stonewall releases – because releases bring problems, and they want to keep things stable. Developers would push for releases, because that's what they are paid to do.

To solve this conflict, someone came with the idea of "DevOps". You don't have a split organization now, you have some developers in your team that can wear the "Ops" hat. That way, they experience production issues first hand, and that allows them to architect the application better – as well as ensuring it is thoroughly tested – noone wants to wake up at 2AM.

And, because they are developers at heart, they will do things to make their job easier and more predictable, like infrastructure as code. And heavy automation.

Somewhere along the way, people misread the whole idea, and now think that "devops" == Terraform or some other tool. Then they rebranded their Ops org as DevOps and hired a few automation folks and called it a day. Ops rebranded is still ops.

It was not supposed to be like that. If you have a "devops" team, you are not doing it correctly. Call it operations like it should be called. You may even have a dedicated team working on common tools or similar – which is fine – but you still need developers doing operations and ultimately responsible for their own shit, otherwise this is not devops.

> I got my start in this by working at a startup where we were responsible for everything just by the nature of the limited staff. I think that's how a lot of people in Devops get their start.

Yup. I got into that by trying (and ultimately failing) to launch my own startup. We didn't have dedicated operations folks, or time to manually tinker with servers, or to respond to repeated production issues. When you are sales, operations and engineering, you try your hardest to not have incidents (so that you don't lose customers), but you are trying to move fast to deploy features (so that you can sell).

Re: Blending complex systems made my latency 10x higher

#64

Earlier quoted context omitted.

Yeah, operations-focused engineers will continue to have a niche carved out for them because too many devs black-box infrastructure. Companies can either choose to have their devs take on ops responsibilities or continue having dedicated ops jobs. In either case, whether or not dedicated ops jobs exist, ops responsibilities always will. I'll be there to pick up the slack because designing and maintaining systems is a…

People overlook that there's a common systematic belief that looks like this: 1) Developers don't code on prod 2) Prod needs to be protected 3) Therefore, developers need to be restricted from prod One of my teams went through the process of "let's do DevOps!" with the intent of giving developers the ability of pushing something all the way through to prod on AWS. Months later, this resulted in having a poorly-suppor…

There's a common systematic belief of that because that's the sort of thing a lot of actual compliance regulations de facto require (i.e., they demand controls around software deploys, and putting enforcing that in the same hands as those wanting to deploy it, i.e., devs, will fail an audit).

Source: My employer is currently undergoing SOX compliance

Re: Blending complex systems made my latency 10x higher

#65
post #5
post #2

> Kubernetes made my latency 10x higher The title is a bit misleading, kub didn't cause the 10x latency - also latency was lower after they fixed their issues TL;DR version - Migrate from EC2 to Kub; due to some default settings in Kiam & AWS Java SDK, latency of application increased, fixed after reconfiguration and kub latency lower than EC2

It is relevant though as k8s makes everything more complicated so you have to deal with stuff like this. Also if it was a brand new app theyd maybe not notice the problem in the first place.

> k8s makes everything more complicated

No it doesn't.

This is only true if you don't need the features in the first place, and you haven't gone through the learning curve yet.

So I can deploy a single container. Works fine. You can tell docker to restart it. Works fine. If that's all you need, you don't need k8s.

But you may now want to deploy two containers. And let's say they are servers. Now, you need incoming connections from a single port going to them. So maybe you are going to deploy something like haproxy – which now you have to learn.

Also, you now need a way to tell if your service is down. So you add a health check. But then you want to kill the container if it goes bad. So now you add a cronjob or similar.

At some point, one machine is not enough. So do you replicate the exact workloads across machines? If you do, it is a bit easier - but potentially more expensive. But if you need to spread them across different machine types, it gets more difficult. And now you also have storage distributed which you need to keep track of. If you haven't automated your deployment, now it is the time.

Now you have this nice setup. It is time to upgrade. How do you do that? Probably some custom script again.

Instead, you can forget all of the above. You can spin up a k8s cluster with any of the currently available tools (or use one of the cloud providers). Once you have k8s, a lot is taken care for you. Multiple copies of the service (with the networking taken care for you) ? Check. Want to increase? kubectl scale deploy --replicas=. Want to upgrade? Apply the new YAML. Health checks and auto-restart? Add a couple of lines to your YAML. Want to have different workloads over different machine types? That's also an easy change. Want to have storage (that follows your workload around)? Easy. You are in a cloud provider like GCP and want a new loadbalancer (with automated SSL cert provisioning!)? A couple of lines of YAML. Want granular access controls? It's built in. I can go on.

Of course, there's a learning curve. But the learning curve is also there if you stitching together a bunch of solutions to replicate the same features. Once you get used to it, it's difficult to go back.

Re: Blending complex systems made my latency 10x higher

#66
post #13

Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.

> this isn't specific to k8s at all. I don't know. k8s is pretty complicated. How many small/medium apps need more than this nginx/Terraform/Docker example? This would be a lot more difficult to set up in k8s (pods, ingress, etc.) Terraform: resource "docker_container" "app" { count = 2 name = "app-${count.index}" hostname = "app-${count.index}" image = "app:${var.app_version}" restart = "always" env = [ ... ] ports…

I don't know understand the point of this comment.

Is there some epidemic of developers setting up complex, multi-node Kubernetes clusters just to run a small web app ?

Re: Blending complex systems made my latency 10x higher

#67
post #56
post #23

The problem wasn't in Kubernetes at all! The problem was in KIAM and the AWS Java SDK. It would be more accurate to criticize AWS's Kubernetes support. Both KIAM and the AWS Java SDK are specific to AWS.

We found EKS to be really disappointing in comparison to Self hosted solution. Not only you simply cannot tweak extremely important kubelet configuration, you also cannot run real HA. Most of AWS implementation around EKS was simply terrible and outclassed by community driven projects. For me personally EKS is the same failed service as Elasticsearch Service. Good for low to medium size workloads but terrible for any…

Why can't you run "real HA" with EKS ?

It is after all running the control plane in multiple AZs.

Re: Blending complex systems made my latency 10x higher

#68
post #51
post #13

Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.

Yes it’s a total linkbait title. I guess it worked because it’s on the front page.

A better way to complain about a title is to suggest a better one—i.e. an accurate and neutral alternative—so we can replace it.

Re: Blending complex systems made my latency 10x higher

#69
post #37

Earlier quoted context omitted.

It wasn’t. It was KIAM and the Java SDK, neither of which are part of K8S.

KIAM is not part of the core Kubernetes system but it was a necessary component to avoid introducing a security regression as part of the switch. Again, my point was that rather than trying to do PR damage-control it would be better to work to improve things so this doesn't understand. Someone went to the trouble of posting a detailed examination of a real problem with a fix and some references to upstream improvemen…

There are many alternatives to kiam within the K8S space.

So would disagree that is specifically is a necessary component.

Re: Blending complex systems made my latency 10x higher

#70
post #13

Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.

Ok, we've replaced "Kubernetes" with a more accurate and representative phrase from the article. If someone suggests a better title, we can change it again.
Post reply on HN