Earlier quoted context omitted.
KIAM is not part of the core Kubernetes system but it was a necessary component to avoid introducing a security regression as part of the switch. Again, my point was that rather than trying to do PR damage-control it would be better to work to improve things so this doesn't understand. Someone went to the trouble of posting a detailed examination of a real problem with a fix and some references to upstream improvemen…
It’s not PR. It’s being accurate that this has nothing to do with Kubernetes.
Blending complex systems made my latency 10x higher
61–70 of 86 posts
Re: Blending complex systems made my latency 10x higher
#62"Once this change was applied, requests started being served without involving the AWS Metadata service and returned to an even lower latency than in EC2." Title should be: My configuration made my latency 10x higher.
Re: Blending complex systems made my latency 10x higher
#63Earlier quoted context omitted.
In my opinion DevOps shouldn't exist at all.
I think the best model is to have your "devops/sre guy" embedded right on the team. I market myself in interviews as a software engineer (on the same team as the other SWEs) that just happens to focus on what actually getting this product into production and keeping it going looks like. I got my start in this by working at a startup where we were responsible for everything just by the nature of the limited staff. I t…
And here lies the problem.
We had "Ops" before. People would throw stuff over the wall from Dev to Ops, and now it became "Ops" problem. Operations would stonewall releases – because releases bring problems, and they want to keep things stable. Developers would push for releases, because that's what they are paid to do.
To solve this conflict, someone came with the idea of "DevOps". You don't have a split organization now, you have some developers in your team that can wear the "Ops" hat. That way, they experience production issues first hand, and that allows them to architect the application better – as well as ensuring it is thoroughly tested – noone wants to wake up at 2AM.
And, because they are developers at heart, they will do things to make their job easier and more predictable, like infrastructure as code. And heavy automation.
Somewhere along the way, people misread the whole idea, and now think that "devops" == Terraform or some other tool. Then they rebranded their Ops org as DevOps and hired a few automation folks and called it a day. Ops rebranded is still ops.
It was not supposed to be like that. If you have a "devops" team, you are not doing it correctly. Call it operations like it should be called. You may even have a dedicated team working on common tools or similar – which is fine – but you still need developers doing operations and ultimately responsible for their own shit, otherwise this is not devops.
> I got my start in this by working at a startup where we were responsible for everything just by the nature of the limited staff. I think that's how a lot of people in Devops get their start.
Yup. I got into that by trying (and ultimately failing) to launch my own startup. We didn't have dedicated operations folks, or time to manually tinker with servers, or to respond to repeated production issues. When you are sales, operations and engineering, you try your hardest to not have incidents (so that you don't lose customers), but you are trying to move fast to deploy features (so that you can sell).
Re: Blending complex systems made my latency 10x higher
#64Earlier quoted context omitted.
Yeah, operations-focused engineers will continue to have a niche carved out for them because too many devs black-box infrastructure. Companies can either choose to have their devs take on ops responsibilities or continue having dedicated ops jobs. In either case, whether or not dedicated ops jobs exist, ops responsibilities always will. I'll be there to pick up the slack because designing and maintaining systems is a…
People overlook that there's a common systematic belief that looks like this: 1) Developers don't code on prod 2) Prod needs to be protected 3) Therefore, developers need to be restricted from prod One of my teams went through the process of "let's do DevOps!" with the intent of giving developers the ability of pushing something all the way through to prod on AWS. Months later, this resulted in having a poorly-suppor…
Source: My employer is currently undergoing SOX compliance
Re: Blending complex systems made my latency 10x higher
#65> Kubernetes made my latency 10x higher The title is a bit misleading, kub didn't cause the 10x latency - also latency was lower after they fixed their issues TL;DR version - Migrate from EC2 to Kub; due to some default settings in Kiam & AWS Java SDK, latency of application increased, fixed after reconfiguration and kub latency lower than EC2
It is relevant though as k8s makes everything more complicated so you have to deal with stuff like this. Also if it was a brand new app theyd maybe not notice the problem in the first place.
No it doesn't.
This is only true if you don't need the features in the first place, and you haven't gone through the learning curve yet.
So I can deploy a single container. Works fine. You can tell docker to restart it. Works fine. If that's all you need, you don't need k8s.
But you may now want to deploy two containers. And let's say they are servers. Now, you need incoming connections from a single port going to them. So maybe you are going to deploy something like haproxy – which now you have to learn.
Also, you now need a way to tell if your service is down. So you add a health check. But then you want to kill the container if it goes bad. So now you add a cronjob or similar.
At some point, one machine is not enough. So do you replicate the exact workloads across machines? If you do, it is a bit easier - but potentially more expensive. But if you need to spread them across different machine types, it gets more difficult. And now you also have storage distributed which you need to keep track of. If you haven't automated your deployment, now it is the time.
Now you have this nice setup. It is time to upgrade. How do you do that? Probably some custom script again.
Instead, you can forget all of the above. You can spin up a k8s cluster with any of the currently available tools (or use one of the cloud providers). Once you have k8s, a lot is taken care for you. Multiple copies of the service (with the networking taken care for you) ? Check. Want to increase? kubectl scale deploy --replicas=. Want to upgrade? Apply the new YAML. Health checks and auto-restart? Add a couple of lines to your YAML. Want to have different workloads over different machine types? That's also an easy change. Want to have storage (that follows your workload around)? Easy. You are in a cloud provider like GCP and want a new loadbalancer (with automated SSL cert provisioning!)? A couple of lines of YAML. Want granular access controls? It's built in. I can go on.
Of course, there's a learning curve. But the learning curve is also there if you stitching together a bunch of solutions to replicate the same features. Once you get used to it, it's difficult to go back.
Re: Blending complex systems made my latency 10x higher
#66Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.
> this isn't specific to k8s at all. I don't know. k8s is pretty complicated. How many small/medium apps need more than this nginx/Terraform/Docker example? This would be a lot more difficult to set up in k8s (pods, ingress, etc.) Terraform: resource "docker_container" "app" { count = 2 name = "app-${count.index}" hostname = "app-${count.index}" image = "app:${var.app_version}" restart = "always" env = [ ... ] ports…
Is there some epidemic of developers setting up complex, multi-node Kubernetes clusters just to run a small web app ?
Re: Blending complex systems made my latency 10x higher
#67The problem wasn't in Kubernetes at all! The problem was in KIAM and the AWS Java SDK. It would be more accurate to criticize AWS's Kubernetes support. Both KIAM and the AWS Java SDK are specific to AWS.
We found EKS to be really disappointing in comparison to Self hosted solution. Not only you simply cannot tweak extremely important kubelet configuration, you also cannot run real HA. Most of AWS implementation around EKS was simply terrible and outclassed by community driven projects. For me personally EKS is the same failed service as Elasticsearch Service. Good for low to medium size workloads but terrible for any…
It is after all running the control plane in multiple AZs.
Re: Blending complex systems made my latency 10x higher
#68Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.
Yes it’s a total linkbait title. I guess it worked because it’s on the front page.
Re: Blending complex systems made my latency 10x higher
#69Earlier quoted context omitted.
It wasn’t. It was KIAM and the Java SDK, neither of which are part of K8S.
KIAM is not part of the core Kubernetes system but it was a necessary component to avoid introducing a security regression as part of the switch. Again, my point was that rather than trying to do PR damage-control it would be better to work to improve things so this doesn't understand. Someone went to the trouble of posting a detailed examination of a real problem with a fix and some references to upstream improvemen…
So would disagree that is specifically is a necessary component.
Re: Blending complex systems made my latency 10x higher
#70Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.