Live data from Hacker News

Blending complex systems made my latency 10x higher

srvaroa.github.io

21–30 of 86 posts

Re: Blending complex systems made my latency 10x higher

#21
This issue has been hit by quite a number of people in the last year [1]

AWS have recently added the feature natively [2], so once that version of the SDK is in use by all your pods you won't need Kiam any more.

[1] https://github.com/uswitch/kiam/issues/191 [2] https://aws.amazon.com/blogs/opensource/introducing-fine-gra...

Re: Blending complex systems made my latency 10x higher

#22
post #16

"Once this change was applied, requests started being served without involving the AWS Metadata service and returned to an even lower latency than in EC2." Title should be: My configuration made my latency 10x higher.

We solved our configuration problems, and most rewardingly, we managed better performance than the original EC2 baseline.

Re: Blending complex systems made my latency 10x higher

#25
post #20
post #18

Earlier quoted context omitted.

Specific to k8s would mean it only affects k8s, which is false. It affects a large group of systems, of which k8s is one. Better (more accurate) title would have been: "Merging complicated software systems made my latency 10x higher"

But with that title the author wouldn't have gotten those juicy clicks

So true hahaha

Re: Blending complex systems made my latency 10x higher

#26
post #13

Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.

Well, it was a service they were running to deal with differences between EC2 and containers in terms of AWS auth, so it is relevant.

Re: Blending complex systems made my latency 10x higher

#27
post #5
post #2

> Kubernetes made my latency 10x higher The title is a bit misleading, kub didn't cause the 10x latency - also latency was lower after they fixed their issues TL;DR version - Migrate from EC2 to Kub; due to some default settings in Kiam & AWS Java SDK, latency of application increased, fixed after reconfiguration and kub latency lower than EC2

It is relevant though as k8s makes everything more complicated so you have to deal with stuff like this. Also if it was a brand new app theyd maybe not notice the problem in the first place.

I don’t think the interactions between the AWS SDK and a third-party AWS IAM plug-in is really an inherent part of k8s’s domain of control.

It’s sort of like cursing Microsoft every time an application crashes... because it’s running on windows.

Re: Blending complex systems made my latency 10x higher

#28
What grinds my gears is hard-coded magic timeout numbers. Somehow microservice people seem to think that these are good (e.g. for circuit breakers) without seeming to realize the unexpected consequences of composing like this. Your timeout is not my timeout. So firstly - don't do it. Time is an awful thing to build behaviour on, and secondly if you ignore that then make it a config parameter - then I've got a chance of finding it without wire level debugging (if you document it).
Post reply on HN