Blending complex systems made my latency 10x higher • Hacker News Board

Re: Blending complex systems made my latency 10x higher

#81

Post by lostcolony » Sat, Nov 09, 2019, 2:16 AM UTC

beat wrote: post #73

lostcolony wrote: post #64

Earlier quoted context omitted.

There's a common systematic belief of that because that's the sort of thing a lot of actual compliance regulations de facto require (i.e., they demand controls around software deploys, and putting enforcing that in the same hands as those wanting to deploy it, i.e., devs, will fail an audit). Source: My employer is currently undergoing SOX compliance

And there's a good reason that separation of duties is in every compliance standard...

Not saying whether it's a good or bad idea, just that it's a common systematic belief because it's a required thing in many organizations.

Re: Blending complex systems made my latency 10x higher

#82

Post by auggierose » Sat, Nov 09, 2019, 4:09 AM UTC

nvarsj wrote: post #50

auggierose wrote: post #30

Earlier quoted context omitted.

In my opinion DevOps shouldn't exist at all.

Can you elaborate? You may as well say “software engineering should not exist at all”.

No, I have just been throwing that statement out there. Pleased by all the comments it generated, was an interesting read :)

Re: Blending complex systems made my latency 10x higher

#83

Post by srvaroa » Sat, Nov 09, 2019, 9:26 AM UTC

jaaron wrote: post #72

bavell wrote: post #13

Very misleading title, was hoping for a more substantive read. Kubernetes itself wasn't causing latency issues, it was some config in their auth service and AWS environment. In the takeaways section, the author blames the issue on merging together complicated software systems. While absolutely true, this isn't specific to k8s at all. To specifically call out k8s as the reason for latency spiking is misleading.

I really like the title and I don't think it's link-bait-y. Why? Because too many engineers I've worked with would bump into this situation and this would be their answer. They wouldn't take the time to debug the situation deeply enough and they'd blame k8s, or blame the network, or blame... In my experience, the most common issues with complex distributed systems are much more likely to be due to misconfiguration be…

Hi, author here. That was exactly the intent of the title, reflecting the reaction we (almost always) get from developers: "k8s is at fault", the result of most investigations is "not really". I try to make that evident at the conclusions, but I agree without realizing that intent the title is misleading.

Re: Blending complex systems made my latency 10x higher

#84

Post by pojzon » Sat, Nov 09, 2019, 9:37 AM UTC

nielsole wrote: post #75

threeseed wrote: post #67

Earlier quoted context omitted.

Why can't you run "real HA" with EKS ? It is after all running the control plane in multiple AZs.

Also why can't you modify kubelet params? You are completely in charge of the nodes and can configure them freely.

There are some cluster parameters you simply cannot change because api refuses or are not available (node cluster dependant parameters also). Example can be HPA downscale grace periods..

Re: Blending complex systems made my latency 10x higher

#85

Post by pojzon » Sat, Nov 09, 2019, 9:40 AM UTC

threeseed wrote: post #67

pojzon wrote: post #56

Earlier quoted context omitted.

We found EKS to be really disappointing in comparison to Self hosted solution. Not only you simply cannot tweak extremely important kubelet configuration, you also cannot run real HA. Most of AWS implementation around EKS was simply terrible and outclassed by community driven projects. For me personally EKS is the same failed service as Elasticsearch Service. Good for low to medium size workloads but terrible for any…

Why can't you run "real HA" with EKS ? It is after all running the control plane in multiple AZs.

Please check how aws recovers from lost master node. Its pretty straightforward from there.

Re: Blending complex systems made my latency 10x higher

#86

Post by smarterclayton » Sat, Nov 16, 2019, 5:26 PM UTC

kodablah wrote: post #74

smarterclayton wrote: post #48

Earlier quoted context omitted.

Yeah, Tim Hockin and I still regret not designing the DNS name search process in Kube better. If we had, we would have avoided the need for 5 and could have kept 90% of the usability win of “name.namespace.svc” and “name” being resolvable to services without having to go to 5. And now we can’t change it by default without breaking existing apps. Forwards compatibility is painful.

Pardon my lack of Kubernetes knowledge, but any regrets supporting the hierarchical lookup where they don't have to qualify their dns requests (and maybe could have used some other way to find their "same namespace")?

Good question. I certainly use the “name” and “name.namespace.svc” forms extensively for both “apps in a single namespace” and “apps generic to a cluster”.

I know a small percentage of clusters makes their service networks public with dns resolution (so a.b.svc.cluster-a.myco is reachable from most places).

The “namespace” auto-injected file was created long after this was settled, so that wasn’t an option. I believe most of the input was “the auto env var injection that docker did we don’t like, so let’s just keep it simple and use DNS the way it was quasi intended”.

Certainly we intended many of the things istio does to be part of Kube natively (like cert injection, auto proxying of requests). We just wound up having too much else to do.