Envoy Proxy Performance on Kubernetes
getambassador.io
Envoy Proxy Performance on Kubernetes
1–10 of 16 posts
Re: Envoy Proxy Performance on Kubernetes
#2The article focus on latency spikes, which happen only sporadically. Can anyone more knowledgeable on the subject chime in and comment if this comparison is fair or relevant? Personally I was expecting to see histograms or empirical distribution functions of each service's latency.
Re: Envoy Proxy Performance on Kubernetes
#3The article focus on latency spikes, which happen only sporadically. Can anyone more knowledgeable on the subject chime in and comment if this comparison is fair or relevant? Personally I was expecting to see histograms or empirical distribution functions of each service's latency.
Personally, I prefer looking at something more like 95th percentile latency, versus average, which is what I think this article is showing. I suppose a histogram would give you the fullest picture, though.
Re: Envoy Proxy Performance on Kubernetes
#4This is somewhat valuable, but misses any investigation as to why there were outlying latency spikes when using non-Envoy software for load balancing. Furthermore, using the average latency like this doesn't tell us much, especially with outliers making the graphs worthless for steady-state performance analysis.
My first thought is that the spikes are somewhat clearly the result of requests getting sent to pods that no longer exist, or are starting and not prepared to process requests. This might just speak to the method of configuration for all three of these underlying softwares and say absolutely nothing about how well they actually fare doing any load balancing.
If someone came to me with this at work, I would say it is the beginning of a series of troubleshooting steps to answer the question of why there are such outlying requests when using our load balancer of choice, and not an analysis of which software to pick.
Edit: Even worse is that this appears to be from a company that sells.. an API gateway built on top of Envoy.
Re: Envoy Proxy Performance on Kubernetes
#5The article focus on latency spikes, which happen only sporadically. Can anyone more knowledgeable on the subject chime in and comment if this comparison is fair or relevant? Personally I was expecting to see histograms or empirical distribution functions of each service's latency.
Personally, I prefer looking at something more like 95th percentile latency, versus average, which is what I think this article is showing. I suppose a histogram would give you the fullest picture, though.
FTA:
> We measure latency for 10% of the requests, and plot each of these latencies individually on the graphs.
So for what it's worth these spikes may very well be single requests that are not relevant and are only triggered by the way the Kubernetes cluster was being manipulated for the test.
Re: Envoy Proxy Performance on Kubernetes
#6This is somewhat valuable, but misses any investigation as to why there were outlying latency spikes when using non-Envoy software for load balancing. Furthermore, using the average latency like this doesn't tell us much, especially with outliers making the graphs worthless for steady-state performance analysis. My first thought is that the spikes are somewhat clearly the result of requests getting sent to pods that…
[deleted]
Re: Envoy Proxy Performance on Kubernetes
#7traefik?
Re: Envoy Proxy Performance on Kubernetes
#8Earlier quoted context omitted.
Personally, I prefer looking at something more like 95th percentile latency, versus average, which is what I think this article is showing. I suppose a histogram would give you the fullest picture, though.
FTA: > We measure latency for 10% of the requests, and plot each of these latencies individually on the graphs. So for what it's worth these spikes may very well be single requests that are not relevant and are only triggered by the way the Kubernetes cluster was being manipulated for the test.
The spikes aren't single requests (at 1000 RPS, the spikes are well over a second long and you see hundreds of requests that spike). As for the reason, we suspect that the different config reload mechanisms in the respective proxies is what triggers the spikes.
(disclaimer: one of the authors)
Re: Envoy Proxy Performance on Kubernetes
#9This is somewhat valuable, but misses any investigation as to why there were outlying latency spikes when using non-Envoy software for load balancing. Furthermore, using the average latency like this doesn't tell us much, especially with outliers making the graphs worthless for steady-state performance analysis. My first thought is that the spikes are somewhat clearly the result of requests getting sent to pods that…
(one of the authors here)
Thanks for the feedback.
So regarding your hypothesis on the spikes being sent to pods that no longer exist/are starting: 1) it is the responsibility of the ingress controller on K8S to properly handle that situation 2) it would be highly unlikely for people to implement their own custom ingress controller around a given proxy (it's actually somewhat complicated) and 3) the pod theory wouldn't address the latency spikes seen on reconfiguration.
But you're right that there probably should be some explanation around why we think this is happening (I just didn't want to speculate too much; I suspect that the issue is with the hitless reloads implementation in the proxies which is tricky to do well).
Re: Envoy Proxy Performance on Kubernetes
#10This is somewhat valuable, but misses any investigation as to why there were outlying latency spikes when using non-Envoy software for load balancing. Furthermore, using the average latency like this doesn't tell us much, especially with outliers making the graphs worthless for steady-state performance analysis. My first thought is that the spikes are somewhat clearly the result of requests getting sent to pods that…
(one of the authors here) Thanks for the feedback. So regarding your hypothesis on the spikes being sent to pods that no longer exist/are starting: 1) it is the responsibility of the ingress controller on K8S to properly handle that situation 2) it would be highly unlikely for people to implement their own custom ingress controller around a given proxy (it's actually somewhat complicated) and 3) the pod theory wouldn…
Could it be at all related to the circuit-breaking behavior that nginx describes[1] in some of their reference architecture? Unclear to me which (if any) of these properties might be in play for this test.
[1]https://www.nginx.com/blog/microservices-reference-architect...