Nginx-ingress (the free version) does a reload when pods are added/removed. I would expect to see latency spikes. As I understand it the paid nginx-ingress can do these reconfigs on-the-fly. If you have pods adding/removing frequently, like maybe from "We then scale up the backend service to four pods and then scale it back down to three pods every thirty seconds", this is what you see. The more interesting question is if that is a realistic expectation for your workload. Unless you are doing aggressive auto-scaling pods can be very stable, only add/removing during upgrades.
Envoy Proxy Performance on Kubernetes
11–16 of 16 posts
Re: Envoy Proxy Performance on Kubernetes
#12Earlier quoted context omitted.
(one of the authors here) Thanks for the feedback. So regarding your hypothesis on the spikes being sent to pods that no longer exist/are starting: 1) it is the responsibility of the ingress controller on K8S to properly handle that situation 2) it would be highly unlikely for people to implement their own custom ingress controller around a given proxy (it's actually somewhat complicated) and 3) the pod theory wouldn…
Could it be at all related to the circuit-breaking behavior that nginx describes[1] in some of their reference architecture? Unclear to me which (if any) of these properties might be in play for this test. [1] https://www.nginx.com/blog/microservices-reference-architect...
I suspect it's because of reloads:
https://kubernetes.github.io/ingress-nginx/how-it-works/#whe...
The NGINX ingress controller goes to some lengths to avoid reloads because it recognizes the hit from reloads. In Ambassador-land, we use Envoy's xDS APIs to avoid this problem. Not clear what the HAProxy ingress controller does.
Re: Envoy Proxy Performance on Kubernetes
#13Nginx-ingress (the free version) does a reload when pods are added/removed. I would expect to see latency spikes. As I understand it the paid nginx-ingress can do these reconfigs on-the-fly. If you have pods adding/removing frequently, like maybe from "We then scale up the backend service to four pods and then scale it back down to three pods every thirty seconds", this is what you see. The more interesting question…
This is a good point. I think an interesting test would be to add multiple services, and see if scaling up/down a single service affects latency to other services.
Re: Envoy Proxy Performance on Kubernetes
#14traefik?
I am wondering as well how that compares to this.
Re: Envoy Proxy Performance on Kubernetes
#15Earlier quoted context omitted.
Could it be at all related to the circuit-breaking behavior that nginx describes[1] in some of their reference architecture? Unclear to me which (if any) of these properties might be in play for this test. [1] https://www.nginx.com/blog/microservices-reference-architect...
I suspect it's because of reloads: https://kubernetes.github.io/ingress-nginx/how-it-works/#whe... The NGINX ingress controller goes to some lengths to avoid reloads because it recognizes the hit from reloads. In Ambassador-land, we use Envoy's xDS APIs to avoid this problem. Not clear what the HAProxy ingress controller does.
The official HAProxy Ingress Controller uses the Runtime API [1] to avoid restarts/reloads and also has hitless reloads configured by default. HAProxy Technologies also contributed the capability to use the Runtime API within the jcmorais-haproxy ingress controller as well but it requires you to activate using the dynamic-scaling option [3]
One thing I wanted to point out is that the HAProxy Ingress Controller actually has over 25 [2] configuration options at the time of publishing, not 8 as mentioned.
While we have identified a few on our own we'd love to work with you further to identify any missing configuration directives that can help perform some more accurate benchmarks using the official HAProxy Ingress Controller.
disclosure: I work at HAProxy Technologies
[1] https://www.haproxy.com/blog/dynamic-configuration-haproxy-r...
[2] https://github.com/haproxytech/kubernetes-ingress/tree/maste...
[3] https://github.com/jcmoraisjr/haproxy-ingress#dynamic-scalin...
Re: Envoy Proxy Performance on Kubernetes
#16Earlier quoted context omitted.
I suspect it's because of reloads: https://kubernetes.github.io/ingress-nginx/how-it-works/#whe... The NGINX ingress controller goes to some lengths to avoid reloads because it recognizes the hit from reloads. In Ambassador-land, we use Envoy's xDS APIs to avoid this problem. Not clear what the HAProxy ingress controller does.
The official HAProxy Ingress Controller uses the Runtime API [1] to avoid restarts/reloads and also has hitless reloads configured by default. HAProxy Technologies also contributed the capability to use the Runtime API within the jcmorais-haproxy ingress controller as well but it requires you to activate using the dynamic-scaling option [3] One thing I wanted to point out is that the HAProxy Ingress Controller actual…
Thanks! Drop me a line (email in my profile) and would love to chat.
I updated the article to clarify that there were 8 configuration options at the time of testing (we started this effort awhile ago) and now there are 25.
We'd definitely like to rerun the tests with the official controller to use the Runtime API.