Announcing Envoy: C++ L7 proxy and communication bus

Disclaimer: I work on Proxygen at FB.

#1: the cost of local proxying is very small, on the order of tenths of ms. it's going to be hard to distinguish against the backdrop of the broader network latency. also, in our environment we've been thinking of only running the proxy on the client end for now, to make the migration easier.

#2: as with haproxy, you leave the existing connections handled with the running proxy, and new connections are taken up by the new proxy. unlike haproxy, envoy actually supports live config reloading, so you don't have to pay that penalty on every config change

#3: multiple connections are multiplexed, but i'm not sure how you specify priority to the proxy. do the docs specify a mechanism? i imagine both requests are served in parallel, in no particular order, and the data is buffered for the client.

I work at Lyft. To answer your questions:

1) There is added cost, though it varies depending on how many things Envoy is configured to do (e.g., logging, tracing, stats, rate limiting, health checking, etc.). Even in complex scenarios (Envoy being used to proxy both inbound connections into a service, as well as proxy outbound connections to Mongo or Dynamo), we measure Envoy overhead to be 2) Envoy supports hot restart (https://lyft.github.io/envoy/docs/intro/arch_overview/hot_re...), as well as graceful drain of existing connections, so there is limited/no impact to existing clients. There is one enhancement that we would like to make to our HTTP/2 graceful draining to make it even more seamless, but that is more complicated than I can type here. :)

3) Right now Envoy does not support HTTP/2 priority so all streams are treated equally. We are currently working on priority support at the routing layer, with different connection pools available for high and low priority traffic, as well as circuit breaking settings. In the future we will likely merge this back into a single HTTP/2 connection with proper priority support. In practice though, within the DC, head of line blocking at the TCP layer isn't too much of an issue.

The one thing I must have is SNI. The docs only have a short blurb [1], does someone know the full status of SNI support?

[1]: https://lyft.github.io/envoy/docs/intro/arch_overview/ssl.ht...

EDIT: Also is there any kind of visualization for the resulting network topology? It looks like Envoy should know everything about who is talking to who.