A terrible, horrible, no-good, very bad day at Slack

I've found that the cloud load balancers lag behind the state of the art in features and that their assumptions and configurations can be pretty brittle.

I haven't used Amazon's ALB, but with the legacy ELB, they can't speak ALPN. So that means, if you use their load balancer to terminate TLS, you can't use HTTP/2. Their automatic certificate renewal silently broke for us as well; whereas using cert-manager to renew Let's Encrypt certificates continues to work perfectly wherever I use it. (At the very least, cert-manager produces a lot of logs, and Envoy produces a lot of metrics. So at the very least, when it does break, you know what to fix. With the ELB, we had to pray to the Amazon gods that someone would fix our stuff on Saturday morning when we noticed the breakage. They did! But I don't like the dependency.)

I have also used Amazon's Network Load Balancer with EKS. It interacts very weirdly. The IP address that the load balancer takes on changes with the pods that back the service. The way the changes happen is that the NLB updates a DNS record with a 5 minute TTL. So you have a worst case rollout latency of 5 minutes, and there is no mechanism in Kubernetes to keep the old pods alive until all cached DNS records have expired. The result is, by default, 5 minutes of downtime every time you update a deployment. Less than ideal! For that reason, I stuck with ELB pointing to Envoy that terminated TLS and did all complicated routing.

The ALB wouldn't have these problems. It's just a HTTP/2 server that you can configure using their proprietary and untestable language. It has some weak integration with the Kubernetes Ingress type, so in the simplest of simple cases you can avoid their configuration and use a generic thing. But Ingress misses a lot of things that you want to do with HTTP, so in my opinion it causes more problems than it solves. (The integration is weak too. You can serve your domain on Route 53, but if you add an Ingress rule for "foo.example.com", it's not going to create the DNS record for you. It's very minimum-viable-product. You will be writing custom code on top of it, or be doing a lot of manual work. All in all, going to scale to a large organization poorly unless you write a tool to manage it, in which case you might as well write a tool to configure Envoy or whatever.)

In general, I am exceedingly disappointed by Layer 3 load balancers. For someone that only serves HTTPS, it is completely pointless. You should be able to tell browsers, via DNS, where all of your backends are and what algorithm they should use to select one (if 503, try another one, if connect fails, try another one, etc.) But... browsers can't do that, so you have to pretend that you only have one IP address and make that IP address highly available. Google does very well with their Maglev-based VIPs. Amazon is much less impressive, with one IP address per AZ and a hope and a prayer that the browser does the right thing when one AZ blows up. Since AZs rarely blow up, you'll never really know what happens when it does. (Chrome handles it OK.)

Lock-in is one of the worst reasons to avoid serverless. If you're really putting in the effort to avoid lock-in, you're wasting engineering time that could be better spent on your product. AWS (or GCP, or whatever cloud provider you choose) has a number of really great products that ultimately save you time and help you get to market faster and better.

It doesn't matter that you're locked-in, with slightly higher than wanted costs when you've failed due to poor priorities.

Become successful, then worry about removing your lock-in if you actually need it (and you probably won't).

One of the reasons Envoy was built was because ELB/ALBs have opaque observability and fail in ways you can't control.

I had trouble getting through this article because my internal monologue was screaming "Envoy and xDS wouldn't have this problem". But that's exactly what they decided ;) HAProxy is a little behind the state of the art on "hey I could just ask some server where the backends are", and it shows in this case. (The "slots" are particularly alarming, as is having to restart when backends come and go.)

xDS lets you give your frontend proxy a complete view of your whole system -- where the other proxies are (region/AZ/machine) and where the backends are, and how many of those. It can then make very good load balancing decisions -- preferring backends in the AZ that the frontend proxy is in, but intelligently spilling over if some other AZ is missing a frontend proxy or has fewer backends. And it emits metrics for each load balancing decision, so you can detect problems or unexpected balancing decisions before it results in an outage. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...

I also like the other features that Envoy has -- it can start distributed traces, it gives every request a unique ID so you can correlate applications and frontend proxy logs, it has a ton of counters/metrics for everything it does, and it can pick apart HTTP to balance requests (rather than TCP connections) between backends. It can also retry failed requests, so that users don't see transient errors (especially during rollouts). And it's retry logic is smart, so that if your requests are failing because a shared backend is down (i.e. your database blew up), it breaks the circuit for a period of time and lets your app potentially recover.

The result is a good experience for end users sending you traffic, and extreme visibility into every incoming request. Mysterious problems become easy to debug just by looking at a dashboard, or perhaps by clicking into your tracing UI in the worst case.

The disadvantage is that it doesn't really support any service discovery other than DNS out of the box. I had to write github.com/jrockway/ekglue to use Kubernetes service discovery to map services to Envoy's "clusters" (upstreams/backends), but I'm glad I did because it works beautifully. Envoy can take advantage of everything that Kubernetes knows about the service, which results in less config to write and a more robust application. (For example, it knows about backends that Kubernetes considers unready -- if all your backends are unready, Envoy will "panic" and try sending them traffic anyway. This can result in less downtime if your readiness check is broken or there's a lot of churn during a big rollout.)

You think you can run something like Slack without intimate knowledge of all the moving parts?

The only thing that 24/7 crack devops team ensures in a situation like this is that you continue to generate billable workload, even if it's just millions of little serverless pods spinning in a circle from some configuration mishap.