Knative = Kubernetes Networking++

Knative = Kubernetes Networking++
ahmet.im

Can we stop stacking shit?

Think of it as an open source lambda that runs on top of Kubernetes.

I like Google's strategy here: Cloud run is (essentially) hosted Knative for those that want to pay someone else to manage it.

For DIY, you can run it on your own cluster.

By default, a Kubernetes "service" exists to create a single IP address for a group of replicas. You see that your "foo service" is at 10.2.3.4, you open a TCP connection to it, your connection is created to one of the available replicas. You open another connection, it goes to another replica. This accomplishes some load balancing and lets new TCP connections avoid unhealthy replicas.

The problem is, a TCP connection per request is relatively uncommon these days. You open up one connection to MySQL and send it multiple queries. gRPC sends multiple requests and responses with one TCP connection. HTTP/2 sends multiple requests with one TCP connection. The days of one request = one TCP connection are over.

So there is a lot of work being done to try to make load balancing transparent at a level above the TCP connection. Your app asks to connect to foo-service, and each request is routed to another healthy replica. Replicas come and go, but your app never has to handle reconnects. You want to send 10% of requests to a canary instance without having to tell your application that; you want the networking layer to handle that for you.

There are several approaches to handling this problem.

The first is having smart clients; if your gRPC library knows how to ask the Kubernetes API for a list of replicas, it can do its own load balancing and health checks. It gets a list of valid endpoints, opens up a connection, subscribes to the streaming health service that each backend provides, decides on a load-balancing algorithm, and goes! This works quite well, but it is difficult to setup and difficult to maintain. The C++ gRPC client has to do this, the C++ HTTP/2 client has to do this, the Go gRPC client has to do this, the Go HTTP/2 client has to do this, the Python gRPC client has to do this, the Python HTTP/2 client has to do this, the node.js gRPC client has to do this... you get the picture.

I am most familiar with this approach from my days at Google, where load balancing typically consisted of a smart client for every language that connected to a coordinator, asked for backends and a rule for splitting traffic among them, and stayed connected to receive updates. It worked extremely well. You could drain a cluster very quickly, and obviously traffic kept flowing if the load balancer died. (Not sure I ever saw the load balancer die, though.) The disadvantage is that a malfunctioning smart client can break everything; every piece of code you deploy to production in every language has to work perfectly. (From reading job ads, I feel like few companies are as disciplined as Google at saying "you have to use one of these 3 supported programming languages". So in the real world, the "smart client" approach will never work. You can check out the existing gRPC client libraries to see this in action; best case, there is round-robin load balancing based on DNS records, with some metadata that can also be retrieved from DNS. C++ supports it best, Go is next, the rest of the language bindings... questionable. Raw HTTP/2? Good luck.)

The second approach is to have a smart proxy that your apps use to get to the outside world. You run this smart proxy as a "sidecar", receive all traffic through it, and send all outgoing traffic through it. It knows where all the backends are and how to load balance, so the app doesn't have to. Personally, I think this is the most practical approach right now. Run Envoy next to your apps, make your services "headless" (so Kubernetes just returns all endpoints over DNS, rather than trying to do TCP-level load balancing), and go! Even your outdated PHP app will now have detailed metrics and participate in distributed tracing. The disadvantage is that you have to manually add an Envoy container and minimal config to everything you run, which people apparently hate.

I like this approach, it's what I personally do. Envoy has its quirks, but it does provide a lot of observability with very little effort. You can code whatever custom tooling you want with an xDS control plane, and even with a static config file and some DNS records, it does a great job of ensuring that every request you make hits a working backend. (And alerting you when there are none.)

The last approach is to redesign Kubernetes networking so that it transparently provides the same thing that a sidecar would. That is where things like Istio, linkerd, and knative's networking stuff fit in. They take your deployment specification, inject a sidecar, and adjust iptables rules to force your application to talk through the sidecar. Some of them, I believe, come with a control plane so that you can perform management operations cleanly (canarys, draining a region or node, etc.).

My personal opinion is that they do too much; if you name your syslog port "grpc" by accident, goodbye traffic! What was supposed to be documentation is now just an undocumented part of the Kubernetes API. And since it's all non-standard, what works with your Istio cluster may not work with your Linkerd cluster. You can actually write well-formed config files that will work on one cluster but not on another. So I don't think this is the approach that will eventually win.

Approach 1, a smart client, is the most likely to be reliable and understandable. It's also the least likely to work in a world where people's Excel spreadsheets run in production and need to load-balance their HTTP/2 requests. Approach 2 is probably the most difficult to set up right now, but it's a good compromise for the real world. You type in the config that you want, and then that is executed in an observable and maintainable way. You have to type stuff though, so it's unpopular. Approach 3 is the most popular and most likely to cause problems. But I'm sure that if we keep bludgeoning it with enough hammers and hire enough Kubernetes Certified Operators at $300,000 a year, we can make it work!

Sorry for the rant but it just blows my mind at how much time people will spend to make something transparent that is actually fine to see. All we need from Kubernetes is to tell me where the replicas are. Then we can write our own load-balancing that works for our application.