The Service Mesh: What Engineers Need to Know

Once you have this underlying grid of services all connecting via a service mesh - in a reliable, secure and observable way - for some of these services you want to define governance and on-boarding rules, or you want to expose them externally as products that developer can consume and you then use an API GW to do those things.

In this regards, an API GW is just a service among the services you are running inside the service mesh.

What amuses me about this is back in the day everyone thought the Mach guys were crazy for thinking things like network routing and IPC services be implemented in user space... and others mocked the OSI model's 7 layers as overly complex (e.g. RFC3439's "layering considered harmful").

Now we've moved all our network services onto a layer 7 protocol (HTTP), and we've discovered we need to reinvent layers we skipped over on top of it. We're doing it all in user space with comparatively new and untested application logic, somehow forgetting that this can be done far more efficiently and scalably with established and far more sophisticated networking tools... if only we'd give up on this silly notion that everything must go over HTTP.

The idea behind a service mesh is that no matter how you design and implement a microservice architecture, you still have all these little services talking to each other somehow. By "default", they're communicating using native sockets and HTTP or gRPC.

There's lots that can go wrong any time you have a bunch of things talking to each other on a network (see Tanenbaum's "Critique of RPCs" for a classic explanation).

If everything was written in the same language, what you'd probably do is come up with a common networking/RPC library for all your services to use. That library would give you a common interface, so everything used the network the same way, and some measure of observability (maybe just by doing a standard log format), and maybe some security controls like a standard HTTPS connection and a standard token format.

But if you've got multiple languages, that option isn't very attractive anymore, because even if it's feasible to build that library once for each platform, now you're keeping an additional component (the library) in sync between the platforms, and that's a drag.

The insight behind a service mesh is that containers make it cheap and easy to just stack a tiny out-of-process proxy alongside your services. So you can just move all the logic you would have had in that common service library into the proxy. The proxy will give you the same features no matter whether your service is a Rust binary, Clojure running on the JVM, or a shell script.

And, because it's an out-of-process standalone component, it can be built and maintained by a third party, which means the features it provides can be a lot more ambitious than you'd build in your own library. So now instead of just hoping to get some TLS and maybe a standard token, everything can be mTLS with client certificates, a standard dashboard for managing certificates, rule systems for what client certs will allow you to talk to which systems, graphical maps of who's communicating with who and trace collection for specific pairs of services, etc.

This is all very un- like what an API gateway does, and the whole concept sort of revolves around having little reverse proxies running alongside the services.