eBPF will help solve service mesh by getting rid of sidecars • Hacker News Board

Re: eBPF will help solve service mesh by getting rid of sidecars

#41

Post by xemdetia » Thu, Dec 09, 2021, 5:03 PM UTC

cfors wrote: post #24

xmodem wrote: post #20

Doing this with eBPF is definitely an improvement, but when I look at some of the sidecars we run in production, I often wonder why we can't just... integrate them into the application.

You can! There are downsides though for any sufficiently polyglot organization, which is maintaining all the different client SDK's that need to use that. Sidecars are often useful for platform-centric teams that would like to have access to help manage something like secrets, mTLS, or traffic shaping in the case of Envoy. The team that's responsible for that just needs to maintain a single sidecar rather than all of…

Another thing too is that if your main application artifact can be static while your sidecar can react to configuration changes/patches/vulns/updates. Depending on your architecture it can make some components last for years without a change even though the sidecar/surrounding configuration is doing all sorts of stuff. Back when more people ran Java environments there were all sorts of settings you can do with just the JVM without the bytecode moving for how JCE worked which was extraordinarily helpful.

It depends on your environment and architecture combined with how fast you can move especially with third party components. Having the microservice be 'dumb' can save everything.

Re: eBPF will help solve service mesh by getting rid of sidecars

#42

Post by tgraf » Thu, Dec 09, 2021, 5:07 PM UTC

davewritescode wrote: post #39

From a resource perspective this makes sense but from a security perspective this drives me a little bit crazy. Sidecars aren't just for managing traffic, they're also a good way to automate managing the security context of the pod itself. The current security model in Istio delivers a pod specific SPIFFE cert to only that pod and pod identity is conveyed via that certificate. That feels like a whole bunch of eggs in…

What the proposed architecture allows is to continue using SPIFFE or another certificate management solution to generate and distribute the certificates but use either a per-node proxy or an eBPF implementation to enforce it. Even if the authentication handshake remains in a proxy but data encryption moves to the kernel then that is a massive benefit from an overhead perspective. This already exists and is called kTLS.

Re: eBPF will help solve service mesh by getting rid of sidecars

#43

Post by miduil » Thu, Dec 09, 2021, 5:07 PM UTC

xmodem wrote: post #20

Doing this with eBPF is definitely an improvement, but when I look at some of the sidecars we run in production, I often wonder why we can't just... integrate them into the application.

Author of linkerd argues that splitting this responsibility will improve stability as you'll have a homogeneous interface (sidecar proxy) over a heterogeneous group of pods. Updating a sidecar-container (or using the same across all applications) is possible, whereas if it's integrated into the application you'll encounter much more barriers and need much wider coordination.

Re: eBPF will help solve service mesh by getting rid of sidecars

#44

Post by darkwater » Thu, Dec 09, 2021, 5:08 PM UTC

cfors wrote: post #24

xmodem wrote: post #20

Doing this with eBPF is definitely an improvement, but when I look at some of the sidecars we run in production, I often wonder why we can't just... integrate them into the application.

You can! There are downsides though for any sufficiently polyglot organization, which is maintaining all the different client SDK's that need to use that. Sidecars are often useful for platform-centric teams that would like to have access to help manage something like secrets, mTLS, or traffic shaping in the case of Envoy. The team that's responsible for that just needs to maintain a single sidecar rather than all of…

> Especially if you have specific sidecars that only work on a specific infrastructure, for example if you have a Vault sidecar that deals with secrets for your service over EKS IAM permissions, you suddenly can't start your service without a decent amount of mocking and feature flags. Its nice to not have to burden your client code with all of that.

Could you please elaborate on this? I don't fully understand what you mean. Especially, I don't understand if "Its nice to not have to burden your client code with all of that" applies to a setup with or without sidecars.

Re: eBPF will help solve service mesh by getting rid of sidecars

#45

Post by tgraf » Thu, Dec 09, 2021, 5:09 PM UTC

codetrotter wrote: post #15

> Identity-based Security: Relying on network identifiers to achieve security is no longer sufficient, both the sending and receiving services must be able to authenticate each other based on identities instead of a network identifier. Kinda semi-offtopic but I am curious to know if anyone has used identity part of a WireGuard setup for this purpose. So say you have a bunch of machines all connected in a WireGuard VP…

One of the methods that Cilium (which implements this eBPF-based service mesh idea) uses to implementation authentication between workloads is Wireguard. It does exactly what you describe above.

In addition it can also be used to enforce based on service specific keys/certificates as well.

Re: eBPF will help solve service mesh by getting rid of sidecars

#46

Post by hestefisk » Thu, Dec 09, 2021, 5:09 PM UTC

__alexs wrote: post #29

zdw wrote: post #8

So instead of making the applications use a good RPC library, we're going to shove more crap into the kernel? No thanks, from a security context and complexity perspective. Per https://blog.dave.tf/post/new-kubernetes/ , the way that this was solved in Borg was: > "Borg solves that complexity by fiat, decreeing that Thou Shalt Use Our Client Libraries For Everything, so there’s an obvious point at which to plug in ar…

I'm sure someone will write leftPad in eBPF any day now.

Indeed. We could even embed a WASM runtime (headless v8?) so one can execute arbitrary JavaScript in-kernel… wait :)

Re: eBPF will help solve service mesh by getting rid of sidecars

#47

Post by tgraf » Thu, Dec 09, 2021, 5:12 PM UTC

zdw wrote: post #8

So instead of making the applications use a good RPC library, we're going to shove more crap into the kernel? No thanks, from a security context and complexity perspective. Per https://blog.dave.tf/post/new-kubernetes/ , the way that this was solved in Borg was: > "Borg solves that complexity by fiat, decreeing that Thou Shalt Use Our Client Libraries For Everything, so there’s an obvious point at which to plug in ar…

If you are in a position where you can do that then great. Most folks out there are in a position where they need to run arbitrary applications delivered by vendors without an ability to modify them.

The second aspect is that this can get extremely expensive if your applications are written in a wide number of language frameworks. That's obviously different at Google where the number of languages can be restricted and standardized.

But even then, you could also link a TCP library into your app. Why don't you?

Re: eBPF will help solve service mesh by getting rid of sidecars

#48

Post by tgraf » Thu, Dec 09, 2021, 5:17 PM UTC

Matthias247 wrote: post #40

I understand how BPF works for transparently steering TCP connections. But the article mentions gRPC - which means HTTP2. How can the BPF module be a replacement for a proxy here. My understanding is it would need to understand http2 framing and having buffers - which all sound like capabilities that require more than BPF? Are they implementing a http2 capable proxy in native kernel C code and making APIs to that acc…

The model I'm describing contains two pieces: 1) Moving away from sidecars to per-node proxies that can be better integrated into the Linux kernel concept of namespacing instead of artificially injecting them with complicated iptables redirection logic at the network level. 2) Providing the HTTP awareness directly with eBPF using eBPF-based protocol parsers. The parser itself is written in eBPF which has a ton of security benefits because it runs in a sandboxed environment.

We are doing both. Aspect 2) is currently done for HTTP visibility and we will be working on connection splicing and HTTP header mutation going forward.

Re: eBPF will help solve service mesh by getting rid of sidecars

#49

Post by cfors » Thu, Dec 09, 2021, 5:20 PM UTC

darkwater wrote: post #44

cfors wrote: post #24

Earlier quoted context omitted.

You can! There are downsides though for any sufficiently polyglot organization, which is maintaining all the different client SDK's that need to use that. Sidecars are often useful for platform-centric teams that would like to have access to help manage something like secrets, mTLS, or traffic shaping in the case of Envoy. The team that's responsible for that just needs to maintain a single sidecar rather than all of…

> Especially if you have specific sidecars that only work on a specific infrastructure, for example if you have a Vault sidecar that deals with secrets for your service over EKS IAM permissions, you suddenly can't start your service without a decent amount of mocking and feature flags. Its nice to not have to burden your client code with all of that. Could you please elaborate on this? I don't fully understand what y…

Take vault for example. Rather than have to toggle a flag in your service to get a secret, you could have the vault sidecar inject the secret automatically into your container, as opposed to having to pass a configuration flag `USE_VAULT` to your application, which will conditionally have a baked in vault client that fetches your secret for you.

Your service doesn't really care where the secret comes from, as long it can use that secret to connect to some database, API or whatever. So IMO it makes your application code a bit cleaner knowing that it doesn't have to worry about where to fetch a secret from.

Re: eBPF will help solve service mesh by getting rid of sidecars

#50

Post by jayd16 » Thu, Dec 09, 2021, 5:34 PM UTC

pjmlp wrote: post #34

outside1234 wrote: post #19

Earlier quoted context omitted.

The industry is moving away from the client library approach. This is possible in a place like Google where they force folks to write software in one of four languages (C++, Java, Go, Python) but doesn't scale to a broader ecosystem.

It sure scales, I am yet to work in organisations where everything goes. There are a set of sanctioned languages and that is about it.

The subtle aspect of the comment you're replying to is that _they write everything_.

Hard to cram a new library into some closed source vendor app.