eBPF will help solve service mesh by getting rid of sidecars • Hacker News Board

Re: eBPF will help solve service mesh by getting rid of sidecars

#31

Post by dharmab » Thu, Dec 09, 2021, 4:17 PM UTC

dijit wrote: post #3

Honestly after I learned that the majority of Kubernetes nodes just proxy traffic between each other using iptables and that a load balancer can't tell the nodes apart (ones where your app lives vs ones that will proxy connection to your app) I got really worried about any kind of persistent connection in k8s land. Since some number of persistent connections will get force terminated on scale down or node replacement…

That's if you're using a NodePort service, which the documentation explains is for niche use cases such as if you don't have a compatible dedicated load balancer. In most professional setups you do have such a load balancer and can use other types of routing that avoid this.

https://kubernetes.io/docs/concepts/services-networking/serv...

Re: eBPF will help solve service mesh by getting rid of sidecars

#32

Post by pjmlp » Thu, Dec 09, 2021, 4:17 PM UTC

dboreham wrote: post #28

xmodem wrote: post #20

Doing this with eBPF is definitely an improvement, but when I look at some of the sidecars we run in production, I often wonder why we can't just... integrate them into the application.

Like it or not the socket has become the demarcation mechanism we use. Therefore all software ends up deployed as a thing that talks on sockets. Therefore you can't/shouldn't put functionality that belongs on the other end of the socket inside that thing. If you do that it's no longer the kind of thing you wanted (a discrete unit of software that does something). It's now a larger kind of component (software that doe…

The irony is arguing for monolithic kernels with a pile of such layers on top.

Re: eBPF will help solve service mesh by getting rid of sidecars

#33

Post by pjmlp » Thu, Dec 09, 2021, 4:19 PM UTC

cfors wrote: post #24

xmodem wrote: post #20

Doing this with eBPF is definitely an improvement, but when I look at some of the sidecars we run in production, I often wonder why we can't just... integrate them into the application.

You can! There are downsides though for any sufficiently polyglot organization, which is maintaining all the different client SDK's that need to use that. Sidecars are often useful for platform-centric teams that would like to have access to help manage something like secrets, mTLS, or traffic shaping in the case of Envoy. The team that's responsible for that just needs to maintain a single sidecar rather than all of…

For a moment I thought you're talking about POSIX directory services.

Re: eBPF will help solve service mesh by getting rid of sidecars

#34

Post by pjmlp » Thu, Dec 09, 2021, 4:22 PM UTC

outside1234 wrote: post #19

zdw wrote: post #8

So instead of making the applications use a good RPC library, we're going to shove more crap into the kernel? No thanks, from a security context and complexity perspective. Per https://blog.dave.tf/post/new-kubernetes/ , the way that this was solved in Borg was: > "Borg solves that complexity by fiat, decreeing that Thou Shalt Use Our Client Libraries For Everything, so there’s an obvious point at which to plug in ar…

The industry is moving away from the client library approach. This is possible in a place like Google where they force folks to write software in one of four languages (C++, Java, Go, Python) but doesn't scale to a broader ecosystem.

It sure scales, I am yet to work in organisations where everything goes.

There are a set of sanctioned languages and that is about it.

Re: eBPF will help solve service mesh by getting rid of sidecars

#35

Post by taeric » Thu, Dec 09, 2021, 4:27 PM UTC

mixedCase wrote: post #22

xmodem wrote: post #20

Doing this with eBPF is definitely an improvement, but when I look at some of the sidecars we run in production, I often wonder why we can't just... integrate them into the application.

There are good reasons more often than not. Being able to pick up something generic rather than something language-specific. Not having to do process supervision (which includes handling monitoring and logs) within your application. Not making the application lifecycle subservient to needs such as log shipping and request rerouting. People get sig traps wrong suprisingly often.

My gut is that using sidecars doesn't really solve these problems straight up. Just moved them to the orchestrator.

Which is not bad. But that area is also often misconfigured for supervision. And trapping signals remains mostly broken in all sidecars.

Re: eBPF will help solve service mesh by getting rid of sidecars

#36

Post by MayeulC » Thu, Dec 09, 2021, 4:52 PM UTC

__alexs wrote: post #30

ZeroCool2u wrote: post #10

Earlier quoted context omitted.

What if a client library does not yet exist for your language?

There are only 5 languages. JavaScript, C++, Java, Python, C# This is basically the same set of languages people were writing 20 years ago and will probably be the same set of languages people will write in 20 years from now.

It really depends on your domain. I haven't seen C# a lot, nor python, in some orgs.

For some (like me), it's more a superset of C, assembly, bash, maybe lisp, python and matlab.

For others, it's going to be JavaScript, PHP, CSS, HTML..

I agree though that a library is usually domain-specific, and that you can probably easily identify the subset of languages that you really need official bindings for (thereby making my comment a bit useless, sorry for the noise).

Re: eBPF will help solve service mesh by getting rid of sidecars

#37

Post by the_duke » Thu, Dec 09, 2021, 4:54 PM UTC

tptacek wrote: post #26

zdw wrote: post #8

So instead of making the applications use a good RPC library, we're going to shove more crap into the kernel? No thanks, from a security context and complexity perspective. Per https://blog.dave.tf/post/new-kubernetes/ , the way that this was solved in Borg was: > "Borg solves that complexity by fiat, decreeing that Thou Shalt Use Our Client Libraries For Everything, so there’s an obvious point at which to plug in ar…

The complexity is an issue (but sidecars are plenty complex too), but the security not so much. BPF C is incredibly limiting (you can't even have loops if the verifier can't prove to its satisfaction that the loop has a low static bound). It's nothing at all like writing kernel C.

You don't have to use C.

There are two projects that enable writing eBPF with Rust [1][2]. I'm sure there is an equivalent with nicer wrappers for C++.

[1] https://github.com/foniod/redbpf

[2] https://github.com/aya-rs/aya

Re: eBPF will help solve service mesh by getting rid of sidecars

#38

Post by MayeulC » Thu, Dec 09, 2021, 4:55 PM UTC

zdw wrote: post #8

So instead of making the applications use a good RPC library, we're going to shove more crap into the kernel? No thanks, from a security context and complexity perspective. Per https://blog.dave.tf/post/new-kubernetes/ , the way that this was solved in Borg was: > "Borg solves that complexity by fiat, decreeing that Thou Shalt Use Our Client Libraries For Everything, so there’s an obvious point at which to plug in ar…

> a good RPC library

I like that approach. If you use client libraries, new RPC mechanisms are "free" to implement (until you need to troubleshoot upgrades). It's also an argument against statically linking.

For instance, if running services on the same machine, io-uring can probably be used? (I'm a noob at this). eBPF for packet switching/forwarding between different hosts, etc.

Re: eBPF will help solve service mesh by getting rid of sidecars

#39

Post by davewritescode » Thu, Dec 09, 2021, 4:59 PM UTC

From a resource perspective this makes sense but from a security perspective this drives me a little bit crazy. Sidecars aren't just for managing traffic, they're also a good way to automate managing the security context of the pod itself.

The current security model in Istio delivers a pod specific SPIFFE cert to only that pod and pod identity is conveyed via that certificate.

That feels like a whole bunch of eggs in 1 basket.

Re: eBPF will help solve service mesh by getting rid of sidecars

#40

Post by Matthias247 » Thu, Dec 09, 2021, 4:59 PM UTC

I understand how BPF works for transparently steering TCP connections. But the article mentions gRPC - which means HTTP2. How can the BPF module be a replacement for a proxy here. My understanding is it would need to understand http2 framing and having buffers - which all sound like capabilities that require more than BPF?

Are they implementing a http2 capable proxy in native kernel C code and making APIs to that accessible via bpf?