Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

eBPF will help solve service mesh by getting rid of sidecars

isovalent.com

Post reply on HN
Print view View original

81–90 of 110 posts

Re: eBPF will help solve service mesh by getting rid of sidecars

#81

Post by tptacek »

the_duke wrote: post #78
tptacek wrote: post #63

Earlier quoted context omitted.

It doesn't make any difference which language you use; the security promises are coming from the verifier, which is analyzing the CFG of the compiled program. C is what most people use, since the underlying APIs are in C, and since the verifier is so limiting that most high-level constructions are off the table.

Sure, I was not implying that Rust would have any security benefits fir eBPF. Just that you can even write eBPF code in more convenient languages.

This has come up here a bunch of times (we do a lot of work in Rust). I've been a little skeptical that Rust is a win here, for basically the reason I gave upthread: you can't really do much with Rust in eBPF, because the verifier won't let you; it seems to me like you'd be writing a dialect of Rust-shaped C. But we did a recent work sample challenge for Rust candidates that included an eBPF component, and a couple good submissions used Rust eBPF, so maybe I'm wrong about that.

I'm also biased because I love writing C code (I know, both viscerally and intellectually, that I should virtually never do so; eBPF is the one sane exception!)

Re: eBPF will help solve service mesh by getting rid of sidecars

#82

Post by bogomipz »

kklimonda wrote: post #69
q3k wrote: post #7

Earlier quoted context omitted.

> Honestly after I learned that the majority of Kubernetes nodes just proxy traffic between each other using iptables and that a load balancer can't tell the nodes apart (ones where your app lives vs ones that will proxy connection to your app) I got really worried about any kind of persistent connection in k8s land. There can be a difference, if your LoadBalancer-type service integration is well implemented. The ext…

How do you keep ecmp hashing stable between rollouts?

ECMP hashing would be between the edge router and the IP of the LBs advertising VIPs no? The LB would maintain the mappings between the VIPs and the nodePort IPs of worker nodes that have a local service Endpoint for the requested service. I don't think this would be any different than it is without Kubernetes or am I completely misunderstanding your question?

Re: eBPF will help solve service mesh by getting rid of sidecars

#83

Post by afrodc_ »

darkwater wrote: post #76
jrockway wrote: post #71

Earlier quoted context omitted.

The big secret is that sidecars can only help so much. If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars). mTLS is similar; it's great to encrypt your internal traffic on the wire, but that needs to get propagated up to the application to make internal authoriza…

> If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars) Why not? AFAIK traces are sent from the instrumented app to some tracing backend, and a trace-id is carried over via an HTTP header from the entry point of the request until the last service that takes part in…

This. Header trace propagation is a godsend.

Re: eBPF will help solve service mesh by getting rid of sidecars

#84

Post by dilyevsky »

dharmab wrote: post #79
kklimonda wrote: post #69

Earlier quoted context omitted.

How do you keep ecmp hashing stable between rollouts?

If you're asking about connection stability in general: - Ideally, you avoid it in your application design. - If you need it, you set up SIGTERM handling in the application to wait for all connections to close before the process exits. You also set up "connection draining" at the load balancer to keep existing sessions to terminating Pods open but send new sessions to the new Pods. The tradeoff is that rollouts take…

You dont just wait until all connections exit, you first need to withdraw bgp announcement to the edge router, then start the wait. It’s not that simple with metal LBs. On the other hand it’s not that simple with cloud LBs either bc they also break long tcp streams when they please

Re: eBPF will help solve service mesh by getting rid of sidecars

#85

Post by colonelxc »

darkwater wrote: post #76
jrockway wrote: post #71

Earlier quoted context omitted.

The big secret is that sidecars can only help so much. If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars). mTLS is similar; it's great to encrypt your internal traffic on the wire, but that needs to get propagated up to the application to make internal authoriza…

> If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars) Why not? AFAIK traces are sent from the instrumented app to some tracing backend, and a trace-id is carried over via an HTTP header from the entry point of the request until the last service that takes part in…

I think the point is that the service mesh can't do the work of propagation. It needs the client to grab the input header, and attach it to any outbound requests. From the perspective of the service mesh, the service is handling X requests, and Y requests are being sent outbound. It doesn't know how each outbound request maps to an input.

So now all of the sudden we do need a client library for each service in order to make sure the header is being propagated correctly.

Re: eBPF will help solve service mesh by getting rid of sidecars

#86

Post by q3k »

kklimonda wrote: post #69
q3k wrote: post #7

Earlier quoted context omitted.

> Honestly after I learned that the majority of Kubernetes nodes just proxy traffic between each other using iptables and that a load balancer can't tell the nodes apart (ones where your app lives vs ones that will proxy connection to your app) I got really worried about any kind of persistent connection in k8s land. There can be a difference, if your LoadBalancer-type service integration is well implemented. The ext…

How do you keep ecmp hashing stable between rollouts?

You don’t :).

To do it properly you want a maglev-style layer that allows for withdrawals/drains of application servers with minimal disruption thanks to a minimum disruption maglev-style hash and draining support. This will allow you to first drain the given application server (continue maintaining existing connections, but send new ones to a secondaries for that part shard) before fully taking down the instance.

Re: eBPF will help solve service mesh by getting rid of sidecars

#87

Post by kelnos »

zdw wrote: post #8

So instead of making the applications use a good RPC library, we're going to shove more crap into the kernel? No thanks, from a security context and complexity perspective. Per https://blog.dave.tf/post/new-kubernetes/ , the way that this was solved in Borg was: > "Borg solves that complexity by fiat, decreeing that Thou Shalt Use Our Client Libraries For Everything, so there’s an obvious point at which to plug in ar…

I don't think client libraries are the answer. If you only have one technology stack (say, Java and Spring) and only use one application-layer network protocol (say, HTTP), then maybe it's fine.

But once you have more than one language or framework, you need to write more and more of these libraries. And what happens if it's not just HTTP? What if you need to speak Redis, MySQL, or some random binary protocol? Do you write client libraries for those too? Maybe a company like Google has the scale to do this, but most orgs do not. But even then, what if you have to run some vendor-supplied code that you don't even have source for?

I agree with you that shoving more of this into the kernel isn't desirable, but libraries aren't great. Been there, done that, don't feel like doing it again. I'd rather stick with sidecars.

Re: eBPF will help solve service mesh by getting rid of sidecars

#88

Post by kklimonda »

bogomipz wrote: post #82
kklimonda wrote: post #69

Earlier quoted context omitted.

How do you keep ecmp hashing stable between rollouts?

ECMP hashing would be between the edge router and the IP of the LBs advertising VIPs no? The LB would maintain the mappings between the VIPs and the nodePort IPs of worker nodes that have a local service Endpoint for the requested service. I don't think this would be any different than it is without Kubernetes or am I completely misunderstanding your question?

q3k has mentioned metallb+bgp, which is basically in-cluster implementation of LoadBalancer Service type (bgp speakers are running on k8s nodes and announce /32 routes to nodes based on configuration), but it doesn't provide an answer for "stabilizing" ecmp connections when there are changes to backends. There has to be something "behind" metallb[1] that will handle not only stable hashing for connections, but keep forwarding "in-flight" flows (like established tcp sessions) to correct backends, even if packets arrive on different ingress nodes. It seems cilium has some solution for that[2] (by both bundling metallb, and having maglev-based loadbalancer implementation) but I haven't had time to dig into it, so I was curious if someone else has solved it and would be willing to share stories from the front. This is one of those rough edges around kubernetes deployments in bare metal environments and I'd love to see what can be done to make it more robust.

[1] metallb only really announces IPs so that "behind" is probably just CNI that actually handles traffic [2] https://cilium.io/blog/2020/11/10/cilium-19#maglev

Re: eBPF will help solve service mesh by getting rid of sidecars

#89

Post by Matthias247 »

tptacek wrote: post #64
tgraf wrote: post #48

Earlier quoted context omitted.

The model I'm describing contains two pieces: 1) Moving away from sidecars to per-node proxies that can be better integrated into the Linux kernel concept of namespacing instead of artificially injecting them with complicated iptables redirection logic at the network level. 2) Providing the HTTP awareness directly with eBPF using eBPF-based protocol parsers. The parser itself is written in eBPF which has a ton of sec…

What does an HTTP parser written in BPF look like? Bounded loops only --- meaning no string libraries --- seems like a hell of a constraint there.

another challenges I can see is wheee to actually store the state of a connection. Even if we just focus on http/1.1 then not all headers will be received at one, and data from previous segments needs to be carried forward. Would it be eBPF maps? Those also seem rather limited for this usecase, and are probably also not extremely fast.

I can imagine getting something to work for http/1.1 - but http/2 with multiplexing and stateful header compression is a completely different beast.

Re: eBPF will help solve service mesh by getting rid of sidecars

#90

Post by rhizome »

q3k wrote: post #86
kklimonda wrote: post #69

Earlier quoted context omitted.

How do you keep ecmp hashing stable between rollouts?

You don’t :). To do it properly you want a maglev-style layer that allows for withdrawals/drains of application servers with minimal disruption thanks to a minimum disruption maglev-style hash and draining support. This will allow you to first drain the given application server (continue maintaining existing connections, but send new ones to a secondaries for that part shard) before fully taking down the instance.

Sounds like Apache's graceful-restart.
Post reply on HN