Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

eBPF will help solve service mesh by getting rid of sidecars

isovalent.com

Post reply on HN
Print view View original

101–110 of 110 posts

Re: eBPF will help solve service mesh by getting rid of sidecars

#101

Post by topspin »

dharmab wrote: post #62
topspin wrote: post #57

Earlier quoted context omitted.

> In most professional setups you do have such a load balancer May I ask what one might use in an AWS cloud environment to provide that load balancer within a Region? Does IPv6 address any of these issues? It seems to me that IPv6 is capable of providing every component in the system its own globally routable address, identity (mTLS perhaps) and transparent encryption with no extra sidecars, eBPF pieces, etc.

> May I ask what one might use in an AWS cloud environment to provide that load balancer within a Region? The AWS cloud controller will automatically set up an ALB for you if you configure a LoadBalancer service in Kubernetes. I've also done custom setups with AWS NLBs. > Does IPv6 address any of these issues? It could address some issues- you could conceivably create a CNI plugin which allocates an externally addres…

Are ALB/NLB employed to handle traffic between pods in the same cluster? Or have I misunderstood the whole discussion?

My take on the 'eBPF will help solve service mesh' proposal is that it deals with not only ingress/egress traffic (where ALB/NLB are typically employed) but all traffic, including traffic between pods in a cluster. This is where my interests lay.

Re: eBPF will help solve service mesh by getting rid of sidecars

#102

Post by justicezyx »

tptacek wrote: post #99
justicezyx wrote: post #98

Earlier quoted context omitted.

Bounded loop plus 1M instruction limits in the 5.4 kernel (no record at hands about the exact version), gives a large range of supported headers. Also note that these BPF code are on the network level, which is subject to the MTU limit as well, which usually is 1500 and now can be 10s of KBs (65,525 bytes maxmial in theory accroding to https://www.lifewire.com/definition-of-mtu-817948 , but my networking knownledge i…

This is interesting and all, but I've also written bounded loop BPF code on 5.6 kernels, and it is not easy to get the verifier to accept seemingly obvious loops. I'm not saying it's impossible, I'm saying I'd like to see what this code actually looks like. I'd be a little shocked if it just looked exactly like Node's HTTP parser.

I need to double verify when was the bounded loop patch got into the kernel, I suppose it's 5.6 as you mentioned above.

What I actually was thinking is that one can write C code and ask the compiler to unroll it.

``` pragma(unroll) for (..., i Also the other comment note the stake bookkeeping for HTTP to maintain the state when the parsing spans multiple packets, assuming here we are talking about XDP probes.

One quick idea is to use BPF_TABLE(, uint128_t, some data structure) I haven't tested if uint128_t is OK as key type. And the data structure in the value needs more thoughts. Roughly I am thinking turn any state bookkeeping into some BPF tables, and keyed through whatever data that matches the context. This probably means uint128_t as Ipve/6 address, and a nested map with key as the port. Or combined v4 IP & port.

It'll be interesting. I suppose the code from Isovalent will eventually be open sourced. Or is it already so? Haven't checked yet.

Re: eBPF will help solve service mesh by getting rid of sidecars

#103

Post by tptacek »

justicezyx wrote: post #102
tptacek wrote: post #99

Earlier quoted context omitted.

This is interesting and all, but I've also written bounded loop BPF code on 5.6 kernels, and it is not easy to get the verifier to accept seemingly obvious loops. I'm not saying it's impossible, I'm saying I'd like to see what this code actually looks like. I'd be a little shocked if it just looked exactly like Node's HTTP parser.

I need to double verify when was the bounded loop patch got into the kernel, I suppose it's 5.6 as you mentioned above. What I actually was thinking is that one can write C code and ask the compiler to unroll it. ``` pragma(unroll) for (..., i Also the other comment note the stake bookkeeping for HTTP to maintain the state when the parsing spans multiple packets, assuming here we are talking about XDP probes. One qui…

Bounded loops are 5.3. I'm just saying that after like 9 months of development following their introduction, it remained tricky to get the verifier to accept loops with seemingly obvious bounds. I know the feature works (I did ultimately get some loops working!) but I could not have straightforwardly ported userland C code to do it.

You've always been able to unroll loops, but of course you're chewing up code space doing that.

I don't know what BPF_TABLE is (I think it's a BCC-ism?) but BPF hash maps can take 16 byte keys. But notice that you're now writing something that looks nothing at all like Node's HTTP parser.

I'm not doubting that they did this work. I just want to know what it ends up looking like!

Re: eBPF will help solve service mesh by getting rid of sidecars

#104

Post by justicezyx »

tptacek wrote: post #103
justicezyx wrote: post #102

Earlier quoted context omitted.

I need to double verify when was the bounded loop patch got into the kernel, I suppose it's 5.6 as you mentioned above. What I actually was thinking is that one can write C code and ask the compiler to unroll it. ``` pragma(unroll) for (..., i Also the other comment note the stake bookkeeping for HTTP to maintain the state when the parsing spans multiple packets, assuming here we are talking about XDP probes. One qui…

Bounded loops are 5.3. I'm just saying that after like 9 months of development following their introduction, it remained tricky to get the verifier to accept loops with seemingly obvious bounds. I know the feature works (I did ultimately get some loops working!) but I could not have straightforwardly ported userland C code to do it. You've always been able to unroll loops, but of course you're chewing up code space d…

Oh nice, we haven't tried bounded loop, because our product is committed to support as old as 4.13.

BPF_TABLE is BCC.

Re: eBPF will help solve service mesh by getting rid of sidecars

#105

Post by dharmab »

dilyevsky wrote: post #84
dharmab wrote: post #79

Earlier quoted context omitted.

If you're asking about connection stability in general: - Ideally, you avoid it in your application design. - If you need it, you set up SIGTERM handling in the application to wait for all connections to close before the process exits. You also set up "connection draining" at the load balancer to keep existing sessions to terminating Pods open but send new sessions to the new Pods. The tradeoff is that rollouts take…

You dont just wait until all connections exit, you first need to withdraw bgp announcement to the edge router, then start the wait. It’s not that simple with metal LBs. On the other hand it’s not that simple with cloud LBs either bc they also break long tcp streams when they please

We reused the LB as much as possible to avoid the BGP thing. There's a thing called MetalLB designed around that though.

https://metallb.universe.tf/

Re: eBPF will help solve service mesh by getting rid of sidecars

#106

Post by dharmab »

topspin wrote: post #101
dharmab wrote: post #62

Earlier quoted context omitted.

> May I ask what one might use in an AWS cloud environment to provide that load balancer within a Region? The AWS cloud controller will automatically set up an ALB for you if you configure a LoadBalancer service in Kubernetes. I've also done custom setups with AWS NLBs. > Does IPv6 address any of these issues? It could address some issues- you could conceivably create a CNI plugin which allocates an externally addres…

Are ALB/NLB employed to handle traffic between pods in the same cluster? Or have I misunderstood the whole discussion? My take on the 'eBPF will help solve service mesh' proposal is that it deals with not only ingress/egress traffic (where ALB/NLB are typically employed) but all traffic, including traffic between pods in a cluster. This is where my interests lay.

> Are ALB/NLB employed to handle traffic between pods in the same cluster?

You can choose to do so, or you can communicate directly via the built-in Kubernetes service discovery and CNI overlay network. There are use cases for both.

Re: eBPF will help solve service mesh by getting rid of sidecars

#107

Post by darkwater »

colonelxc wrote: post #85
darkwater wrote: post #76

Earlier quoted context omitted.

> If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars) Why not? AFAIK traces are sent from the instrumented app to some tracing backend, and a trace-id is carried over via an HTTP header from the entry point of the request until the last service that takes part in…

I think the point is that the service mesh can't do the work of propagation. It needs the client to grab the input header, and attach it to any outbound requests. From the perspective of the service mesh, the service is handling X requests, and Y requests are being sent outbound. It doesn't know how each outbound request maps to an input. So now all of the sudden we do need a client library for each service in order…

But tracing cannot be done anyway with a sidecar and no modification to the service code anyway. With a sidecar (or eBPF) you will get blackbox metrics for free (connections throughput, latency, errors etc) but tracing it needs to be done inside the code (even automatically by some third-party library/addon or instrumenting manually). I understand the point that, once you are there instrumenting for tracing, you can also instrument for metrics and not use a sidecar. But to be fair distributed tracing is something that's catching on only now and metrics give you already some kind of visibility that it's better to have that not to have. On top of that you can add tracing and improve the observability.

Re: eBPF will help solve service mesh by getting rid of sidecars

#108

Post by xmodem »

cfors wrote: post #49
darkwater wrote: post #44

Earlier quoted context omitted.

> Especially if you have specific sidecars that only work on a specific infrastructure, for example if you have a Vault sidecar that deals with secrets for your service over EKS IAM permissions, you suddenly can't start your service without a decent amount of mocking and feature flags. Its nice to not have to burden your client code with all of that. Could you please elaborate on this? I don't fully understand what y…

Take vault for example. Rather than have to toggle a flag in your service to get a secret, you could have the vault sidecar inject the secret automatically into your container, as opposed to having to pass a configuration flag `USE_VAULT` to your application, which will conditionally have a baked in vault client that fetches your secret for you. Your service doesn't really care where the secret comes from, as long it…

This is actually exactly the case I was thinking of. We have a few applications that use vault in a much more in-depth way than fetching a couple of secrets, so have the need to interact with it directly. We then have the much more common case of applications that use vault to fetch their database credentials, and for those we use a sidecar to fetch them at startup and another to renew them every 30 minutes.

The 2x sidecars do with 150 lines of YAML configuration what could be done with a library and 10 lines of java. And I don't buy the other theoretical benefits either. Easier to update? Each service can reference the library centrally from our monorepo whereas the YAML is copy-pasted to every service. It's also statically type-checked. Polyglot? Yeah, fair, but we're an almost entirely JVM shop.

Some of this could maybe be made easier with something like Kubevela but I don't think you're actually eliminating any complexity that way, just hiding it.

Re: eBPF will help solve service mesh by getting rid of sidecars

#109

Post by dilyevsky »

dharmab wrote: post #105
dilyevsky wrote: post #84

Earlier quoted context omitted.

You dont just wait until all connections exit, you first need to withdraw bgp announcement to the edge router, then start the wait. It’s not that simple with metal LBs. On the other hand it’s not that simple with cloud LBs either bc they also break long tcp streams when they please

We reused the LB as much as possible to avoid the BGP thing. There's a thing called MetalLB designed around that though. https://metallb.universe.tf/

Pretty sure metallb will have same problem when you need to rotate nodes in bgp mode

Re: eBPF will help solve service mesh by getting rid of sidecars

#110

Post by rhizome »

transitorykris wrote: post #97
rhizome wrote: post #90

Earlier quoted context omitted.

Sounds like Apache's graceful-restart.

Sort of. Processes on the same node (graceful restart) vs processes on different nodes (maglev).

Eh, a signal is a signal even if it's an RPC, but my point was to focus on the "waiting for something to end or empty before restarting" part.
Post reply on HN