Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

eBPF will help solve service mesh by getting rid of sidecars

isovalent.com

Post reply on HN
Print view View original

91–100 of 110 posts

Re: eBPF will help solve service mesh by getting rid of sidecars

#91

Post by jayd16 »

lamontcg wrote: post #75
jayd16 wrote: post #52

Earlier quoted context omitted.

It does feel a bit like we're trying to monkey patch compiled code but the benefits are pretty clear.

I would argue pretty strenuously that this is not what is being done. The sockets layer is becoming a facade which can guarantee additional things to applications which are compiled against it, and you've got dependency injection here so that the application layer can be written agnostically and not care about any of those concerns at all.

Well ok, but the dependency inject is not statically checked in this case, its changed dynamically, perhaps while the application is running. Is that not similar to a monkey patch?

Re: eBPF will help solve service mesh by getting rid of sidecars

#92

Post by klysm »

zdw wrote: post #8

So instead of making the applications use a good RPC library, we're going to shove more crap into the kernel? No thanks, from a security context and complexity perspective. Per https://blog.dave.tf/post/new-kubernetes/ , the way that this was solved in Borg was: > "Borg solves that complexity by fiat, decreeing that Thou Shalt Use Our Client Libraries For Everything, so there’s an obvious point at which to plug in ar…

That’s great if you write all your software. If you want to use someone else’s thing then you have to wrap it in that magic everywhere client.

Re: eBPF will help solve service mesh by getting rid of sidecars

#93

Post by sciurus »

colonelxc wrote: post #85
darkwater wrote: post #76

Earlier quoted context omitted.

> If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars) Why not? AFAIK traces are sent from the instrumented app to some tracing backend, and a trace-id is carried over via an HTTP header from the entry point of the request until the last service that takes part in…

I think the point is that the service mesh can't do the work of propagation. It needs the client to grab the input header, and attach it to any outbound requests. From the perspective of the service mesh, the service is handling X requests, and Y requests are being sent outbound. It doesn't know how each outbound request maps to an input. So now all of the sudden we do need a client library for each service in order…

You described the problem correctly.

I think it's a stretch to say that requires a client library, though. It should be straightforward to have whatever library you are already using for http requests pass those headers through.

https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...

Re: eBPF will help solve service mesh by getting rid of sidecars

#94

Post by sciurus »

darkwater wrote: post #76
jrockway wrote: post #71

Earlier quoted context omitted.

The big secret is that sidecars can only help so much. If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars). mTLS is similar; it's great to encrypt your internal traffic on the wire, but that needs to get propagated up to the application to make internal authoriza…

> If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars) Why not? AFAIK traces are sent from the instrumented app to some tracing backend, and a trace-id is carried over via an HTTP header from the entry point of the request until the last service that takes part in…

> a trace-id is carried over via an HTTP header from the entry point of the request until the last service that takes part in that request.

But it's not, unless you specifically code your services to do that. Which isn't hard, but just means plugging an unmodified service into a service mesh isn't enough.

Re: eBPF will help solve service mesh by getting rid of sidecars

#95

Post by gz5 »

codetrotter wrote: post #15

> Identity-based Security: Relying on network identifiers to achieve security is no longer sufficient, both the sending and receiving services must be able to authenticate each other based on identities instead of a network identifier. Kinda semi-offtopic but I am curious to know if anyone has used identity part of a WireGuard setup for this purpose. So say you have a bunch of machines all connected in a WireGuard VP…

Ziti (Apache) provides bootstrapped* identity based security (and programmable, least privilege overlays).

Disclosure: founder of company which sells SaaS on top of Ziti FOSS.

* https://ziti.dev/blog/bootstrapping-trust-part-5-bootstrappi...

Re: eBPF will help solve service mesh by getting rid of sidecars

#96

Post by rhizome »

jjtheblunt wrote: post #80
outside1234 wrote: post #21

Earlier quoted context omitted.

The network is the base library - that is the shift you are seeing. You make a call out to a network address with a specific protocol. Also, as an aside, I think WebAssembly has the potential to shift this back. In a world where libraries and programs are compiled to WebAssembly, it doesn't matter what their source language was, and as such, the client library based approach might swing back into vogue.

> The network is the base library you remind me of the 20+ years ago Sun Microsystems assertion "The Network IS the Computer". citation: https://www.networkcomputing.com/cloud-infrastructure/networ...

Well they did put the dot in dot-com.

Re: eBPF will help solve service mesh by getting rid of sidecars

#97

Post by transitorykris »

rhizome wrote: post #90
q3k wrote: post #86

Earlier quoted context omitted.

You don’t :). To do it properly you want a maglev-style layer that allows for withdrawals/drains of application servers with minimal disruption thanks to a minimum disruption maglev-style hash and draining support. This will allow you to first drain the given application server (continue maintaining existing connections, but send new ones to a secondaries for that part shard) before fully taking down the instance.

Sounds like Apache's graceful-restart.

Sort of. Processes on the same node (graceful restart) vs processes on different nodes (maglev).

Re: eBPF will help solve service mesh by getting rid of sidecars

#98

Post by justicezyx »

tptacek wrote: post #64
tgraf wrote: post #48

Earlier quoted context omitted.

The model I'm describing contains two pieces: 1) Moving away from sidecars to per-node proxies that can be better integrated into the Linux kernel concept of namespacing instead of artificially injecting them with complicated iptables redirection logic at the network level. 2) Providing the HTTP awareness directly with eBPF using eBPF-based protocol parsers. The parser itself is written in eBPF which has a ton of sec…

What does an HTTP parser written in BPF look like? Bounded loops only --- meaning no string libraries --- seems like a hell of a constraint there.

Bounded loop plus 1M instruction limits in the 5.4 kernel (no record at hands about the exact version), gives a large range of supported headers. Also note that these BPF code are on the network level, which is subject to the MTU limit as well, which usually is 1500 and now can be 10s of KBs (65,525 bytes maxmial in theory accroding to https://www.lifewire.com/definition-of-mtu-817948, but my networking knownledge is poor). These makes it possible to effectively handle all possible headers.

HTTP is actually fine.

HTTP2 will be a bigger issue as it has HPACK, and Huffman coding, that would be very complicated to maintain inside BPF runtime. I haven't thought about it closely yet. But based on our experience at http://px.dev, I am not aware of any glaring technical obstacles.

Re: eBPF will help solve service mesh by getting rid of sidecars

#99

Post by tptacek »

justicezyx wrote: post #98
tptacek wrote: post #64

Earlier quoted context omitted.

What does an HTTP parser written in BPF look like? Bounded loops only --- meaning no string libraries --- seems like a hell of a constraint there.

Bounded loop plus 1M instruction limits in the 5.4 kernel (no record at hands about the exact version), gives a large range of supported headers. Also note that these BPF code are on the network level, which is subject to the MTU limit as well, which usually is 1500 and now can be 10s of KBs (65,525 bytes maxmial in theory accroding to https://www.lifewire.com/definition-of-mtu-817948 , but my networking knownledge i…

This is interesting and all, but I've also written bounded loop BPF code on 5.6 kernels, and it is not easy to get the verifier to accept seemingly obvious loops. I'm not saying it's impossible, I'm saying I'd like to see what this code actually looks like. I'd be a little shocked if it just looked exactly like Node's HTTP parser.

Re: eBPF will help solve service mesh by getting rid of sidecars

#100

Post by bogomipz »

kklimonda wrote: post #88
bogomipz wrote: post #82

Earlier quoted context omitted.

ECMP hashing would be between the edge router and the IP of the LBs advertising VIPs no? The LB would maintain the mappings between the VIPs and the nodePort IPs of worker nodes that have a local service Endpoint for the requested service. I don't think this would be any different than it is without Kubernetes or am I completely misunderstanding your question?

q3k has mentioned metallb+bgp, which is basically in-cluster implementation of LoadBalancer Service type (bgp speakers are running on k8s nodes and announce /32 routes to nodes based on configuration), but it doesn't provide an answer for "stabilizing" ecmp connections when there are changes to backends. There has to be something "behind" metallb[1] that will handle not only stable hashing for connections, but keep f…

Ah OK, I missed that this was MetalLB specific. Interesting that Cilium using Google's Maglev which amongst other things handles the issue of ECMP churn when nodes are taken out of service. I remember reading this in the white paper when it came out. I believe Facebook's Katran does similar. Thanks for the link.
Post reply on HN