Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on BPF and eBPF

jvns.ca

Post reply on HN
Print view View original

41–46 of 46 posts

Re: Notes on BPF and eBPF

#41

Post by _3u10 »

tptacek wrote: post #30
_3u10 wrote: post #9

Earlier quoted context omitted.

It’s not an easier interface. It’s much easier to write kernel modules than to mess around with eBPF.

That seems wrong. The build environment for eBPF programs is simpler (you don't even need a working kernel tree), and, much more importantly, eBPF programs are constrained and can't crash the kernel, which is easy to do accidentally when writing an freestyle C LKM. You can learn enough to get stuff done with eBPF inside of a couple days; the same is absolutely not true of kernel modules.

I find the verifier to be extremely frustrating. It's also very difficult to debug eBPF programs, especially in regards to why it's not passing verifier. Combine with ridiculous restrictions from BCC like not being able to use functions in macros, or the vagueries of the bpf target in clang and it's extremely frustrating.

I've personally never found obtaining a working kernel tree to be difficult, certainly easier than a working BCC toolchain. Or all of the various compiler flags needed for clang not to emit code incompatible with the verifier.

Re: Notes on BPF and eBPF

#42

Post by tptacek »

_3u10 wrote: post #41
tptacek wrote: post #30

Earlier quoted context omitted.

That seems wrong. The build environment for eBPF programs is simpler (you don't even need a working kernel tree), and, much more importantly, eBPF programs are constrained and can't crash the kernel, which is easy to do accidentally when writing an freestyle C LKM. You can learn enough to get stuff done with eBPF inside of a couple days; the same is absolutely not true of kernel modules.

I find the verifier to be extremely frustrating. It's also very difficult to debug eBPF programs, especially in regards to why it's not passing verifier. Combine with ridiculous restrictions from BCC like not being able to use functions in macros, or the vagueries of the bpf target in clang and it's extremely frustrating. I've personally never found obtaining a working kernel tree to be difficult, certainly easier th…

I don't use BCC or really understand why anyone would; I use clang, and just produce .o's.

The verifier is definitely annoying, especially at first, but I found myself sort of quickly working out the verifier's expected idiom, and a lot of it can be wrapped with macros.

All of this drama pales in comparison to writing freestyle C code in the Linux kernel without causing random panics.

Re: Notes on BPF and eBPF

#43

Post by xyzzy_plugh »

tptacek wrote: post #31
xyzzy_plugh wrote: post #15

Earlier quoted context omitted.

I've crashed multiple kernels with eBPF programs. I don't buy this argument at all.

Can you describe some of those bugs? What helpers were you using? I've been doing bonkers stuff with eBPF for the last year and, while I've definitely had bugs, none of them took my kernel down.

I no longer have access to the code, and the kernel has since been patched. Mostly doing some observability around new network connections.

We definitely hit this one at some point: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1763454

We also ran into a couple of similar but unrelated panic bugs on much newer kernels on non-Ubuntu distros.

Re: Notes on BPF and eBPF

#44

Post by xyzzy_plugh »

zokula wrote: post #28
xyzzy_plugh wrote: post #15

Earlier quoted context omitted.

I've crashed multiple kernels with eBPF programs. I don't buy this argument at all.

I call bullshit! Ebpf programs cannot crash the kernel since they cannot be run if they contain bugs in the first place.

The kernel, however, can and does have bugs. I've seen several thousand hosts get taken down with a perfectly correct eBPF program due to a buggy kernel.

Re: Notes on BPF and eBPF

#45

Post by justicezyx »

xyzzy_plugh wrote: post #15
stormbrew wrote: post #13

Earlier quoted context omitted.

Easier to get started maybe, but there's something to be said for the 'ease' of having to worry a lot less about crashing the kernel you're working on.

I've crashed multiple kernels with eBPF programs. I don't buy this argument at all.

Could you share the ebpf program and the external dependencies that can reproduce the crash?

It'll be valuable to learn this, so that we might be able to proactively address them. Ebpf is the core of our product.

Re: Notes on BPF and eBPF

#46

Post by justicezyx »

Sirened wrote: post #19
pwnna wrote: post #4

BPF is indeed a pretty interesting technology. As the knowledge about it becomes more widespread, I anticipate that we will unlock some new capabilities both in terms of tracing. Brendan Gregg's book ( https://www.brendangregg.com/bpf-performance-tools-book.html ) serves as a good intro to this, although you probably only need to read a small chunk of it as a lot of it is reference-book-style material. The author's m…

>With the proper BPF tracing scripts, I think we can validate that programs indeed avoids page faults Sorry, I'm a little confused why this would be necessary? Like, sure, it's a nice to have on a CI as a basic sanity check but if you just invoke mlockall you'll end up with everything wired down and you're good to go regardless?

mloclall () requires the application under check to call them such that the application process cannot modify memory pages.

Whereas ebpf allows instrumentation free enforcement. Plus, app devs do not need to be aware of this fact. This facilitate separate of responsibility in code and organization.

Post reply on HN