How to add eBPF observability to your product • Hacker News Board

Re: How to add eBPF observability to your product

#11

Post by bch » Sat, Jul 03, 2021, 7:05 PM UTC

enedil wrote: post #9

hn_throwaway_99 wrote: post #8

Earlier quoted context omitted.

My biggest pet peeve is when authors don't just start with a 1-2 sentence outline of the topic that they're discussing, and instead go directly into acronym soup. And I've been in software development for a long time and have never heard of BPF/eBPF before. Thank you for your explanation.

Brendan Gregg is the author of eBPF, and he writes on his blog. It is fair to assume a basic knowledge of his works when reading his website.

Maybe I’m misunderstanding what you mean by “author“, but BPF was invented by Steven McCanne, and Van Jacobson in 1992[0], and tweaked over the years by various developers and operating systems. It looks[1] like the Linux eBPF work was initiated by Alexei Starovoitov, and came to be exposed to users c. 2014.

[0] https://en.wikipedia.org/wiki/Berkeley_Packet_Filter

[1] https://lwn.net/Articles/740157/

Re: How to add eBPF observability to your product

#12

Post by nyellin » Sat, Jul 03, 2021, 7:59 PM UTC

This works great if you control the servers or have a uniform fleet. I used to work at a security vendor which sold ebpf based software that ran as a daemonset on customer kubernetes clusters. Compiling ebpf bytecode on the customer's hosts wasn't an option.

Iirc the real challenge was with writing kprobe ebpf functions that access native structs. I don't think we ever found a good solution for that because you need kernel headers for each machine which we didn't have.

Of course if I'm missing something obvious, do tell!

(I'm the same person who left this comment on Brendan's site as well. Not a random copy paster)

Re: How to add eBPF observability to your product

#13

Post by puptelpete » Sat, Jul 03, 2021, 8:01 PM UTC

alexchamberlain wrote: post #2

According to Wikipedia, BPF stands for [Berkeley Packet Filter][1] and can be used to observe packets sent by a process. [1]: https://en.m.wikipedia.org/wiki/Berkeley_Packet_Filter

For me, BPF is the most exciting development in the kernel world for ages.

Brendan Gregg, the author of the posted blog entry, works with BPF on observability at Netflix and delivered a keynote at UbuntuMasters 2019. The video is on his blog is a great intro. [0]

I've been watching BPF for a couple of years now, and it seems slow on the uptake, but I hope Gregg is right that people will eventually start writing new drivers, firewalls, observability and security tools, loaded from userspace, but all running safely in a kernel vm, maybe even written in new async-first programming languages.

[0] https://brendangregg.com/blog/2019-12-02/bpf-a-new-type-of-s...

Re: How to add eBPF observability to your product

#14

Post by Ristovski » Sat, Jul 03, 2021, 9:37 PM UTC

bch wrote: post #11

enedil wrote: post #9

Earlier quoted context omitted.

Brendan Gregg is the author of eBPF, and he writes on his blog. It is fair to assume a basic knowledge of his works when reading his website.

Maybe I’m misunderstanding what you mean by “author“, but BPF was invented by Steven McCanne, and Van Jacobson in 1992[0], and tweaked over the years by various developers and operating systems. It looks[1] like the Linux e BPF work was initiated by Alexei Starovoitov, and came to be exposed to users c. 2014. [0] https://en.wikipedia.org/wiki/Berkeley_Packet_Filter [1] https://lwn.net/Articles/740157/

Brendan Gregg is the author of most of the utilities, not bpf/ebpf itself. You can find a non-exhaustive list here: http://www.brendangregg.com/ebpf.html

Re: How to add eBPF observability to your product

#15

Post by boomskats » Sat, Jul 03, 2021, 9:57 PM UTC

I've been on and off playing with the same scripts described in this article for the last few months (installable as bcc-tools under Fedora). While some of them are a little unstable or need a bit of Python version handholding in fc34, I've been blown away with the data I've been able to get with seemingly negligible overhead. I was also really surprised to find them in the default repos, and how relatively few people reference them when discussing other eBPF tooling (e.g. Pixie).

As someone who has been building observability software for the last few years I'm ridiculously excited about eBPF. Looking forward to seeing what the Netflix dashboards mentioned in the post look like & the data pipelines that support them.

Re: How to add eBPF observability to your product

#16

Post by bch » Sat, Jul 03, 2021, 10:32 PM UTC

Ristovski wrote: post #14

bch wrote: post #11

Earlier quoted context omitted.

Maybe I’m misunderstanding what you mean by “author“, but BPF was invented by Steven McCanne, and Van Jacobson in 1992[0], and tweaked over the years by various developers and operating systems. It looks[1] like the Linux e BPF work was initiated by Alexei Starovoitov, and came to be exposed to users c. 2014. [0] https://en.wikipedia.org/wiki/Berkeley_Packet_Filter [1] https://lwn.net/Articles/740157/

Brendan Gregg is the author of most of the utilities, not bpf/ebpf itself. You can find a non-exhaustive list here: http://www.brendangregg.com/ebpf.html

I get that, and appreciate he’s productive and sharing his skill. I wasn’t sure though if the long history of other people’s hard work wasn’t being swept up in appreciation for ‘brendangregg

Re: How to add eBPF observability to your product

#17

Post by brendangregg » Sat, Jul 03, 2021, 11:04 PM UTC

hn_throwaway_99 wrote: post #8

yjftsjthsd-h wrote: post #3

Earlier quoted context omitted.

Technically, yes, but in practice not really; eBPF started as an extension of the packet filtering framework and then got generalized to be a nearly-universal debugging/monitoring/tracing system that's only sometimes used for packets. https://ebpf.io/ is a decent starting point for reading.

My biggest pet peeve is when authors don't just start with a 1-2 sentence outline of the topic that they're discussing, and instead go directly into acronym soup. And I've been in software development for a long time and have never heard of BPF/eBPF before. Thank you for your explanation.

Sorry, I just added a link to https://ebpf.io. Although I suspect that will also annoy people as it doesn't explain the BPF acronym: The reason is that we're now treating BPF as a technology name (like LLVM) and not an acronym, as the historical acronym (Berkeley Packet Filter) is plain confusing.

I think the real problem is that BPF was built by engineers without professional marketing help, who would have come up with a better name!

Re: How to add eBPF observability to your product

#18

Post by brendangregg » Sat, Jul 03, 2021, 11:09 PM UTC

bch wrote: post #11

enedil wrote: post #9

Earlier quoted context omitted.

Brendan Gregg is the author of eBPF, and he writes on his blog. It is fair to assume a basic knowledge of his works when reading his website.

Maybe I’m misunderstanding what you mean by “author“, but BPF was invented by Steven McCanne, and Van Jacobson in 1992[0], and tweaked over the years by various developers and operating systems. It looks[1] like the Linux e BPF work was initiated by Alexei Starovoitov, and came to be exposed to users c. 2014. [0] https://en.wikipedia.org/wiki/Berkeley_Packet_Filter [1] https://lwn.net/Articles/740157/

Thanks; I also summarize the history of eBPF in my BPF Performance Tools book (which I had checked by Alexei & Dan) and list the main contributors in the Acknowledgements page, as well as include them in talks (e.g. https://www.slideshare.net/brendangregg/bpf-internals-ebpf/1...). I do try hard to thank all involved. :-)

Modern BPF (eBPF) was created by Alexei Starovoitov and Daniel Borkmann, who are still maintainers but are joined by over a hundred contributors. I've spent most of my time contributing to the observability frontends, bcc and bpftrace. Apart from the people, the major companies contributing include Facebook and Isovalent; Microsoft are becoming its own major contributor to its own Windows implementation.

Re: How to add eBPF observability to your product

#19

Post by scottlamb » Sat, Jul 03, 2021, 11:14 PM UTC

nyellin wrote: post #12

This works great if you control the servers or have a uniform fleet. I used to work at a security vendor which sold ebpf based software that ran as a daemonset on customer kubernetes clusters. Compiling ebpf bytecode on the customer's hosts wasn't an option. Iirc the real challenge was with writing kprobe ebpf functions that access native structs. I don't think we ever found a good solution for that because you need…

I think that's what BTF and CO-RE are for. They are new; they may not have existed when you had this problem. https://brendangregg.com/blog/2020-11-04/bpf-co-re-btf-libbp...

Re: How to add eBPF observability to your product

#20

Post by fazzone » Sat, Jul 03, 2021, 11:30 PM UTC

nyellin wrote: post #12

This works great if you control the servers or have a uniform fleet. I used to work at a security vendor which sold ebpf based software that ran as a daemonset on customer kubernetes clusters. Compiling ebpf bytecode on the customer's hosts wasn't an option. Iirc the real challenge was with writing kprobe ebpf functions that access native structs. I don't think we ever found a good solution for that because you need…

The Linux kernel can be configured at build time to expose its own headers as /proc/kheaders.tar.xz. Not sure if distribution kernels generally have this option enabled or not.