Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to add eBPF observability to your product

brendangregg.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: How to add eBPF observability to your product

#21

Post by brendangregg »

scottlamb wrote: post #19
nyellin wrote: post #12

This works great if you control the servers or have a uniform fleet. I used to work at a security vendor which sold ebpf based software that ran as a daemonset on customer kubernetes clusters. Compiling ebpf bytecode on the customer's hosts wasn't an option. Iirc the real challenge was with writing kprobe ebpf functions that access native structs. I don't think we ever found a good solution for that because you need…

I think that's what BTF and CO-RE are for. They are new; they may not have existed when you had this problem. https://brendangregg.com/blog/2020-11-04/bpf-co-re-btf-libbp...

Yes, BTF provides the kernel struct info in the vmlinux file. Like debuginfo, but far more lightweight.

Of course, tracepoints are better, as they are stable and don't need BTF (although raw_tracepoints do), but there's often cases where they don't exist for the thing I am tracing (and arguably shouldn't exist, if it's too niche/hot-path).

Re: How to add eBPF observability to your product

#22

Post by soheil »

BPF: "The Berkeley Packet Filter is a technology used in certain computer operating systems for programs that need to, among other things, analyze network traffic. It provides a raw interface to data link layers, permitting raw link-layer packets to be sent and received." -Wikipedia

Re: How to add eBPF observability to your product

#23

Post by brendangregg »

soheil wrote: post #22

BPF: "The Berkeley Packet Filter is a technology used in certain computer operating systems for programs that need to, among other things, analyze network traffic. It provides a raw interface to data link layers, permitting raw link-layer packets to be sent and received." -Wikipedia

Unfortunately that description is so old it's misleading. I described BPF in another comment here.

Re: How to add eBPF observability to your product

#24

Post by quickthrower2 »

hn_throwaway_99 wrote: post #8
yjftsjthsd-h wrote: post #3

Earlier quoted context omitted.

Technically, yes, but in practice not really; eBPF started as an extension of the packet filtering framework and then got generalized to be a nearly-universal debugging/monitoring/tracing system that's only sometimes used for packets. https://ebpf.io/ is a decent starting point for reading.

My biggest pet peeve is when authors don't just start with a 1-2 sentence outline of the topic that they're discussing, and instead go directly into acronym soup. And I've been in software development for a long time and have never heard of BPF/eBPF before. Thank you for your explanation.

He did link to something, but for me that link was broken (https://brendangregg.com/blog/2021-07-03/BPF)

Re: How to add eBPF observability to your product

#25

Post by hn_throwaway_99 »

brendangregg wrote: post #17
hn_throwaway_99 wrote: post #8

Earlier quoted context omitted.

My biggest pet peeve is when authors don't just start with a 1-2 sentence outline of the topic that they're discussing, and instead go directly into acronym soup. And I've been in software development for a long time and have never heard of BPF/eBPF before. Thank you for your explanation.

Sorry, I just added a link to https://ebpf.io . Although I suspect that will also annoy people as it doesn't explain the BPF acronym: The reason is that we're now treating BPF as a technology name (like LLVM) and not an acronym, as the historical acronym (Berkeley Packet Filter) is plain confusing. I think the real problem is that BPF was built by engineers without professional marketing help, who would have come up…

Thank you!

Re: How to add eBPF observability to your product

#26

Post by kbenson »

hn_throwaway_99 wrote: post #8
yjftsjthsd-h wrote: post #3

Earlier quoted context omitted.

Technically, yes, but in practice not really; eBPF started as an extension of the packet filtering framework and then got generalized to be a nearly-universal debugging/monitoring/tracing system that's only sometimes used for packets. https://ebpf.io/ is a decent starting point for reading.

My biggest pet peeve is when authors don't just start with a 1-2 sentence outline of the topic that they're discussing, and instead go directly into acronym soup. And I've been in software development for a long time and have never heard of BPF/eBPF before. Thank you for your explanation.

That's more because you're coming into an established site with an established topic that's been discussed for years. It doesn't really make sense to introduce every topic from scratch every time, because the assumption after a while is that the people reading have at least some prior knowledge or at least an interest in looking up what they don't know because they're there for a reason.

In other words, not every site is designed around link aggregators like HN, and some things will be assumed, just as if you open a book to some random chapter.

Re: How to add eBPF observability to your product

#27

Post by nyellin »

scottlamb wrote: post #19
nyellin wrote: post #12

This works great if you control the servers or have a uniform fleet. I used to work at a security vendor which sold ebpf based software that ran as a daemonset on customer kubernetes clusters. Compiling ebpf bytecode on the customer's hosts wasn't an option. Iirc the real challenge was with writing kprobe ebpf functions that access native structs. I don't think we ever found a good solution for that because you need…

I think that's what BTF and CO-RE are for. They are new; they may not have existed when you had this problem. https://brendangregg.com/blog/2020-11-04/bpf-co-re-btf-libbp...

Yeah, I'm familiar with them. Once they're widespread this will be a whole different story. (Here too, if you control the fleet of servers then this is easy. If you're deploying software as a vendor you're stuck with older kernel versions.)

Re: How to add eBPF observability to your product

#28

Post by enedil »

bch wrote: post #11
enedil wrote: post #9

Earlier quoted context omitted.

Brendan Gregg is the author of eBPF, and he writes on his blog. It is fair to assume a basic knowledge of his works when reading his website.

Maybe I’m misunderstanding what you mean by “author“, but BPF was invented by Steven McCanne, and Van Jacobson in 1992[0], and tweaked over the years by various developers and operating systems. It looks[1] like the Linux e BPF work was initiated by Alexei Starovoitov, and came to be exposed to users c. 2014. [0] https://en.wikipedia.org/wiki/Berkeley_Packet_Filter [1] https://lwn.net/Articles/740157/

I'm sorry, this indeed isn't correct. I cannot modify my comment for some reason now.

Re: How to add eBPF observability to your product

#29

Post by nthngtshr »

I'm not an expert in this field so maybe I'm missing some important context here, but this idea of preventing people from porting your software seems odd. Who cares if people make shitty ports? If they're not good then no one will use them. The whole thing also reads a little bit condescending.

Any time there's some popular public API, people will copy other people's projects built around this API and make them "better". I don't think Daniel Stenberg (author of cURL) is upset that there's HTTPie. I don't think it hurts cURL and I don't think it hurts the developer community as a whole.

I guess one could argue it would make life easier if there was just one HTTP library (or one eBPF library I guess). But that's just unrealistic, that's not how people work. Plus it removes the competition aspect which is nice to have for any project.

Is there any examples of open source projects that died because people were making too many ports?

Re: How to add eBPF observability to your product

#30

Post by Bayart »

brendangregg wrote: post #18
bch wrote: post #11

Earlier quoted context omitted.

Maybe I’m misunderstanding what you mean by “author“, but BPF was invented by Steven McCanne, and Van Jacobson in 1992[0], and tweaked over the years by various developers and operating systems. It looks[1] like the Linux e BPF work was initiated by Alexei Starovoitov, and came to be exposed to users c. 2014. [0] https://en.wikipedia.org/wiki/Berkeley_Packet_Filter [1] https://lwn.net/Articles/740157/

Thanks; I also summarize the history of eBPF in my BPF Performance Tools book (which I had checked by Alexei & Dan) and list the main contributors in the Acknowledgements page, as well as include them in talks (e.g. https://www.slideshare.net/brendangregg/bpf-internals-ebpf/1... ). I do try hard to thank all involved. :-) Modern BPF (eBPF) was created by Alexei Starovoitov and Daniel Borkmann, who are still maintaine…

>Microsoft are becoming its own major contributor to its own Windows implementation.

Wait, what now !?

After checking : he wasn't joking[0]. Good job MS.

[0]:https://github.com/microsoft/ebpf-for-windows

Post reply on HN