Bpftrace for Linux 2018
brendangregg.com
Bpftrace for Linux 2018
1–10 of 24 posts
Re: Bpftrace for Linux 2018
#2[deleted]
Re: Bpftrace for Linux 2018
#3Hi Brendan, thanks for the update and the work on this! Coming from dtrace, the main thing missing in bpftrace when doing kernel debugging is being able to navigate structs.
You've mentioned that we do have BTF now in Linux 4.18. I've tried to find if it was leveraged in bpftrace, but it looks like it isn't yet. I see you've opened an issue in one of the bpftrace forks related to this here: https://github.com/ajor/bpftrace/issues/30.
Do you know if there's been any update on this? Support for BTF/CTF would be amazing.
Re: Bpftrace for Linux 2018
#4Hi Brendan, thanks for the update and the work on this! Coming from dtrace, the main thing missing in bpftrace when doing kernel debugging is being able to navigate structs. You've mentioned that we do have BTF now in Linux 4.18. I've tried to find if it was leveraged in bpftrace, but it looks like it isn't yet. I see you've opened an issue in one of the bpftrace forks related to this here: https://github.com/ajor/bp…
That's the old repo (we should add a note to it pointing people to https://github.com/iovisor/bpftrace instead!)
Alastair added struct support for kprobes yesterday, based on the functionality in bcc (which bpftrace uses). That was the final missing piece, and why I'm posting about it now. See the last example here:
https://github.com/iovisor/bpftrace/blob/master/docs/referen...
It's the same struct support we had in bcc, using linux kernel headers. Some things are available, some are not. Just like in bcc, we sometimes need to define some of a missing struct.
BTF will mean that all structs are available, plus, bpftrace should be able to automatically figure out the arguments so you don't have to cast them. It'll be great, but in the meantime, I am able to do tons of stuff.
Re: Bpftrace for Linux 2018
#5The 2 major problems (ie the Elefant in the room) compared to DTrace are missing:
1. security: eBPF still supports arrays and and is therefore inherently insecure to be used in the kernel. DTrace didn't support that feature on purpose and not because they couldn't do it. see spectre/meltdown.
2. availability: eBPF probes only work within the kernel. DTrace extends the same interface to the libc level and the user space level. you can trace not only kernel calls, also lib calls and with its high-level language support (Java, python, perl, ruby,...) into userspace. Linux-only eBPF will never be able to support that seemless integration. thanksfully we now have Oracle Linux with proper DTrace support on Linux.
Re: Bpftrace for Linux 2018
#6Seeing this come together makes me super happy.
Now the next time the hot rage of an audio xrun overcomes me I will have a path forward!
In general audio xruns actually is an area that could use a permanent userspace program.
Musicians need low latency audio processing which Jack2 can provide but the random xrun and the internet flimflam/mummery of how to sort it out is waaaaay beyond most of us (I have less of an excuse but yah its a vibe killer when you are trying to make music).
The goal would be "hey you had a/some xrun and its might be mostly this disk or process or..."
Maybe this needs hard ai but maybe it just needs a compass like tool...
Re: Bpftrace for Linux 2018
#7Hi Brendan, thanks for the update and the work on this! Coming from dtrace, the main thing missing in bpftrace when doing kernel debugging is being able to navigate structs. You've mentioned that we do have BTF now in Linux 4.18. I've tried to find if it was leveraged in bpftrace, but it looks like it isn't yet. I see you've opened an issue in one of the bpftrace forks related to this here: https://github.com/ajor/bp…
That's the old repo (we should add a note to it pointing people to https://github.com/iovisor/bpftrace instead!) Alastair added struct support for kprobes yesterday, based on the functionality in bcc (which bpftrace uses). That was the final missing piece, and why I'm posting about it now. See the last example here: https://github.com/iovisor/bpftrace/blob/master/docs/referen... It's the same struct support we had in…
Fair enough. Having to include headers is one thing I found tricky with BCC, as there's sometimes non-trivial dependencies between various header files, especially for kernel modules that aren't included in the mainline kernel. That said I'm very excited to see all the progress that's been done in bpftrace, and looking forward to what's to come.
Re: Bpftrace for Linux 2018
#8The 2 major problems (ie the Elefant in the room) compared to DTrace are missing: 1. security: eBPF still supports arrays and and is therefore inherently insecure to be used in the kernel. DTrace didn't support that feature on purpose and not because they couldn't do it. see spectre/meltdown. 2. availability: eBPF probes only work within the kernel. DTrace extends the same interface to the libc level and the user spa…
1. I'd say that the other way around: eBPF has already been hardened in a post spectre/meltdown world (see lkml), and I'm not sure DTrace has had the same level of security attention.
2. That's inaccurate, see the bpftrace docs for uprobes and usdt.
Re: Bpftrace for Linux 2018
#9The 2 major problems (ie the Elefant in the room) compared to DTrace are missing: 1. security: eBPF still supports arrays and and is therefore inherently insecure to be used in the kernel. DTrace didn't support that feature on purpose and not because they couldn't do it. see spectre/meltdown. 2. availability: eBPF probes only work within the kernel. DTrace extends the same interface to the libc level and the user spa…
Hmm, looks like you are missing some crucial details, eBPF has the same mitigations to the CPU bugs in place as the rest of the kernel (e.g. retpolines on tail calls, index masking for arrays, sanitation of speculative store bypass, etc). Also, eBPF probes do not only work in the kernel but user space as well via uprobes as one example.
Re: Bpftrace for Linux 2018
#10This is huge. The Linux community should be glad that Netflix hired Brendan Gregg to work on Linux performance.