Excellent article! I learned a lot from it. Thanks!
A tale about fixing eBPF spinlock issues in the Linux kernel
21–27 of 27 posts
Re: A tale about fixing eBPF spinlock issues in the Linux kernel
#22Thanks for sharing. Ebpf is so powerful.
Re: A tale about fixing eBPF spinlock issues in the Linux kernel
#23How is this a kernel issue? The code that deadlocked was entirely written by Superluminal who grabbed a shared lock from a interrupt handler. Not doing that is literally the very first lesson of writing interrupt handlers and if you do not know that you have no business doing so. The only way this could be considered a issue is that it appears that the Linux kernel added the rqspinlock which is supposed to automatica…
Leaving aside the vitriol... > The code that deadlocked was entirely written by Superluminal who grabbed a shared lock from a interrupt handler We don't "grab a shared lock". We call a kernel-provided eBPF helper function `bpf_ringbuf_reserve`, which, we now know, internally grabs a lock. The spinlock usage is entirely internal to the eBPF ringbuffer implementation and is not exposed to or controlled by the eBPF prog…
One thing wasn't clear for me in the article: is there only one such ringbuffer defined by the kernel or can the eBPF program specify as many ringbuffers as it wants?
Re: A tale about fixing eBPF spinlock issues in the Linux kernel
#24Very cool and technical
Re: A tale about fixing eBPF spinlock issues in the Linux kernel
#25Earlier quoted context omitted.
Leaving aside the vitriol... > The code that deadlocked was entirely written by Superluminal who grabbed a shared lock from a interrupt handler We don't "grab a shared lock". We call a kernel-provided eBPF helper function `bpf_ringbuf_reserve`, which, we now know, internally grabs a lock. The spinlock usage is entirely internal to the eBPF ringbuffer implementation and is not exposed to or controlled by the eBPF prog…
Geez, your company really needs to not be writing code in interrupt context until you learn how it works. bpf_ringbuf_reserve() is perfectly fine to call from interrupt context. The problem is that you are manipulating the same data structure from non-interrupt and interrupt context. Your code was deadlocking with itself. You wrote every side of that deadlock. For that matter, how are you even handling the deadlock d…
Not OP: how would you handle the second interrupt during the interrupt handler here then? I can see how you could use two separate ring buffers for different contexts, but I don't see how to handle the nested interrupt. Also indeed they just drop all these samples that get deadlocked.
Actually, as long as you use different ring buffers for interrupt/non-interrupt context, it should be fine to just drop if you encounter a deadlock due to interrupting an already running interrupt handler.
Re: A tale about fixing eBPF spinlock issues in the Linux kernel
#26Earlier quoted context omitted.
Geez, your company really needs to not be writing code in interrupt context until you learn how it works. bpf_ringbuf_reserve() is perfectly fine to call from interrupt context. The problem is that you are manipulating the same data structure from non-interrupt and interrupt context. Your code was deadlocking with itself. You wrote every side of that deadlock. For that matter, how are you even handling the deadlock d…
Not OP: how would you handle the second interrupt during the interrupt handler here then? I can see how you could use two separate ring buffers for different contexts, but I don't see how to handle the nested interrupt. Also indeed they just drop all these samples that get deadlocked. Actually, as long as you use different ring buffers for interrupt/non-interrupt context, it should be fine to just drop if you encount…
The code described is not nested interrupt handlers. It is eBPF code executing during a context switch which is interrupted by the sampling NMI which is also configured to execute eBPF code.
NMIs will not nest, so there is no risk of arbitrary nesting. So, there should be at most three nesting levels: regular, interrupt (I suspect they do not do logging during interrupts so this may not even exist in their use case), non-maskable interrupt.
Off the top of my head I can think of at least 5 unique ways to not drop the sample with your idea of separate ring buffers being one of them.
Re: A tale about fixing eBPF spinlock issues in the Linux kernel
#27Earlier quoted context omitted.
Not OP: how would you handle the second interrupt during the interrupt handler here then? I can see how you could use two separate ring buffers for different contexts, but I don't see how to handle the nested interrupt. Also indeed they just drop all these samples that get deadlocked. Actually, as long as you use different ring buffers for interrupt/non-interrupt context, it should be fine to just drop if you encount…
The code described is not nested interrupt handlers. It is eBPF code executing during a context switch which is interrupted by the sampling NMI which is also configured to execute eBPF code. NMIs will not nest, so there is no risk of arbitrary nesting. So, there should be at most three nesting levels: regular, interrupt (I suspect they do not do logging during interrupts so this may not even exist in their use case),…
Indeed I misread. I figured the NMI must have been nested or otherwise they wouldn't have gone through all this trouble just to drop samples :)