A kernel bug froze my machine: Debugging an async-profiler deadlock
1–10 of 19 posts
Re: A kernel bug froze my machine: Debugging an async-profiler deadlock
#2Author here. I've always been kernel-curious despite never having worked on one myself. Consider this either a collection of impractical party tricks or a hands-on way to get a feel for kernel internals.
Re: A kernel bug froze my machine: Debugging an async-profiler deadlock
#3Great article! Just yesterday I watched a Devoxx talk by Andrei Pangin [1], the creator of async-profiler where I learned about the new heatmap support. To many folks it might not sound that exciting, until you realise that these heatmaps make it much easier to see patterns over time. If you’re interested there’s a solid blog post [2] from Netflix that walks through the format and why it can be incredibly useful.
[1]: https://www.youtube.com/watch?v=u7-S-Hn-7Do
[2]: https://netflixtechblog.com/netflix-flamescope-a57ca19d47bb
Re: A kernel bug froze my machine: Debugging an async-profiler deadlock
#4Great article! Just yesterday I watched a Devoxx talk by Andrei Pangin [1], the creator of async-profiler where I learned about the new heatmap support. To many folks it might not sound that exciting, until you realise that these heatmaps make it much easier to see patterns over time. If you’re interested there’s a solid blog post [2] from Netflix that walks through the format and why it can be incredibly useful. [1]…
Thanks for the kind words!
Heatmaps are amazing for pattern spotting. I also use them when hunting irregular hiccups or outliers. More people should know about this feature.
Re: A kernel bug froze my machine: Debugging an async-profiler deadlock
#5Question, isn't this a bug?
static enum hrtimer_restart perf_swevent_hrtimer(struct hrtimer *hrtimer)
{
- if (event->state != PERF_EVENT_STATE_ACTIVE)
+ if (event->state != PERF_EVENT_STATE_ACTIVE ||
+ event->hw.state & PERF_HES_STOPPED)
return HRTIMER_NORESTART;
The bug being that the precedence of || is higher than the precedence of != ? Consider writing it if ((event->state != PERF_EVENT_STATE_ACTIVE) || (event->hw_state & PERF_HES_STOPPED))
This coming from a person who has too many scars from not parenthesizing my expressions in conditionals to ensure they work the way I meant them to work.
Re: A kernel bug froze my machine: Debugging an async-profiler deadlock
#6Question, isn't this a bug? static enum hrtimer_restart perf_swevent_hrtimer(struct hrtimer *hrtimer) { - if (event->state != PERF_EVENT_STATE_ACTIVE) + if (event->state != PERF_EVENT_STATE_ACTIVE || + event->hw.state & PERF_HES_STOPPED) return HRTIMER_NORESTART; The bug being that the precedence of || is higher than the precedence of != ? Consider writing it if ((event->state != PERF_EVENT_STATE_ACTIVE) || (event->h…
Wow, someone is actually reading the article in detail, that's a good feeling!
In C, the != operator has higher precedence than the || operator. That said, extra parentheses never hurt readability.
Re: A kernel bug froze my machine: Debugging an async-profiler deadlock
#7Great debugging effort.
Now, with the complexity (MLoCs!) of the Linux kernel, this is definitely not the only bug to be found in there.
This is why Linux is just an interim kernel for these use cases in which we still cannot use seL4[0].
Re: A kernel bug froze my machine: Debugging an async-profiler deadlock
#8Question, isn't this a bug? static enum hrtimer_restart perf_swevent_hrtimer(struct hrtimer *hrtimer) { - if (event->state != PERF_EVENT_STATE_ACTIVE) + if (event->state != PERF_EVENT_STATE_ACTIVE || + event->hw.state & PERF_HES_STOPPED) return HRTIMER_NORESTART; The bug being that the precedence of || is higher than the precedence of != ? Consider writing it if ((event->state != PERF_EVENT_STATE_ACTIVE) || (event->h…
Which language(s?) have || before !=/==?
Re: A kernel bug froze my machine: Debugging an async-profiler deadlock
#9I'm glad to hear I'm not alone. Due to the nature of what I do, I'm often accumulating ~800-900GB of Docker images and volumes on my machine, sometimes running 20-30 containers at once starting/stopping them concurrently. Somehow, very rarely, but still quite often (once every couple of weeks) - it leads to a complete deadlock somewhere inside of the kernel due to some crazy race condition that I'm absolutely in no way able to reliably reproduce.
Re: A kernel bug froze my machine: Debugging an async-profiler deadlock
#10I'm glad to hear I'm not alone. Due to the nature of what I do, I'm often accumulating ~800-900GB of Docker images and volumes on my machine, sometimes running 20-30 containers at once starting/stopping them concurrently. Somehow, very rarely, but still quite often (once every couple of weeks) - it leads to a complete deadlock somewhere inside of the kernel due to some crazy race condition that I'm absolutely in no w…
It's much tougher when it's so hard to reproduce. Perhaps the NMI watchdog could help? https://docs.kernel.org/admin-guide/lockup-watchdogs.html