Live data from Hacker News

A one in a million bug in Switch kernel

gist.githubusercontent.com

41–50 of 95 posts

Re: A one in a million bug in Switch kernel

#41
post #27
post #25

Earlier quoted context omitted.

Even low-probability bugs will surface often enough if you give it enough potential times to do so. There are >100 Mn Switch'es out there, and the interrupts happens at least tens to hundreds of times a second when in use, so plenty of opportunities :)

Yep but can they reproduce it? When we say "low probability" we're acting like it's truly random, but in reality they could have stumbled across steps that reproduce it very frequently.

You don't necessarily need to reproduce this one, once noticed you can work through on paper where the timing problems could lie.

Noticing the issue in the first place is the big problem, as TFA says the side effects of this are likely graphical glitches that would not stop any shows and just get marked down as a hardware timing issue¹. Once noticed by someone with the skills to notice it while looking at the code for other reasons, it is obvious².

Proving the fix fully resolves the matter without adding others could be a detailed task, perhaps with multiple skilled people passing their eye over the result to try ensure there isn't another “oh, interesting” moment³ waiting to happen, but again doesn't necessarily mean needing to reliably simulate the exact situation.

[1] pick up another couple of devices, yep confirmed, it doesn't happen on these

[2] well, obvious to that someone with that skill, I'll not claim it is as obvious to such as myself!

[3] like one that may have been how this was found, assuming it was spotted in passing while working on that area for other reasons

Re: A one in a million bug in Switch kernel

#42
What might be a few simple hello world projects to begin a journey understanding how to debug something like this? I suppose understanding of OS is important along with assembly. Given basic knowledge here, could someone list a few lessons to try and any toolsets?

Re: A one in a million bug in Switch kernel

#44

offtopic: I really wish reading preformatted text files on ios safari was good.. I have to export the file to Books in order to read it properly

I think people need to use text files less. Or at least stop hard-wapping them.

Hard-wrapping is my bane. Especially in RFCs.

Re: A one in a million bug in Switch kernel

#45

What might be a few simple hello world projects to begin a journey understanding how to debug something like this? I suppose understanding of OS is important along with assembly. Given basic knowledge here, could someone list a few lessons to try and any toolsets?

Write a toy operating system: https://wiki.osdev.org/Main_Page

For example, start with https://wiki.osdev.org/Bare_Bones or https://wiki.osdev.org/Raspberry_Pi_Bare_Bones

You'll never build anything practical, but it's a great way to learn thing that you'd rarely have the opportunity to learn otherwise. Armed with that wide but shallow knowledge, you'll suddenly see many new opportunities to learn / do things that you wouldn't even have thought of before.

Re: A one in a million bug in Switch kernel

#46
First of all, it is amazing that the author managed to analyze the patch in so much details, it probably is an effort comparable to the bug fix itself. Still I think the article is missing some bits. I would expect any core migration to require barriers (either implicit or explicit) on both the old and new core otherwise the process would risk seeing its own stores and loads out of order.

But in this case the barrier is predicated on the execution of some cache manipulation instruction, so I suspect things are more complicated. Maybe these specific cache manipulation instructions do not respect the usual architectural memory ordering and require some different set of barriers. Possibly they bypass cache coherence completely and require an actual flush of the cache. That is going to be very expensive and it make sense that it is only done only if the process was actually fiddling with these instructions. 'jmgao' else thread reported that tegra has coherency issues on migration, so it might be related.

Re: A one in a million bug in Switch kernel

#47
post #17
post #11

Earlier quoted context omitted.

FWIW: I re-read this a bunch of times, and I don't understand how this isn't a hardware bug. How can an asynchronous interrupt be specified in any rigorous way if it does NOT act as a memory barrier to the interrupted code? Clearly the CPU isn't going to cache its in-flight state for every interrupt (and remember interrupts can be themselves interrupted!). So certainly "most" of its state is being serialized. And we'…

Knowing the Tegra chip in question, I'd bet it's probably not ARM's fault. Tegra X1, unlike pretty much every other SoC, did big.LITTLE via cluster migration with a custom cache coherence system, instead of just having a bunch of heterogenous cores. It turns out that their custom cache coherence was unfixably broken and would randomly corrupt memory when doing migration between the big and little cores, so everyone w…

You are most likely right. I would expect that barriers would be needed unconditionally. This patch is likely working around some hole in the SoC where the usual barrier is not sufficient for some instructions and some additional ritual need to be performed.

Re: A one in a million bug in Switch kernel

#48

Reminds me of a similar bug that I worked on a few years ago that led to my single-line contribution to xnu (apologies for the dissertation): We had increasing reports of devices panicking because the kernel stopped draining a buffer, causing the buffer to fill. This particular buffer should never fill, so if it does -> panic. The first problem was that this bug was getting 'hot'. The bug needed to be fixed yesterday…

> To this day I'm still mystified as to why this bug hadn't caused broken interrupts (+ mysterious behavior) or mass panics before then. There must've been some other change to xnu that exposed the bug somehow, but I'll probably never know.

My favourite species of software bugs are the ones that when you find the source you realise the code is fundamentally broken, and you get to investigate how the hell it worked for so long!

Re: A one in a million bug in Switch kernel

#49

> Makes you think, do Linux, Windows and Mac handle this properly? Honestly, I doubt it! Rubbish. These kernels (well Linux and Windows) run on systems with hundreds even thousands of cores, on CPUs which are very weakly ordered, with a pretty reasonable level of reliability. A race like this will blow up immediately. Linux handles this by requiring that a context switch operation includes a full memory barrier so sw…

Note that this specific bug seem to happen only after cache operations (i.e. something like CLFLUSH, CLZERO in x86 parlance). It is possible that these instructions on the Switch SoC require a different barrier either because of spec details or hardware bugs.
Post reply on HN