Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one in a million bug in Switch kernel

gist.githubusercontent.com

Post reply on HN
Print view View original

51–60 of 95 posts

Re: A one in a million bug in Switch kernel

#51

Post by throwawaylinux »

ajross wrote: post #11
broodbucket wrote: post #7

>Makes you think, do Linux, Windows and Mac handle this properly? Honestly, I doubt it! Context switches, idle state transitions, etc tend to be fairly delicately handled as a common cause of CVEs and Heisenbugs. I'm sure there's still plenty of bugs but more attention ends up being paid to these things on general purpose operating systems. More eyeballs on the code, more security researchers, more hardware variants…

FWIW: I re-read this a bunch of times, and I don't understand how this isn't a hardware bug. How can an asynchronous interrupt be specified in any rigorous way if it does NOT act as a memory barrier to the interrupted code? Clearly the CPU isn't going to cache its in-flight state for every interrupt (and remember interrupts can be themselves interrupted!). So certainly "most" of its state is being serialized. And we'…

> FWIW: I re-read this a bunch of times, and I don't understand how this isn't a hardware bug. How can an asynchronous interrupt be specified in any rigorous way if it does NOT act as a memory barrier to the interrupted code? Clearly the CPU isn't going to cache its in-flight state for every interrupt (and remember interrupts can be themselves interrupted!). So certainly "most" of its state is being serialized. And we're supposed to magically guess on a per-IP basis which state isn't? Yikes.

Memory barriers only concern interactions with other agents that access memory.

Any given thread of execution is always consistent with respect to itself, including when taking interrupts.

"Serialized" is also not the same as a barrier and is not really related to memory consistency. Serialization does only matter within a single thread of execution.

Re: A one in a million bug in Switch kernel

#52

Post by klauspost »

broodbucket wrote: post #7

>Makes you think, do Linux, Windows and Mac handle this properly? Honestly, I doubt it! Context switches, idle state transitions, etc tend to be fairly delicately handled as a common cause of CVEs and Heisenbugs. I'm sure there's still plenty of bugs but more attention ends up being paid to these things on general purpose operating systems. More eyeballs on the code, more security researchers, more hardware variants…

x86 Darwin had a bug where AVX512 K (opmask) registers would not be saved and restored on context switches when YMM registers had all 0 values. The kernel assumed that AVX512 wasn't used. They fixed it in Monterey 12.2, but still hasn't backported. Heisenbug for sure.

Ref: https://github.com/golang/go/issues/49233#issuecomment-96373...

Re: A one in a million bug in Switch kernel

#53

Post by svenpeter »

broodbucket wrote: post #7

>Makes you think, do Linux, Windows and Mac handle this properly? Honestly, I doubt it! Context switches, idle state transitions, etc tend to be fairly delicately handled as a common cause of CVEs and Heisenbugs. I'm sure there's still plenty of bugs but more attention ends up being paid to these things on general purpose operating systems. More eyeballs on the code, more security researchers, more hardware variants…

Will Deacon replied on Twitter how at least Linux handles this correctly [1][2] and I assume that the same is true for XNU and Windows as well.

[1] https://twitter.com/WillDeacon/status/1506375874161086471

[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Re: A one in a million bug in Switch kernel

#54

Post by Someone »

ajross wrote: post #11
broodbucket wrote: post #7

>Makes you think, do Linux, Windows and Mac handle this properly? Honestly, I doubt it! Context switches, idle state transitions, etc tend to be fairly delicately handled as a common cause of CVEs and Heisenbugs. I'm sure there's still plenty of bugs but more attention ends up being paid to these things on general purpose operating systems. More eyeballs on the code, more security researchers, more hardware variants…

FWIW: I re-read this a bunch of times, and I don't understand how this isn't a hardware bug. How can an asynchronous interrupt be specified in any rigorous way if it does NOT act as a memory barrier to the interrupted code? Clearly the CPU isn't going to cache its in-flight state for every interrupt (and remember interrupts can be themselves interrupted!). So certainly "most" of its state is being serialized. And we'…

> I don't understand how this isn't a hardware bug

From what I understand, if the chip’s documentation would say “all interrupt handlers must start with a memory barrier”, this would be a software bug.

Isn’t it the case that a hardware bug for which a workaround is documented before shipping is ‘just’ a misfeature? (In this case, supporting user-supplied interrupt handlers would be a bit complicated. When it gets installed, you’d have to check their first instruction after first making its memory page non-writable by user code)

Back to the workaround: they seem confident that this only is a problem when doing “user-mode cache operations (flush / clean / zero)”, and those, apparently, can all be fixed to set that TLS flag. If I were trying to break into this system, I would look at both assumptions.

In particular, can you clear that secret byte directly after the kernel set it, and get the old behavior back? Worse, does “user-mode cache operations” imply those are completely run in user mode (since they can make this fix, presumably using a library provided by Nintendo)? If so, what prevents you from using your own cache flush code that doesn’t set the flag?

Re: A one in a million bug in Switch kernel

#55

Post by fps-hero »

throwawaybyeeee wrote: post #35

Reminds me of a similar bug that I worked on a few years ago that led to my single-line contribution to xnu (apologies for the dissertation): We had increasing reports of devices panicking because the kernel stopped draining a buffer, causing the buffer to fill. This particular buffer should never fill, so if it does -> panic. The first problem was that this bug was getting 'hot'. The bug needed to be fixed yesterday…

Thank you for sharing, gosh that must have been horrific to track down.

There really is something about USB buses that cause the worst kinds of errors. I happen to know that the USB2 driver in the RPI 1/2/3/4 has a Linux kernel corruption bug which is completely masked by the use of a USB hub.

Why does it matter? Because the RPI 1, 2, 3 all hide the USB port behind a hub. Only the zero, and the a series have a naked port. Now, try searching the RPI forum for USB problems and start to notice a correlation.

The problem is that the hive mind decided that all USB errors must be power related, and given the complete dodgeyness of most RPI zero setups it was always assumed this was the culprit.

Unfortunately it isn’t. No amount of probing, decoupling, externally powering ever fixed the glitches, ah but yes, not using an official 3A RPI branded psi was definitely the issue, sorry Agilent your psu’s just aren’t up to task, probably the reason you had to “rebrand” in the first place. :S

We ended up retrofiting a USB hub in-line with a USB connector for a prototype, and we’ve since designed in the hub just for that one USB port used for USB data storage, which is brilliant at the moment because USB hubs ICs are unobtainium, so, we can’t make any more product, because of this software bug.

Every couple of months I would try the latest kernel, but all you needed to do was write to disk continuously and you would hit the bug in 4 hours max, 10 minutes on average. The best part is the kernel corruption kills the file system, we got a trace on a monitor (normally a headless system) but if you ssh’d in, you were dropped into and empty file system and you couldn’t run any tool to diagnose a problem, a simply tab competition would hang the shell. Fun times.

Never bother reporting the bug because I found hundreds of threads on the forums detailing similar issues. It is very uncool that to this day they still insist on using their bespoke driver instead of trying to mainline their performance fixes, otherwise everyone using the dwc2 ip would have befitted, and this bug would have been fixed with hundreds more eyeballs on the problem, not just the one USB guy at RPI towers.

Re: A one in a million bug in Switch kernel

#57

Post by saagarjha »

gpderetta wrote: post #46

First of all, it is amazing that the author managed to analyze the patch in so much details, it probably is an effort comparable to the bug fix itself. Still I think the article is missing some bits. I would expect any core migration to require barriers (either implicit or explicit) on both the old and new core otherwise the process would risk seeing its own stores and loads out of order. But in this case the barrier…

> But in this case the barrier is predicated on the execution of some cache manipulation instruction, so I suspect things are more complicated.

Why do you think so? The explanation given seem reasonable to me…

Re: A one in a million bug in Switch kernel

#58

Post by akino_germany »

fps-hero wrote: post #55
throwawaybyeeee wrote: post #35

Reminds me of a similar bug that I worked on a few years ago that led to my single-line contribution to xnu (apologies for the dissertation): We had increasing reports of devices panicking because the kernel stopped draining a buffer, causing the buffer to fill. This particular buffer should never fill, so if it does -> panic. The first problem was that this bug was getting 'hot'. The bug needed to be fixed yesterday…

Thank you for sharing, gosh that must have been horrific to track down. There really is something about USB buses that cause the worst kinds of errors. I happen to know that the USB2 driver in the RPI 1/2/3/4 has a Linux kernel corruption bug which is completely masked by the use of a USB hub. Why does it matter? Because the RPI 1, 2, 3 all hide the USB port behind a hub. Only the zero, and the a series have a naked…

Avoid high-throughput devices is even in their list of known USB issues, so they must have some idea: https://www.raspberrypi.com/documentation/computers/raspberr...

Re: A one in a million bug in Switch kernel

#59

Post by gpderetta »

saagarjha wrote: post #57
gpderetta wrote: post #46

First of all, it is amazing that the author managed to analyze the patch in so much details, it probably is an effort comparable to the bug fix itself. Still I think the article is missing some bits. I would expect any core migration to require barriers (either implicit or explicit) on both the old and new core otherwise the process would risk seeing its own stores and loads out of order. But in this case the barrier…

> But in this case the barrier is predicated on the execution of some cache manipulation instruction, so I suspect things are more complicated. Why do you think so? The explanation given seem reasonable to me…

As I sad, I would expect the barriers to be needed unconditionally on a core migration. The fact that there is a special flag that is set when (and only when) the cache control instructions are used seem to point to some special handling specifically for those instructions.

Edit: having read the page for the nth time, I think I finally understand your point. The code using the cache instructions had an explicit barrier already, but it would be executed on the wrong thread.

I know nothing about the arm memory model, but likely the dsb sy barrier is a stronger barrier than needed for intercore communication, and it is needed for IO serialisation, for example with an mapped PCI device.

So yes, the article is clear and likely correct, I just failed to understand it fully originally.

Re: A one in a million bug in Switch kernel

#60

Post by Thrymr »

kevincox wrote: post #43
throwaway5371 wrote: post #2

offtopic: I really wish reading preformatted text files on ios safari was good.. I have to export the file to Books in order to read it properly

I think people need to use text files less. Or at least stop hard-wapping them.

Does that mean that a browser shouldn't be able to display a text file with explicit line breaks, though?
Post reply on HN