Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one in a million bug in Switch kernel

gist.githubusercontent.com

Post reply on HN
Print view View original

61–70 of 95 posts

Re: A one in a million bug in Switch kernel

#61

Post by snarf21 »

veltas wrote: post #19

> And how do you even find / debug a bug like this? As someone who has worked on cache code, I suspect it's quite possible they were just reviewing this code again and realised the potential hole. Or they were trying to track down some horrific bug and fixed this along the way (whether or not it caused it), reviewing anything to do with caching is probably worth doing because it's notoriously difficult to get right,…

In Computer Science there are only two hard problems: Cache Invalidation, Naming Things and Off By One Errors.

Re: A one in a million bug in Switch kernel

#62

Post by toast0 »

throwawaylinux wrote: post #28
chc4 wrote: post #20

Earlier quoted context omitted.

For the record: Returning from an interrupt is fully serializing on x86, which I believe means that all modern operating systems on x86 will handle this properly. https://pvk.ca/Blog/2019/01/09/preemption-is-gc-for-memory-r... is a very good blog post about exploiting this for a high-performance membarrier daemon.

But a thread can be context switched via a system call, so switching back to it will return to it via a return from system call which is not necessarily serializing on x86. Also memory operations and barriers executed in kernel mode have to be ordered correctly within a thread and context switches there can happen cooperatively so there may be no interrupt at all. Also a serializing operation is not a memory barrier.…

> But a thread can be context switched via a system call, so switching back to it will return to it via a return from system call which is not necessarily serializing on x86.

System call entry is under the control of the thread though. If your sequence of operations fails due to a missing barrier and being rescheduled after a system call, it might be a kernel bug, but you can fix it by putting in the barrier and it would likely be fixed on the switch via game patches and maybe a quality control check that the troublesome sequence is not present in the code.

Re: A one in a million bug in Switch kernel

#63

Post by fallat »

TravelPiglet wrote: post #39
richardfey wrote: post #34

Could this have been used for some exploit and that is why Nintendo prioritised it and fixed it?

Or, hopefully, they are releasing a new Switch with more cores and it manifested it self more often on that hardware. :)

...this sounds so plausible. So so plausible.

Re: A one in a million bug in Switch kernel

#64

Post by throwra620 »

gpderetta wrote: post #46

First of all, it is amazing that the author managed to analyze the patch in so much details, it probably is an effort comparable to the bug fix itself. Still I think the article is missing some bits. I would expect any core migration to require barriers (either implicit or explicit) on both the old and new core otherwise the process would risk seeing its own stores and loads out of order. But in this case the barrier…

No post body was provided.

Re: A one in a million bug in Switch kernel

#65

Post by mturmon »

throwawaybyeeee wrote: post #35

Reminds me of a similar bug that I worked on a few years ago that led to my single-line contribution to xnu (apologies for the dissertation): We had increasing reports of devices panicking because the kernel stopped draining a buffer, causing the buffer to fill. This particular buffer should never fill, so if it does -> panic. The first problem was that this bug was getting 'hot'. The bug needed to be fixed yesterday…

Great anecdote, thanks for sharing your dissertation ;)

Re: A one in a million bug in Switch kernel

#66

Post by grumple »

dstick wrote: post #29

"In the fragile reality of Discworld, and with the gods who like to play games, a million-to-one chance succeeds nine times out of ten." https://wiki.lspace.org/Million-to-one_chance

If you have millions of ops per day, a million-to-one chance of something means you'll see it every day! And it only takes a few noisy customers to bring these issues to light.

Re: A one in a million bug in Switch kernel

#67

Post by cwzwarich »

gpderetta wrote: post #46

First of all, it is amazing that the author managed to analyze the patch in so much details, it probably is an effort comparable to the bug fix itself. Still I think the article is missing some bits. I would expect any core migration to require barriers (either implicit or explicit) on both the old and new core otherwise the process would risk seeing its own stores and loads out of order. But in this case the barrier…

…

Re: A one in a million bug in Switch kernel

#68

Post by jacquesm »

gpderetta wrote: post #59
saagarjha wrote: post #57

Earlier quoted context omitted.

> But in this case the barrier is predicated on the execution of some cache manipulation instruction, so I suspect things are more complicated. Why do you think so? The explanation given seem reasonable to me…

As I sad, I would expect the barriers to be needed unconditionally on a core migration. The fact that there is a special flag that is set when (and only when) the cache control instructions are used seem to point to some special handling specifically for those instructions. Edit: having read the page for the nth time, I think I finally understand your point. The code using the cache instructions had an explicit barri…

Very classy!

Re: A one in a million bug in Switch kernel

#69

Post by gpderetta »

cwzwarich wrote: post #67
gpderetta wrote: post #46

First of all, it is amazing that the author managed to analyze the patch in so much details, it probably is an effort comparable to the bug fix itself. Still I think the article is missing some bits. I would expect any core migration to require barriers (either implicit or explicit) on both the old and new core otherwise the process would risk seeing its own stores and loads out of order. But in this case the barrier…

…

After rereading the article, the "missing bit", which is actually tangentially touched in the article is that, the barrier is not needed to synchronize between the two core, but to synchronize with other hardware, for example the GPU (hence the note about graphic glitches). So the context switching code need to issue the barrier from the correct core. The Linux kernel for example always issue the additional I/O barrier on core migration.

Re: A one in a million bug in Switch kernel

#70

Post by opan »

kevincox wrote: post #43
throwaway5371 wrote: post #2

offtopic: I really wish reading preformatted text files on ios safari was good.. I have to export the file to Books in order to read it properly

I think people need to use text files less. Or at least stop hard-wapping them.

I love text files and kinda like hard wrapping as well.
Post reply on HN