Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A one in a million bug in Switch kernel

gist.githubusercontent.com

Post reply on HN
Print view View original

71–80 of 95 posts

Re: A one in a million bug in Switch kernel

#71

Post by kabdib »

I spent a couple weeks finding a similar bug, a one instruction window where a hardware "wake up" register could get a stale value if an interrupt-and-reschedule happened at just the right instruction. The fix was to swap two instructions, so a register write happened in the correct order.

I still remember the moment of clarity when the very thorny, complicated problem resolved into something obvious and simple, with a trivial fix. Hard problems seldom resolve so easily. You don't get these very often, cherish them :-)

Re: A one in a million bug in Switch kernel

#72

Post by foobiekr »

IshKebab wrote: post #31
veltas wrote: post #27

Earlier quoted context omitted.

Yep but can they reproduce it? When we say "low probability" we're acting like it's truly random, but in reality they could have stumbled across steps that reproduce it very frequently.

Sometimes you can figure out the bug without reliably reproducing it if you have enough logs/stack traces etc.

a lot of bugs in the embedded world get fixed just by code inspection; I think most people who have done systems or embedded coding have casually noted bugs just by going through some code looking for something else or adding a new path or feature.

Re: A one in a million bug in Switch kernel

#73

Post by KMag »

klauspost wrote: post #52
broodbucket wrote: post #7

>Makes you think, do Linux, Windows and Mac handle this properly? Honestly, I doubt it! Context switches, idle state transitions, etc tend to be fairly delicately handled as a common cause of CVEs and Heisenbugs. I'm sure there's still plenty of bugs but more attention ends up being paid to these things on general purpose operating systems. More eyeballs on the code, more security researchers, more hardware variants…

x86 Darwin had a bug where AVX512 K (opmask) registers would not be saved and restored on context switches when YMM registers had all 0 values. The kernel assumed that AVX512 wasn't used. They fixed it in Monterey 12.2, but still hasn't backported. Heisenbug for sure. Ref: https://github.com/golang/go/issues/49233#issuecomment-96373...

It's unfortunate that more architectures aren't implemented with something like DEC Alpha's PALCode, where the CPU's firmware is essentially a single-tenant hypervisor. The OS kernel proper uses upcalls to the firmware for manipulating page tables, etc. If abstracted properly, the OS would allocate the storage space, but ask the firmware to serialize and deserialize processor state on context switches.

In retrospect, the Alpha went a bit too far with the permissiveness of its memory model, and it turned out they really did need single-byte load and store instructions. However, it was really an elegant high-level design, and the implementation team was top-notch (same folks worked on StrongArm, AMD Athlon, P.A. Semi's PWRficient, and Apple silicon).

Re: A one in a million bug in Switch kernel

#74

Post by ajross »

tedunangst wrote: post #22
ajross wrote: post #11

Earlier quoted context omitted.

FWIW: I re-read this a bunch of times, and I don't understand how this isn't a hardware bug. How can an asynchronous interrupt be specified in any rigorous way if it does NOT act as a memory barrier to the interrupted code? Clearly the CPU isn't going to cache its in-flight state for every interrupt (and remember interrupts can be themselves interrupted!). So certainly "most" of its state is being serialized. And we'…

It makes sense to me you might need a memory flush when doing a core migration. Everything is still coherent from a single core perspective. But if you point a different core at the same PC, well, maybe it sees things differently that haven't been flushed.

ARM caches are (normally) coherent, though. That's not the issue here. It's something about the instruction reordering playing badly with the cache flush and interrupt entry hardware. And my point was that's a hardware errata and not a software bug.

And there seems to be no link to docs from ARM describing the issue, which is IMHO kinda horrifying.

Re: A one in a million bug in Switch kernel

#75

Post by TuxSH »

ajross wrote: post #11
broodbucket wrote: post #7

>Makes you think, do Linux, Windows and Mac handle this properly? Honestly, I doubt it! Context switches, idle state transitions, etc tend to be fairly delicately handled as a common cause of CVEs and Heisenbugs. I'm sure there's still plenty of bugs but more attention ends up being paid to these things on general purpose operating systems. More eyeballs on the code, more security researchers, more hardware variants…

FWIW: I re-read this a bunch of times, and I don't understand how this isn't a hardware bug. How can an asynchronous interrupt be specified in any rigorous way if it does NOT act as a memory barrier to the interrupted code? Clearly the CPU isn't going to cache its in-flight state for every interrupt (and remember interrupts can be themselves interrupted!). So certainly "most" of its state is being serialized. And we'…

> and which configurations thereof, remember the Tegra in question has both A53 and A57 cores

Only the A57 cores are enabled at all, perhaps due to a hardware limitation. Consider the A53 missing for all intent and purposes.

Re: A one in a million bug in Switch kernel

#76

Post by grayclhn »

grumple wrote: post #66
dstick wrote: post #29

"In the fragile reality of Discworld, and with the gods who like to play games, a million-to-one chance succeeds nine times out of ten." https://wiki.lspace.org/Million-to-one_chance

If you have millions of ops per day, a million-to-one chance of something means you'll see it every day! And it only takes a few noisy customers to bring these issues to light.

To be super pedantic, for one million ops it's closer to a 63% chance every day:

Pr[something happens across 1_000_000 events]

= 1 - Pr[nothing happens across 1_000_000 events]

= 1 - Pr[nothing happens once]^1_000_000 ## assuming independence

= 1 - (1 - Pr[something happens once])^1_000_000

= 1 - (1 - 1/1_000_000)^1_000_000

≈ 1 - 0.378

= 0.632

It's still below 99% for 4 million ops.

Re: A one in a million bug in Switch kernel

#77

Post by Dylan16807 »

ajross wrote: post #74
tedunangst wrote: post #22

Earlier quoted context omitted.

It makes sense to me you might need a memory flush when doing a core migration. Everything is still coherent from a single core perspective. But if you point a different core at the same PC, well, maybe it sees things differently that haven't been flushed.

ARM caches are (normally) coherent, though. That's not the issue here. It's something about the instruction reordering playing badly with the cache flush and interrupt entry hardware. And my point was that's a hardware errata and not a software bug. And there seems to be no link to docs from ARM describing the issue, which is IMHO kinda horrifying.

I don't see anything about instruction reordering? Where'd you get that from?

And I don't really see how the interaction with interrupt entry could be the problem, since the code works just fine if the code in the interrupt leaves the thread on the same core.

> ARM caches are (normally) coherent, though.

Don't you need a memory barrier to get that coherency, though?

Re: A one in a million bug in Switch kernel

#78

Post by Dylan16807 »

grayclhn wrote: post #76
grumple wrote: post #66

Earlier quoted context omitted.

If you have millions of ops per day, a million-to-one chance of something means you'll see it every day! And it only takes a few noisy customers to bring these issues to light.

To be super pedantic, for one million ops it's closer to a 63% chance every day: Pr[something happens across 1_000_000 events] = 1 - Pr[nothing happens across 1_000_000 events] = 1 - Pr[nothing happens once]^1_000_000 ## assuming independence = 1 - (1 - Pr[something happens once])^1_000_000 = 1 - (1 - 1/1_000_000)^1_000_000 ≈ 1 - 0.378 = 0.632 It's still below 99% for 4 million ops.

> for one million ops

Okay, but they said millions.

> It's still below 99% for 4 million ops.

I find this misleading, because it's... 98%.

Which completely undermines your argument. If something happens 98% of days, it's fine to call that "every day".

Re: A one in a million bug in Switch kernel

#79

Post by Agentlien »

grumple wrote: post #66
dstick wrote: post #29

"In the fragile reality of Discworld, and with the gods who like to play games, a million-to-one chance succeeds nine times out of ten." https://wiki.lspace.org/Million-to-one_chance

If you have millions of ops per day, a million-to-one chance of something means you'll see it every day! And it only takes a few noisy customers to bring these issues to light.

I saw this first hand when I was overseeing the crash reports during the release of an online AAA game. A game with millions of players.

A fairly frequent crash bug was caused by a line with a comment explaining that it could theoretically cause a crash but that risk would be one in a million.

Re: A one in a million bug in Switch kernel

#80

Post by Agentlien »

TravelPiglet wrote: post #39
richardfey wrote: post #34

Could this have been used for some exploit and that is why Nintendo prioritised it and fixed it?

Or, hopefully, they are releasing a new Switch with more cores and it manifested it self more often on that hardware. :)

As a player I feel that would be pretty cool. As a developer I would absolutely love it.

edit: this says something about priorities. It bothers me quite a lot how much I need to simplify the graphics for the switch versions of games I work on. It hardly bothers me at all when I play games on switch that the visual fidelity is lower on switch

Post reply on HN