[deleted]
A one in a million bug in Switch kernel
81–90 of 95 posts
Re: A one in a million bug in Switch kernel
#82Earlier quoted context omitted.
ARM caches are (normally) coherent, though. That's not the issue here. It's something about the instruction reordering playing badly with the cache flush and interrupt entry hardware. And my point was that's a hardware errata and not a software bug. And there seems to be no link to docs from ARM describing the issue, which is IMHO kinda horrifying.
I don't see anything about instruction reordering? Where'd you get that from? And I don't really see how the interaction with interrupt entry could be the problem, since the code works just fine if the code in the interrupt leaves the thread on the same core. > ARM caches are (normally) coherent, though. Don't you need a memory barrier to get that coherency, though?
Memory barrier instructions on ARM exist to force ordering of memory operations as seen by external hardware (generally software on other cores). They obviously interact with the cache at a hardware level, but they're different layers of abstraction.
And the reason those memory operations might be seen in an order different from their appearance in the machine code is precisely the fact that the processor executes them in parallel and potentially out of order. On x86, the hardware does magic (in almost all cases) to prevent this artifact. But ARM puts the responsibility on the programmer.
But all that stuff is specified (even if it's hard to reason about). What's happening here is extra-specification, something about that cache invalidate and barrier interacts in a way that an interrupt can mess up. But we don't know what it is, because it seems like ARM didn't tell anyone.
Basically: as I see it, any OS author writing interrupt entry code on ARM64 (I work on Zephyr, though not on the ARM port) needs to put a barrier instruction on the entry path for safety, because at least some hardware misbehaves without it. But that said, almost all real OSes are going to have one anyway for locking purposes (i.e. you have to take a spinlock to interact with OS state somewhere, and htat requires a barrier on SMP ARM systems). It's likely that this Nintendo sequence is part of some kind of micro-optimized thing and not a general purpose ISR.
Re: A one in a million bug in Switch kernel
#83Earlier quoted context omitted.
I don't see anything about instruction reordering? Where'd you get that from? And I don't really see how the interaction with interrupt entry could be the problem, since the code works just fine if the code in the interrupt leaves the thread on the same core. > ARM caches are (normally) coherent, though. Don't you need a memory barrier to get that coherency, though?
Memory barrier instructions on ARM exist to force ordering of memory operations as seen by external hardware (generally software on other cores). They obviously interact with the cache at a hardware level, but they're different layers of abstraction. And the reason those memory operations might be seen in an order different from their appearance in the machine code is precisely the fact that the processor executes th…
On what basis are you saying the interrupt messes it up?
The post directly says that if a migration doesn't happen, then nothing goes wrong.
What messes up is when you do the barrier-needing instructions on one core, and a memory barrier on a completely different core. Which seems pretty expected to me.
If the thing you're describing happens, that does sound like a hardware bug, but I don't see where you got that description from.
Re: A one in a million bug in Switch kernel
#84Earlier quoted context omitted.
To be super pedantic, for one million ops it's closer to a 63% chance every day: Pr[something happens across 1_000_000 events] = 1 - Pr[nothing happens across 1_000_000 events] = 1 - Pr[nothing happens once]^1_000_000 ## assuming independence = 1 - (1 - Pr[something happens once])^1_000_000 = 1 - (1 - 1/1_000_000)^1_000_000 ≈ 1 - 0.378 = 0.632 It's still below 99% for 4 million ops.
> for one million ops Okay, but they said millions. > It's still below 99% for 4 million ops. I find this misleading, because it's... 98%. Which completely undermines your argument. If something happens 98% of days, it's fine to call that "every day".
Dude, it was meant to be mildly educational, not an "argument."
Re: A one in a million bug in Switch kernel
#85Earlier quoted context omitted.
> But in this case the barrier is predicated on the execution of some cache manipulation instruction, so I suspect things are more complicated. Why do you think so? The explanation given seem reasonable to me…
As I sad, I would expect the barriers to be needed unconditionally on a core migration. The fact that there is a special flag that is set when (and only when) the cache control instructions are used seem to point to some special handling specifically for those instructions. Edit: having read the page for the nth time, I think I finally understand your point. The code using the cache instructions had an explicit barri…
[deleted]
Re: A one in a million bug in Switch kernel
#86Earlier quoted context omitted.
> for one million ops Okay, but they said millions. > It's still below 99% for 4 million ops. I find this misleading, because it's... 98%. Which completely undermines your argument. If something happens 98% of days, it's fine to call that "every day".
Dude, it was meant to be mildly educational, not an "argument."
Then present it as a fun fact rather than as a correction?
And I'm not trying to be mean but I think the way you phrased your last line is accidentally anti-educational. Your last line treats 1 million ops and 4 million ops as nearly equivalent, when the truth is that 1 million ops is far from "every day" while 4 million ops can easily be called "every day".
And if you dislike the word "argument" pretend I said "point"? I think you're reading connotations into that word that I didn't intend.
Re: A one in a million bug in Switch kernel
#87Earlier quoted context omitted.
If you have millions of ops per day, a million-to-one chance of something means you'll see it every day! And it only takes a few noisy customers to bring these issues to light.
I saw this first hand when I was overseeing the crash reports during the release of an online AAA game. A game with millions of players. A fairly frequent crash bug was caused by a line with a comment explaining that it could theoretically cause a crash but that risk would be one in a million.
How often did that code run? Even with one player, that would be too high for a per-frame risk...
Re: A one in a million bug in Switch kernel
#88Earlier quoted context omitted.
I saw this first hand when I was overseeing the crash reports during the release of an online AAA game. A game with millions of players. A fairly frequent crash bug was caused by a line with a comment explaining that it could theoretically cause a crash but that risk would be one in a million.
How often did that code run? Even with one player, that would be too high for a per-frame risk...
It only happened on players connecting or disconnecting from the server. Unfortunately it was a server crash so once it did occur it discontented up to eight players.
Re: A one in a million bug in Switch kernel
#89Earlier quoted context omitted.
Memory barrier instructions on ARM exist to force ordering of memory operations as seen by external hardware (generally software on other cores). They obviously interact with the cache at a hardware level, but they're different layers of abstraction. And the reason those memory operations might be seen in an order different from their appearance in the machine code is precisely the fact that the processor executes th…
On what basis are you saying the interrupt messes it up? The post directly says that if a migration doesn't happen, then nothing goes wrong. What messes up is when you do the barrier-needing instructions on one core, and a memory barrier on a completely different core. Which seems pretty expected to me. If the thing you're describing happens, that does sound like a hardware bug, but I don't see where you got that des…
> On what basis are you saying the interrupt messes it up?
Because nothing else makes sense. The code as posted in the linked article does not seem to have an ordering violation that I can see. The linked blog just asserts that it's there, but AFAICT it isn't unless there's a symmetric ordering bug in the putative context switch code that isn't presented.
Re: A one in a million bug in Switch kernel
#90>Makes you think, do Linux, Windows and Mac handle this properly? Honestly, I doubt it! Context switches, idle state transitions, etc tend to be fairly delicately handled as a common cause of CVEs and Heisenbugs. I'm sure there's still plenty of bugs but more attention ends up being paid to these things on general purpose operating systems. More eyeballs on the code, more security researchers, more hardware variants…
Cache coherency on SoCs is one of the more hairy aspects, and one that's getting an increasing amount of attention from the software world. Certainly for Mac it's a well-known topic, though I won't necessarily say it's all safe. Like I said: it's hairy. (fun fact: when you have 10s-100s of millions of units out there, those "1 in a million" chances become all too frequent...)
Cache coherency for SoCs is usually proven with formal verification. It's a very good target for such techniques because the needed properties are straightforward to write down.
I'd be very surprised to see any respectable firm shipping a multicore chip where formal verification has not been done on the cache protocol.
Note that this isn't perfect; if a needed property was not written down and was not checked, errors can be missed. But people have been doing this for at least 15 years now, and there's academic work that is older.
Edit: this doesn't mean everyone does it right, there's at least one example in the comments about someone shipping buggy cache coherence.