There's no such thing as a simple cache bug. - Rob Pike
Caches are bugs waiting to happen. Rob Pike @rob_pike 21 Mar 2014
A tale of an impossible bug: big.LITTLE and caching
41–50 of 116 posts
Re: A tale of an impossible bug: big.LITTLE and caching
#42Properly configured big.LITTLE clusters should be set up so that all CPUs report the same cache line size (which might be smaller than the true cache line size for some of the CPUs), to avoid exactly this kind of problem. The libgcc code assumes the hardware is correctly put together. There is a Linux kernel patchset currently going through review which provides a workaround for this kind of erratum by trapping the C…
Also, if I'm reading the proposed fix in the mono pull request correctly, it doesn't deal with the problem entirely because there's a race condition where the code might start execution on the core with the larger cache line size, and then get context-switched to the core with the smaller cache line size midway through executing its cache-maintenance loop. The chances of things going wrong are much smaller, but they'…
It's fine if core migration happens during the invalidation loop - the core migration itself surely must wipe the non-shared cache levels thoroughly, otherwise nothing would work.
EDIT: Actually if the big and little cores are used together, and not exclusively, then this might still be an issue, yeah.
Re: A tale of an impossible bug: big.LITTLE and caching
#43Earlier quoted context omitted.
Also, if I'm reading the proposed fix in the mono pull request correctly, it doesn't deal with the problem entirely because there's a race condition where the code might start execution on the core with the larger cache line size, and then get context-switched to the core with the smaller cache line size midway through executing its cache-maintenance loop. The chances of things going wrong are much smaller, but they'…
It's fine if core migration happens during the invalidation loop - the core migration itself surely must wipe the non-shared cache levels thoroughly, otherwise nothing would work. EDIT: Actually if the big and little cores are used together, and not exclusively, then this might still be an issue, yeah.
Yes and no.
Core migration don't need to reach a global synchronization point, just enough so that the 2 cores in question agree with each other. This can be done without requiring global visibility of all operations of the source core.
Re: A tale of an impossible bug: big.LITTLE and caching
#44Earlier quoted context omitted.
Also, if I'm reading the proposed fix in the mono pull request correctly, it doesn't deal with the problem entirely because there's a race condition where the code might start execution on the core with the larger cache line size, and then get context-switched to the core with the smaller cache line size midway through executing its cache-maintenance loop. The chances of things going wrong are much smaller, but they'…
It's fine if core migration happens during the invalidation loop - the core migration itself surely must wipe the non-shared cache levels thoroughly, otherwise nothing would work. EDIT: Actually if the big and little cores are used together, and not exclusively, then this might still be an issue, yeah.
No, in general Linux migrating processes between cores won't nuke the caches. The hardware's cache coherency protocols between CPUs in the cluster ensures that they are all in sync sufficiently that it's not needed.
Re: A tale of an impossible bug: big.LITTLE and caching
#45> Worse, not even the ARM ISA is ready for this. An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache function with a certain cache line size, where it might not be valid anymore. I rather see the problem in the fact that there seems to be no possibility to sa…
In that case, you could read /proc/cpuinfo and set your affinity mask.
Re: A tale of an impossible bug: big.LITTLE and caching
#46Earlier quoted context omitted.
It's fine if core migration happens during the invalidation loop - the core migration itself surely must wipe the non-shared cache levels thoroughly, otherwise nothing would work. EDIT: Actually if the big and little cores are used together, and not exclusively, then this might still be an issue, yeah.
No, in general Linux migrating processes between cores won't nuke the caches. The hardware's cache coherency protocols between CPUs in the cluster ensures that they are all in sync sufficiently that it's not needed.
I understood that the configurations currently in use usually only power up either the big or little cores at the same time, and that kind of migration has to wipe the caches, right? But that might be inaccurate, and you are of course right in the general case.
Re: A tale of an impossible bug: big.LITTLE and caching
#47> Worse, not even the ARM ISA is ready for this. An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache function with a certain cache line size, where it might not be valid anymore. I rather see the problem in the fact that there seems to be no possibility to sa…
In that case, you could read /proc/cpuinfo and set your affinity mask.
Suggest this to the authors of the original article. :-)
Re: A tale of an impossible bug: big.LITTLE and caching
#48I wonder why no one tried to validate Asymmetric MultiProcessing by first validating all cases with either little or big first. And then bisect further down when both are enabled.
Re: A tale of an impossible bug: big.LITTLE and caching
#49From the pseudo code, what is disadvantage that making get_current_cpu_cache_line_size() always get called?
That would create a race condition addressed at the bottom of the article: the process can get switched onto another CPU between the invocation of get_current_cpu_cache_line_size() and the invalidation. An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache func…
The follow up doesn't make sense to me
Therefore, we have to try to figure out a global minimum of the cache line sizes across all CPUs.
Re: A tale of an impossible bug: big.LITTLE and caching
#50> Worse, not even the ARM ISA is ready for this. An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache function with a certain cache line size, where it might not be valid anymore. I rather see the problem in the fact that there seems to be no possibility to sa…
That's too specific a feature to expose to developers - most people wouldn't even know about the feature, and the ones that did might needlessly enable it just to be conservative.
The intention of the big.LITTLE architecture is to let processes be migrated seamlessly between the small and big core and let the unused core be turned off to save power. The kernel and the hardware should work together to make the core switching transparent and expose a safe way to invalidate the cache independent of the current processor.