Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A tale of an impossible bug: big.LITTLE and caching

mono-project.com

Post reply on HN
Print view View original

91–100 of 116 posts

Re: A tale of an impossible bug: big.LITTLE and caching

#91

Post by pm215 »

userbinator wrote: post #80

Seeing bugs like this reminds me of how much nicer things are on x86 where JITs do not need to flush caches. You can actually modify the instruction immediately ahead of the currently executing one, and the CPU will naturally "do the right thing"[1] --- it does slow down execution, as the CPU is essentially automatically detecting and flushing its cache/pipeline, but used sparingly can be a great optimisation. The wr…

x86 is really the oddity here -- it has its no-explicit-cache-maintenance design because of wanting to maintain backwards-compatibility with self-modifying code that was written for x86 cores that had no caches at all. Almost all other architectures have explicit cache maintenance because it's more efficient (and requires less hardware), at the minor cost of requiring the very few bits of software which do odd things like JITting to explicitly tell the CPU what they're doing.

An instruction for flushing an entire region would potentially have a very long execution time, which is awkward because you would want to be able to interrupt and resume it. So it would need "how far have I got" state stored somewhere. The obvious observation from a RISC-architecture point of view is that you can get the equivalent effect without the pain of making a long-running interruptible instruction, by having an "invalidate one cache line" instruction plus an explicit loop in the code, and that's what most architectures do.

Re: A tale of an impossible bug: big.LITTLE and caching

#92

Post by gcp »

pawadu wrote: post #90
gcp wrote: post #86

Earlier quoted context omitted.

ARM's own designs (A53, A57, A72, A73) all have 64-byte cache line sizes and avoid the problem entirely. The one at fault appears to be Samsung, who designed M1 Mongoose with 128 byte lines and packed it together with A53 cores in their SoC.

You could also say that the ARM implementation is not optimal since it uses the same cache size for vastly different designs (different pipeline length and memory access characteristics). Samsung tried to improve performance, I don't think they are at fault at all.

Samsung's design breaks correctly written user-land code. Hence, they're 100% at fault.

Going to 128-byte cache lines was fine, but they should have made the lower-power part of the chip match (which in this case they likely couldn't because they licensed that design), or they should have made sure 128-byte lines were reported in all circumstances (which requires a hack similar to the workaround done in the kernel because again they can't make the A53 report 64 bytes).

Re: A tale of an impossible bug: big.LITTLE and caching

#93

Post by wolf550e »

betolink wrote: post #82

Can someone explain why cache flush is used for on ARM or in general low level programming?

Overwriting executable machine code in memory is a rare case (only JITs need it, only for emitting code). Some CPUs do wire the memory write operation to the instruction cache to make sure the instruction cache knows some code changed, but ARM chose a simpler design where the instruction cache assumes the code never changes in memory and if this assumption is wrong, the programmer has to use a special instruction to tell the instruction cache to throw something out of cache. The cache clearing instruction throws out either 64 bytes aligned to multiple of 64 or 128 bytes aligned to multiple of 128, depending on the core.

Re: A tale of an impossible bug: big.LITTLE and caching

#94

Post by Senji »

pm215 wrote: post #26
pm215 wrote: post #10

Properly configured big.LITTLE clusters should be set up so that all CPUs report the same cache line size (which might be smaller than the true cache line size for some of the CPUs), to avoid exactly this kind of problem. The libgcc code assumes the hardware is correctly put together. There is a Linux kernel patchset currently going through review which provides a workaround for this kind of erratum by trapping the C…

Also, if I'm reading the proposed fix in the mono pull request correctly, it doesn't deal with the problem entirely because there's a race condition where the code might start execution on the core with the larger cache line size, and then get context-switched to the core with the smaller cache line size midway through executing its cache-maintenance loop. The chances of things going wrong are much smaller, but they'…

Spin up N threads with single core affinities where N = total cores.

If anything, the OS should have an API to tell you this info in advance.

Re: A tale of an impossible bug: big.LITTLE and caching

#95

Post by Senji »

wolfgke wrote: post #22

> Worse, not even the ARM ISA is ready for this. An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache function with a certain cache line size, where it might not be valid anymore. I rather see the problem in the fact that there seems to be no possibility to sa…

>An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache function with a certain cache line size, where it might not be valid anymore.

Doesn't ARM have an "Enter Critical Region" instruction?

Re: A tale of an impossible bug: big.LITTLE and caching

#96

Post by pm215 »

Senji wrote: post #94
pm215 wrote: post #26

Earlier quoted context omitted.

Also, if I'm reading the proposed fix in the mono pull request correctly, it doesn't deal with the problem entirely because there's a race condition where the code might start execution on the core with the larger cache line size, and then get context-switched to the core with the smaller cache line size midway through executing its cache-maintenance loop. The chances of things going wrong are much smaller, but they'…

Spin up N threads with single core affinities where N = total cores. If anything, the OS should have an API to tell you this info in advance.

This won't work in the presence of CPU hotplug (which Android uses for power management), because some of the CPUs might not be online when you do it.

The API for "tell me this info" is "read the CTR_EL0 register"; it's a hardware bug that it doesn't do the right thing on this particular chip.

Re: A tale of an impossible bug: big.LITTLE and caching

#97

Post by userbinator »

pm215 wrote: post #91
userbinator wrote: post #80

Seeing bugs like this reminds me of how much nicer things are on x86 where JITs do not need to flush caches. You can actually modify the instruction immediately ahead of the currently executing one, and the CPU will naturally "do the right thing"[1] --- it does slow down execution, as the CPU is essentially automatically detecting and flushing its cache/pipeline, but used sparingly can be a great optimisation. The wr…

x86 is really the oddity here -- it has its no-explicit-cache-maintenance design because of wanting to maintain backwards-compatibility with self-modifying code that was written for x86 cores that had no caches at all. Almost all other architectures have explicit cache maintenance because it's more efficient (and requires less hardware), at the minor cost of requiring the very few bits of software which do odd things…

Actually, Intel explicitly broke backwards-compatibility starting with the Pentium, by adding the hardware to make SMC work without additional effort. The 486 and below needed an explicit branch to flush the prefetch queue, and this effect has been exploited for various anti-debugging tricks and even this amazing 8088-only optimisation:

https://news.ycombinator.com/item?id=9340231

An instruction for flushing an entire region would potentially have a very long execution time, which is awkward because you would want to be able to interrupt and resume it. So it would need "how far have I got" state stored somewhere.

x86 has the REP prefix for this purpose; used with certain instructions, it decrements a register and if it's nonzero, executes the instruction. The earlier implementations simply didn't update the instruction pointer in this case so the CPU would repeatedly fetch and execute the same instruction, and it's interruptable between each step. The register counts down how many iterations remain. Otherwise, the instruction pointer moves to the next instruction. Modern x86 handles this by generating uops instead in the decoder, but the basic functionality is the same.

Re: A tale of an impossible bug: big.LITTLE and caching

#98

Post by sangnoir »

gcp wrote: post #92
pawadu wrote: post #90

Earlier quoted context omitted.

You could also say that the ARM implementation is not optimal since it uses the same cache size for vastly different designs (different pipeline length and memory access characteristics). Samsung tried to improve performance, I don't think they are at fault at all.

Samsung's design breaks correctly written user-land code. Hence, they're 100% at fault. Going to 128-byte cache lines was fine, but they should have made the lower-power part of the chip match (which in this case they likely couldn't because they licensed that design), or they should have made sure 128-byte lines were reported in all circumstances (which requires a hack similar to the workaround done in the kernel be…

> Samsung's design breaks correctly written user-land code. Hence, they're 100% at fault.

This is similar to blaming OS/BIOS manufaturers for Y2K breaking "correctly written code" at the turn of the millenium.The code was incorrect in this instance because it assumed the cache size would be the same for all cores, Samsung simply manufactured a SOC that breaks that faulty assumption.

Re: A tale of an impossible bug: big.LITTLE and caching

#99

Post by cesarb »

gcp wrote: post #86
anarazel wrote: post #8

Different cacheline sizes for the different cores seems like an absurdly bad idea. One because it opens one up to bugs like these, but also because it makes optimization a lot harder. I have a hard time believing the savings due to a larger line size are worth it.

ARM's own designs (A53, A57, A72, A73) all have 64-byte cache line sizes and avoid the problem entirely. The one at fault appears to be Samsung, who designed M1 Mongoose with 128 byte lines and packed it together with A53 cores in their SoC.

Perhaps a better (and simpler) workaround, then, could be to clamp the reported cache line size to 64 bytes. So even if the Samsung core reports 128-byte cache lines, the code would simply invalidate each line twice, and if it is migrated in the middle of the invalidation loop, it would still work correctly.

Re: A tale of an impossible bug: big.LITTLE and caching

#100

Post by pawadu »

gcp wrote: post #92
pawadu wrote: post #90

Earlier quoted context omitted.

You could also say that the ARM implementation is not optimal since it uses the same cache size for vastly different designs (different pipeline length and memory access characteristics). Samsung tried to improve performance, I don't think they are at fault at all.

Samsung's design breaks correctly written user-land code. Hence, they're 100% at fault. Going to 128-byte cache lines was fine, but they should have made the lower-power part of the chip match (which in this case they likely couldn't because they licensed that design), or they should have made sure 128-byte lines were reported in all circumstances (which requires a hack similar to the workaround done in the kernel be…

> Samsung's design breaks correctly written user-land code.

Are you sure? From the article, it sounds to me that the real cause of the error is an incorrect assumption made by the GCC people.

Post reply on HN