Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A tale of an impossible bug: big.LITTLE and caching

mono-project.com

Post reply on HN
Print view View original

81–90 of 116 posts

Re: A tale of an impossible bug: big.LITTLE and caching

#81

Post by caf »

rodrigokumpera wrote: post #68
brandmeyer wrote: post #64

Earlier quoted context omitted.

I don't see why they have to do this in userspace at all. If they did: * allocate read/write buffer * JIT instructions into it * change mapping to read/execute * run the JITted code Then the kernel manages flushing the data caches on the mapping change, and Mono gets to wrap a Somebody Else's Problem field around it. It sounds like they are instead: * allocate read/write/execute buffer * JIT instructions into it * ma…

That approach is harder to use in practice that in sounds. It's not like people have not tried it. The OS only let you alloc in large granules, like 4k or 16k, and the vast majority of the methods are significantly smaller than that, meaning a JIT must colocate multiple methods in the same allocation block or waste a significant amount of memory. We could get around that by remapping memory between read/write to read…

How is this safe in the multithreaded case anyway? If a process has just written a new JITted method and is flushing the i$ on the CPU it's executing on, but then gets scheduled away part-way through the flush, if you were very unlucky then couldn't another thread then get scheduled on that CPU and try to execute the just-written method, which failed to be fully flushed from that CPU's cache?

Re: A tale of an impossible bug: big.LITTLE and caching

#83

Post by brandmeyer »

rodrigokumpera wrote: post #68
brandmeyer wrote: post #64

Earlier quoted context omitted.

I don't see why they have to do this in userspace at all. If they did: * allocate read/write buffer * JIT instructions into it * change mapping to read/execute * run the JITted code Then the kernel manages flushing the data caches on the mapping change, and Mono gets to wrap a Somebody Else's Problem field around it. It sounds like they are instead: * allocate read/write/execute buffer * JIT instructions into it * ma…

That approach is harder to use in practice that in sounds. It's not like people have not tried it. The OS only let you alloc in large granules, like 4k or 16k, and the vast majority of the methods are significantly smaller than that, meaning a JIT must colocate multiple methods in the same allocation block or waste a significant amount of memory. We could get around that by remapping memory between read/write to read…

I see the argument about the page size potentially being large relative to the size of a jitted function. But,

> modifying a memory mapping is very expensive

It used to be true that operations on memory mappings were appallingly expensive. However, the advent of virtualization has driven a significant change in performance. IIRC, ARMv8 has a TLB invalidation operation that is per-entry, addressed by the virtual address being invalidated. You don't need to flush the entire TLB cache.

Re: A tale of an impossible bug: big.LITTLE and caching

#84

Post by hexa00 »

I had this problem too with GDB on the Odroid UX4 big.LITTLE SoC.

Since GDB is patching the instrution with ptrace to insert a breakpoint for example.

See my blog post about it: https://www.kayaksoft.com/blog/2016/05/11/random-sigill-on-a...

Or the post on the GDB mailling list: https://www.sourceware.org/ml/gdb/2015-11/msg00030.html

Too bad however that the kernel patchset mentionned in a previous post only covers arm64..

So it's still a problem from arm32.

Re: A tale of an impossible bug: big.LITTLE and caching

#85

Post by dfox »

Animats wrote: post #58

"first mass produced AMP architecture" Nope. Remember the Cell? The processor in the Playstation 3? One main CPU with 8 little CPUs and no shared memory, just channels. The Playstation 4 isn't a AMP machine because programming the Cell was so hard.

Cell is essentially distributed memory cluster on single chip, because each SPU has it's own address space and cannot directly access main memory. I'm not sure about what the exact definition of AMP is, but it does not exactly match my feeling of what AMP should be. In this regard Wii seems more like AMP systems with two completely different CPUs (PPC and ARM) sharing what essentially amounts to be same address space (and in WiiU there are 3 PPC cores where one of them is slightly different than other two and cache coherency between them can only be described as broken).

There is no question of hardness of programming for Cell, but I think it's mostly about middleware support (probably because the platform is so different from PC and xbox360).

Re: A tale of an impossible bug: big.LITTLE and caching

#86

Post by gcp »

anarazel wrote: post #8

Different cacheline sizes for the different cores seems like an absurdly bad idea. One because it opens one up to bugs like these, but also because it makes optimization a lot harder. I have a hard time believing the savings due to a larger line size are worth it.

ARM's own designs (A53, A57, A72, A73) all have 64-byte cache line sizes and avoid the problem entirely.

The one at fault appears to be Samsung, who designed M1 Mongoose with 128 byte lines and packed it together with A53 cores in their SoC.

Re: A tale of an impossible bug: big.LITTLE and caching

#87

Post by rodrigokumpera »

caf wrote: post #81
rodrigokumpera wrote: post #68

Earlier quoted context omitted.

That approach is harder to use in practice that in sounds. It's not like people have not tried it. The OS only let you alloc in large granules, like 4k or 16k, and the vast majority of the methods are significantly smaller than that, meaning a JIT must colocate multiple methods in the same allocation block or waste a significant amount of memory. We could get around that by remapping memory between read/write to read…

How is this safe in the multithreaded case anyway? If a process has just written a new JITted method and is flushing the i$ on the CPU it's executing on, but then gets scheduled away part-way through the flush, if you were very unlucky then couldn't another thread then get scheduled on that CPU and try to execute the just-written method, which failed to be fully flushed from that CPU's cache?

Multi-threaded safety is simply due to JIT controlling the visibility of the newly compiled code. First flush, then make it visible for execution, can't go wrong with that and scheduling won't matter.

Things get a lot more complicated when it comes to code patching, but the principle is similar.

Re: A tale of an impossible bug: big.LITTLE and caching

#88

Post by colejohnson66 »

mayank10j wrote: post #36
pm215 wrote: post #10

Properly configured big.LITTLE clusters should be set up so that all CPUs report the same cache line size (which might be smaller than the true cache line size for some of the CPUs), to avoid exactly this kind of problem. The libgcc code assumes the hardware is correctly put together. There is a Linux kernel patchset currently going through review which provides a workaround for this kind of erratum by trapping the C…

what's the incentive for cpu manufacturer to make effort of building extra cache memory in hardware for bigger cpu in ARM64, if there is no sane way to use it ?

The difference between cache line size and cache size is like paper. You can make it wider (bigger cache line size), taller (bigger cache size), or both.

The problem is like printing. If you put in an A4 or letter (ANSI A) sized sheet and tell your printer it's A3 or tabloid (ANSI B), you're gonna have problems.

Re: A tale of an impossible bug: big.LITTLE and caching

#89

Post by caf »

rodrigokumpera wrote: post #87
caf wrote: post #81

Earlier quoted context omitted.

How is this safe in the multithreaded case anyway? If a process has just written a new JITted method and is flushing the i$ on the CPU it's executing on, but then gets scheduled away part-way through the flush, if you were very unlucky then couldn't another thread then get scheduled on that CPU and try to execute the just-written method, which failed to be fully flushed from that CPU's cache?

Multi-threaded safety is simply due to JIT controlling the visibility of the newly compiled code. First flush, then make it visible for execution, can't go wrong with that and scheduling won't matter. Things get a lot more complicated when it comes to code patching, but the principle is similar.

I don't think that helps - the point is that the flush might not be effective if the flushing thread gets scheduled away from the core which has the stale I$ before it manages to fully issue the flush.

Or is the flush guaranteed to flush all cores caches? That would be a fairly unusual design.

Re: A tale of an impossible bug: big.LITTLE and caching

#90

Post by pawadu »

gcp wrote: post #86
anarazel wrote: post #8

Different cacheline sizes for the different cores seems like an absurdly bad idea. One because it opens one up to bugs like these, but also because it makes optimization a lot harder. I have a hard time believing the savings due to a larger line size are worth it.

ARM's own designs (A53, A57, A72, A73) all have 64-byte cache line sizes and avoid the problem entirely. The one at fault appears to be Samsung, who designed M1 Mongoose with 128 byte lines and packed it together with A53 cores in their SoC.

You could also say that the ARM implementation is not optimal since it uses the same cache size for vastly different designs (different pipeline length and memory access characteristics).

Samsung tried to improve performance, I don't think they are at fault at all.

Post reply on HN