Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A tale of an impossible bug: big.LITTLE and caching

mono-project.com

Post reply on HN
Print view View original

71–80 of 116 posts

Re: A tale of an impossible bug: big.LITTLE and caching

#71

Post by rayiner »

pm215 wrote: post #10

Properly configured big.LITTLE clusters should be set up so that all CPUs report the same cache line size (which might be smaller than the true cache line size for some of the CPUs), to avoid exactly this kind of problem. The libgcc code assumes the hardware is correctly put together. There is a Linux kernel patchset currently going through review which provides a workaround for this kind of erratum by trapping the C…

That probably preserves correctness, but also invalidates any attempt to rely on the cache line size to lay out data structures to avoid cache line ping-ponging. Which is a major reason a program would care about the cache line size to begin with.

I think the really correct answer might be to abandon the idea that the system has a single cache line size...

Re: A tale of an impossible bug: big.LITTLE and caching

#73

Post by JoeAltmaier »

randyrand wrote: post #72

When would a programmer explicitly need to invalidate the CPU cache? Does this not happen automatically on a context switch?

One example is when doing a dma operation to memory. That often bypasses the cache. If the new data is to 'stick' the cache needs to be convinced it doesn't know what's in that buffer any more. This is an ARM thing; Intel architectures integrate DMA with the cache.

Re: A tale of an impossible bug: big.LITTLE and caching

#74

Post by cesarb »

wolfgke wrote: post #57
wolfgke wrote: post #22

> Worse, not even the ARM ISA is ready for this. An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache function with a certain cache line size, where it might not be valid anymore. I rather see the problem in the fact that there seems to be no possibility to sa…

Another idea how one could solve this problem by a clever kernel interface. This idea is probably a better than the approach in my post above, since this enables to migrate threads from big to little cores and vice versa except for critical regions. Add an interface to the kernel/scheduler void lock_cacheline_size(...) void unlock_cacheline_size(...) Calling lock_cacheline_size tells that from now on the thread must…

A simpler and more robust approach would be to add a system call to flush a memory range. The kernel can then do all the locking, without the risk that migration to the other core gets indefinitely blocked by user space code.

Re: A tale of an impossible bug: big.LITTLE and caching

#75

Post by PDoyle »

flamedoge wrote: post #48

I wonder why no one tried to validate Asymmetric MultiProcessing by first validating all cases with either little or big first. And then bisect further down when both are enabled.

Because that's the sort of thing you think of only when you already know the answer.

Re: A tale of an impossible bug: big.LITTLE and caching

#76

Post by gens »

GrumpyNl wrote: post #33
dsp1234 wrote: post #24

Earlier quoted context omitted.

Watson was actually Dr. Watson. Which is/was also the name of a debugger in Windows[0]. [0] - https://en.wikipedia.org/wiki/Dr._Watson_(debugger)

You are right, although i was referring to a Sherlock Holmes quote, but that is also Dr.

Funny thing is that Sherlock Holmes usually used abductive reasoning, instead of strictly deductive reasoning.

Re: A tale of an impossible bug: big.LITTLE and caching

#77

Post by deathanatos »

mikeash wrote: post #59
akavel wrote: post #53

Earlier quoted context omitted.

"There are only two hard things in Computer Science: cache invalidation and naming things" ― Phil Karlton; not sure to what extent this quote is compatible with the second one by Rob. Or does it mean by implication that simply "Computer Science is bugs waiting to happen"?...

It's a great quote, but it's wrong. There are actually two hard things in CS: cache invalidation, naming, and off-by-one errors.

The parent actually appears to have the quote both correct in content and attribution. Supposedly, someone else added the "off by one"[1][2][3]. That seems to be the extent of the Internet's knowledge on the quote, though the Skeptics link notes that there's nothing direct to the supposed originator.

This is one of those quotes where I feel there's more than one right answer. I like the addition of "off-by-one", and to make it a nice round three things, I usually use this version:

  There are three hard things in computer science:

  1. Naming things
  2. 3. Concurrency
  Cache Invalidation
  4. Off-by-one errors
[1]: https://twitter.com/timbray/status/506146595650699264

[2]: https://skeptics.stackexchange.com/questions/19836/has-phil-...

[3]: http://martinfowler.com/bliki/TwoHardThings.html

Re: A tale of an impossible bug: big.LITTLE and caching

#79

Post by bzbarsky »

rodrigokumpera wrote: post #68
brandmeyer wrote: post #64

Earlier quoted context omitted.

I don't see why they have to do this in userspace at all. If they did: * allocate read/write buffer * JIT instructions into it * change mapping to read/execute * run the JITted code Then the kernel manages flushing the data caches on the mapping change, and Mono gets to wrap a Somebody Else's Problem field around it. It sounds like they are instead: * allocate read/write/execute buffer * JIT instructions into it * ma…

That approach is harder to use in practice that in sounds. It's not like people have not tried it. The OS only let you alloc in large granules, like 4k or 16k, and the vast majority of the methods are significantly smaller than that, meaning a JIT must colocate multiple methods in the same allocation block or waste a significant amount of memory. We could get around that by remapping memory between read/write to read…

Firefox is shipping W^X for jitcode, as far as I know. Or at least https://bugzilla.mozilla.org/show_bug.cgi?id=1215479 is marked as fixed and I don't see any obvious bugs blocking it that are fallout from that change.

And the "Firefox (Ion)" and "Firefox (non writable jitcode)" lines on https://arewefastyet.com/ seem to coincide...

But yes, actually making this work in practice is not at all simple.

Re: A tale of an impossible bug: big.LITTLE and caching

#80

Post by userbinator »

Seeing bugs like this reminds me of how much nicer things are on x86 where JITs do not need to flush caches. You can actually modify the instruction immediately ahead of the currently executing one, and the CPU will naturally "do the right thing"[1] --- it does slow down execution, as the CPU is essentially automatically detecting and flushing its cache/pipeline, but used sparingly can be a great optimisation. The write can even come from another core ("cross-modifying code") and everything will still work. Someone I know used this to great effect in squeezing out the last bits of performance from an application by eliminating checks on a few flag variables and the associated branching in a tight loop --- it simply "poked" instruction bytes from another core into the loop when it was time for that core to do something else.

[1] With the exception of pre-Pentium CPUs, where modifying at various forward offsets from the locus of execution could give insight into how big the prefetch queue is. With the Pentium it was fully detected and the later multithreaded/multicored react similarly to cross-modifying code as described above, which leads me to believe that Intel is very much supportive of these things as otherwise they could've just told programmers to do as ARM does.

Maybe what ARM needs, short of doing it the Intel way, is a "flush region" instruction which takes both the address and size, so it can automatically flush the appropriate cache lines based on the current hardware's cacheline size.

Post reply on HN