Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Frame pointers vs. DWARF – my verdict

rwmj.wordpress.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: Frame pointers vs. DWARF – my verdict

#31

Post by masklinn »

tialaramex wrote: post #19
masklinn wrote: post #12

Earlier quoted context omitted.

> frame pointer = 1 less register on an already set of registers under heavy strain (x86_64). X86-64 is not under heavy register pressure strain. That idea is a legacy from x86 (plain, 32b). x86-64 has the same register count as ARMv7 and few bothered disabling the frame pointer there, even though it’s a load-store architecture.

Right, x86-64 offers eight extra register names† (r8 through r15). If you choose to go from x86-without-frame-pointer to x86-64-with-frame-pointer you gained 7 register names which is huge. This makes the case where that one extra register name makes all the difference much rarer, arguably turning it from "I demand a compiler flag" to "Let's just hand-write the machine code for this one very special routine if our pe…

And that’s before accounting for SP which the compiler will usually avoid touching. So for an x86 target, omitting the frame pointer means going from 6 to 7 usable registers, on x86-64 it’s 14 to 15, a lot less necessary outside of very specific work loads.

Re: Frame pointers vs. DWARF – my verdict

#32

Post by DaveFlater »

zznzz wrote: post #29
DaveFlater wrote: post #15

Uh... [Begin flashback] 2012: A change made in GCC 4.7 allowed optimization to reschedule and defer the push of the frame pointer that previously occurred in the function prologue whenever frame pointers were enabled. When binaries are profiled using frame pointers, incorrect call chains are derived whenever a sample is taken between the top of the function and the instruction that pushes the frame pointer. I complai…

I cannot comment on whether “everyone” is oblivious but yes, this is still the case - frame pointer based unwinding sometimes skips the caller when the IP is sampled before the callee sets up a frame. This is also common for samples in leaf functions. compiler & tool chain folks tend to think (quite justifiably imo) that this and similar stuff is fine because dwarf allows reconstructing everything perfectly. The prob…

AFAIK, the optimization undermined the only use case that -fno-omit-frame-pointer actually had on x86_64. Is there a real use case that benefitted from allowing the frame pointer push to wander? Why why why

Re: Frame pointers vs. DWARF – my verdict

#33

Post by titzer »

zznzz wrote: post #23
titzer wrote: post #16

> But collecting the whole stack would consume far too much storage, so by default it only collects the first 8K. Many userspace stacks will be larger than this, in which case the data collection will simply be incomplete – it will never be possible to recover the full stack trace. Yeah, doing an 8KB memcopy for every profile sample sounds like a lot of overhead. Is DWARF unwinding so slow that that is actually faste…

> Is DWARF unwinding so slow that that is actually faster? No. The only reason it works like this is because the upstream Linux kernel has thus far rejected in-kernel dwarf unwinders, but copying the stack is simpler and available / implemented.

Why is profiling done in the kernel for userspace stacks?

Re: Frame pointers vs. DWARF – my verdict

#34

Post by yvdriess »

remexre wrote: post #26
the_mitsuhiko wrote: post #21

Earlier quoted context omitted.

> In any case, the runtime performance hit of frame pointers is quite high That's not true at all. It's in some very rare cases . Firefox I think at this point ships with framepointers enabled and so does every M1/M2 mac app and all iOS applications as it's mandatory for the calling convention on Apple.

On AArch64 I think it's cheaper because losing one register doesn't hurt as much, since you've got twice as many.

nitpick: it's the number of physical registers that matter more here, not ISA registers

Re: Frame pointers vs. DWARF – my verdict

#35

Post by irogers »

pjc50 wrote: post #9

Relevant discussion on profiling without frame pointers: https://twitter.com/halvarflake/status/1577644229853151233 / https://prodfiler.com/blog/introducing-prodfiler/ via the "prodfiler" tool.

Twitter: "No need to recompile with frame pointers."

Web page: "always-on profiling powered by eBPF technology."

BPF stack traces are gathered using frame pointers:

https://github.com/torvalds/linux/blob/master/kernel/bpf/sta...

https://github.com/torvalds/linux/blob/master/arch/x86/event...

Re: Frame pointers vs. DWARF – my verdict

#36

Post by irogers »

zznzz wrote: post #25
the_mitsuhiko wrote: post #20

Earlier quoted context omitted.

I think generally the talk about "there are not enough registers" ignore pipelining and register renaming way too much. The loss of performance of the frame pointer register even on x86 is not that problematic, and on x86_64 it's completely negligible unless you're in a tight switch heavy interpreter loop.

Register renaming doesn’t significantly address the impact of reducing the number of architectural registers available to the compiler. With fewer register names available, the compiler will spill locals to stack more often, and register renaming doesn’t help - memory renaming is needed to really mitigate this. But i agree the impact of preserving frame pointers is generally quite small and doesn’t often actually nee…

Agner speaks about memory renaming back on Zen 2:

https://www.agner.org/forum/viewtopic.php?t=41

Intel Alderlake has performance events for tracking it:

https://github.com/intel/perfmon/blob/974c69919b2a9dfd8278cf...

But even before this you had store to load forwarding on x86. I'm not saying you have, but before inventing a performance problem it is worth spending time trying to diagnose it with thorough profiling (e.g. [1]). The Fedora frame pointer patch did a thorough performance analysis and performance will be revisited again. Unfortunately there are a lot of arm chair performance experts who haven't spent time looking into the details.

[1] https://perf.wiki.kernel.org/index.php/Top-Down_Analysis

Re: Frame pointers vs. DWARF – my verdict

#37

Post by irogers »

ishitatsuyuki wrote: post #4

IMHO, perf's decision to write whole stacks directly to the disk and unwinding them as a post-process is a really bad design. It wastes disk space, and as the author pointed out, it also has a lot of IO overhead. As an alternative approach, https://github.com/mstange/samply processes data streamed from perf and unwinds it in realtime. The unwinding overhead is surprisingly low: it only takes around 1% of (single) CPU…

This could be a great Linux perf GSoC project. Projects and mentors are being looked for: https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf

Re: Frame pointers vs. DWARF – my verdict

#38

Post by tdullien »

nneonneo wrote: post #13

The article points out that the kernel uses ORC instead of DWARF for unwinding. I wonder if that could ever become an option in userspace? I imagine that if all you’re interested in is stack traces, instead of debugging (which DWARF is designed for), ORC would be a very nice performance win. And it’s not as if they’re mutually exclusive, either: there’s no reason why a binary or debuginfo couldn’t just ship both. In…

For prodfiler.com we convert .eh_frame (DWARF) unwinding format to something that is more similar to ORC and optimized for lookup in eBPF data structures.

Re: Frame pointers vs. DWARF – my verdict

#39

Post by remexre »

yvdriess wrote: post #34
remexre wrote: post #26

Earlier quoted context omitted.

On AArch64 I think it's cheaper because losing one register doesn't hurt as much, since you've got twice as many.

nitpick: it's the number of physical registers that matter more here, not ISA registers

Why wouldn't the lower ISA register count lead to more spills, regardless of the physical count? Is store forwarding really that effective?

Re: Frame pointers vs. DWARF – my verdict

#40

Post by tdullien »

What everybody in this discussion seems to miss is that you don't need to unwind the DWARF data structures during profiling time, you are free to convert DWARF to a fast-lookup data structure on the machine.

DWARF needs to support every CPU under the sun. Every unwinder on the other hand is CPU-specific. For prodfiler.com's continuous in-production unwinding, we convert DWARF into something compact and fast-to-lookup that is then placed in eBPF maps.

It all works like a charm. We can have our cake (e.g. use RBP as GPR) and eat it too (e.g. use .eh_frame, converted at runtime into a fast-to-lookup format) to do reliable whole-system unwinding in production.

Post reply on HN