Frame pointers vs. DWARF – my verdict • Hacker News Board

Re: Frame pointers vs. DWARF – my verdict

#41

Post by nneonneo » Wed, Feb 15, 2023, 4:56 PM UTC

the_mitsuhiko wrote: post #21

nneonneo wrote: post #13

The article points out that the kernel uses ORC instead of DWARF for unwinding. I wonder if that could ever become an option in userspace? I imagine that if all you’re interested in is stack traces, instead of debugging (which DWARF is designed for), ORC would be a very nice performance win. And it’s not as if they’re mutually exclusive, either: there’s no reason why a binary or debuginfo couldn’t just ship both. In…

> In any case, the runtime performance hit of frame pointers is quite high That's not true at all. It's in some very rare cases . Firefox I think at this point ships with framepointers enabled and so does every M1/M2 mac app and all iOS applications as it's mandatory for the calling convention on Apple.

The ORC documentation [1] cites a 5-10% perf hit [2] on x86. Now, granted, we're talking about an architecture that has only 8 logical registers to begin with, so losing one is going to hurt a lot more than x86-64 (16 logical registers) or AARCH64 (32 logical registers). But, 5-10% is definitely a noticeable perf hit.

Phoronix [3] did a test of "-fno-omit-frame-pointer" during the discussion on whether to enable the flag for Fedora, this time on x86-64. They found an average 14% performance hit on a wide variety of benchmarks.

[1] https://www.kernel.org/doc/html/latest/x86/orc-unwinder.html [2] https://lore.kernel.org/all/20170602104048.jkkzssljsompjdwy@... [3] https://www.phoronix.com/review/fedora-frame-pointer/5

Re: Frame pointers vs. DWARF – my verdict

#42

Post by irogers » Wed, Feb 15, 2023, 4:57 PM UTC

zznzz wrote: post #23

titzer wrote: post #16

> But collecting the whole stack would consume far too much storage, so by default it only collects the first 8K. Many userspace stacks will be larger than this, in which case the data collection will simply be incomplete – it will never be possible to recover the full stack trace. Yeah, doing an 8KB memcopy for every profile sample sounds like a lot of overhead. Is DWARF unwinding so slow that that is actually faste…

> Is DWARF unwinding so slow that that is actually faster? No. The only reason it works like this is because the upstream Linux kernel has thus far rejected in-kernel dwarf unwinders, but copying the stack is simpler and available / implemented.

DWARF bytecode is a full VM. Do compiler writers test their DWARF output? (my experience is not - especially for architectures out of the big 2 or 3) How does the kernel access the ELF file pages with the DWARF information in when in an NMI handler? You could mlock all your debug information when a program loads but the memory overhead wouldn't be nice. It is hard enough getting a build ID.

The elephant in the room btw is LBR call stacks, but they aren't exposed in the kernel/BPF yet. Userland perf has them and they recently became available on AMD.

Re: Frame pointers vs. DWARF – my verdict

#43

Post by irogers » Wed, Feb 15, 2023, 5:03 PM UTC

fooblaster wrote: post #22

There is an option far better than either suggested. In my experience using the --callgraph=lbr option produces far more reliable callstacks than relying on frame pointers. Sadly it's only available on Intel cpus at the moment.

AMD will have support in Zen4 and Linux 6.1 (which is LTS):

https://lore.kernel.org/lkml/Yz%2FcpNTSacRMh1FK@gmail.com/

Further, precise events are fixed in Linux 6.2:

https://lore.kernel.org/lkml/Y5eQeR2tpZ%2FBos49@gmail.com/

Re: Frame pointers vs. DWARF – my verdict

#44

Post by pjc50 » Wed, Feb 15, 2023, 5:03 PM UTC

irogers wrote: post #35

pjc50 wrote: post #9

Relevant discussion on profiling without frame pointers: https://twitter.com/halvarflake/status/1577644229853151233 / https://prodfiler.com/blog/introducing-prodfiler/ via the "prodfiler" tool.

Twitter: "No need to recompile with frame pointers." Web page: "always-on profiling powered by eBPF technology." BPF stack traces are gathered using frame pointers: https://github.com/torvalds/linux/blob/master/kernel/bpf/sta... https://github.com/torvalds/linux/blob/master/arch/x86/event...

Someone from prodfiler appears to be explaining in this thread https://news.ycombinator.com/item?id=34806693

Re: Frame pointers vs. DWARF – my verdict

#45

Post by irogers » Wed, Feb 15, 2023, 5:10 PM UTC

pjc50 wrote: post #44

irogers wrote: post #35

Earlier quoted context omitted.

Twitter: "No need to recompile with frame pointers." Web page: "always-on profiling powered by eBPF technology." BPF stack traces are gathered using frame pointers: https://github.com/torvalds/linux/blob/master/kernel/bpf/sta... https://github.com/torvalds/linux/blob/master/arch/x86/event...

Someone from prodfiler appears to be explaining in this thread https://news.ycombinator.com/item?id=34806693

For every running application turn DWARF data into BPF maps. Does this scale?

Re: Frame pointers vs. DWARF – my verdict

#46

Post by zznzz » Wed, Feb 15, 2023, 5:11 PM UTC

nneonneo wrote: post #41

the_mitsuhiko wrote: post #21

Earlier quoted context omitted.

> In any case, the runtime performance hit of frame pointers is quite high That's not true at all. It's in some very rare cases . Firefox I think at this point ships with framepointers enabled and so does every M1/M2 mac app and all iOS applications as it's mandatory for the calling convention on Apple.

The ORC documentation [1] cites a 5-10% perf hit [2] on x86. Now, granted, we're talking about an architecture that has only 8 logical registers to begin with, so losing one is going to hurt a lot more than x86-64 (16 logical registers) or AARCH64 (32 logical registers). But, 5-10% is definitely a noticeable perf hit. Phoronix [3] did a test of "-fno-omit-frame-pointer" during the discussion on whether to enable the…

See also https://lists.fedoraproject.org/archives/list/devel@lists.fe... - the botan phoronix results with frame pointers were probably measuring debug builds.

Re: Frame pointers vs. DWARF – my verdict

#47

Post by zznzz » Wed, Feb 15, 2023, 5:24 PM UTC

irogers wrote: post #42

zznzz wrote: post #23

Earlier quoted context omitted.

> Is DWARF unwinding so slow that that is actually faster? No. The only reason it works like this is because the upstream Linux kernel has thus far rejected in-kernel dwarf unwinders, but copying the stack is simpler and available / implemented.

DWARF bytecode is a full VM. Do compiler writers test their DWARF output? (my experience is not - especially for architectures out of the big 2 or 3) How does the kernel access the ELF file pages with the DWARF information in when in an NMI handler? You could mlock all your debug information when a program loads but the memory overhead wouldn't be nice. It is hard enough getting a build ID. The elephant in the room b…

It is not required to unwind the user space stack in the NMI handler. It can be done later before returning to user space in a context that can handle faults.

Re: Frame pointers vs. DWARF – my verdict

#48

Post by irogers » Wed, Feb 15, 2023, 5:28 PM UTC

zznzz wrote: post #47

irogers wrote: post #42

Earlier quoted context omitted.

DWARF bytecode is a full VM. Do compiler writers test their DWARF output? (my experience is not - especially for architectures out of the big 2 or 3) How does the kernel access the ELF file pages with the DWARF information in when in an NMI handler? You could mlock all your debug information when a program loads but the memory overhead wouldn't be nice. It is hard enough getting a build ID. The elephant in the room b…

It is not required to unwind the user space stack in the NMI handler. It can be done later before returning to user space in a context that can handle faults.

Allowing processes to sniff each others stacks has some fairly obvious security issues.

Re: Frame pointers vs. DWARF – my verdict

#49

Post by zznzz » Wed, Feb 15, 2023, 5:30 PM UTC

titzer wrote: post #33

zznzz wrote: post #23

Earlier quoted context omitted.

> Is DWARF unwinding so slow that that is actually faster? No. The only reason it works like this is because the upstream Linux kernel has thus far rejected in-kernel dwarf unwinders, but copying the stack is simpler and available / implemented.

Why is profiling done in the kernel for userspace stacks?

because this is about PMU based sampling, which involves triggering interrupts at some interval and doing the sampling while handling the interrupt

Re: Frame pointers vs. DWARF – my verdict

#50

Post by zznzz » Wed, Feb 15, 2023, 5:31 PM UTC

irogers wrote: post #48

zznzz wrote: post #47

Earlier quoted context omitted.

It is not required to unwind the user space stack in the NMI handler. It can be done later before returning to user space in a context that can handle faults.

Allowing processes to sniff each others stacks has some fairly obvious security issues.

I don’t understand your concern - what about this would involve one process sniffing another process’s memory? The kernel would still be doing the unwinding, just not in the NMI handler.