Frame pointers vs. DWARF – my verdict
rwmj.wordpress.com
Frame pointers vs. DWARF – my verdict
1–10 of 67 posts
Re: Frame pointers vs. DWARF – my verdict
#2Nice analysis.
It would be interesting to see comparison to Intel LBR.
Also would be nice to know how profiling unwinding is done on Windows (maybe someone knows how to summon Bruce Dawson).
Re: Frame pointers vs. DWARF – my verdict
#3> Frame pointers have some corner cases which they don’t handle well (certain leaf and most inlined functions aren’t collected), but these don’t matter a great deal in reality.
> DWARF unwinding can show inlined functions as if they are separate stack frames. (Opinions differ as to whether or not this is an advantage.)
This conflates unwinding and symbolization. Unwinding collects the list of frames, which by definition cannot have inlined functions (functions that don't have their own frame); the unwinding mechanism does not matter here.
DWARF can be used to resolve the "stack" of inlined functions for an instruction address, even if that was collected via frame pointers. That can be done in post-processing, possibly on a different machine, so the cost does not affect the workload being profiled.
For example, using addr2line from the LLVM distribution
$ llvm-addr2line -pfi --demangle -e /path/to/unstripped/binary
# Enter instruction address in the form 0x...
# Outputs inlined stack
And on whether reporting inlined functions is important or not, at least for C++, which heavily relies on inlining to mitigate abstraction penalty, I'd say it is crucial.
Re: Frame pointers vs. DWARF – my verdict
#4IMHO, perf's decision to write whole stacks directly to the disk and unwinding them as a post-process is a really bad design. It wastes disk space, and as the author pointed out, it also has a lot of IO overhead.
As an alternative approach, https://github.com/mstange/samply processes data streamed from perf and unwinds it in realtime. The unwinding overhead is surprisingly low: it only takes around 1% of (single) CPU per CPU profiled. Solving the disk waste alone has been a tremendous improvement of profiling experience. As a bonus, the unwinding and symbolization works reliably while I frequently had postprocessing not terminating when using the perf CLI directly.
Re: Frame pointers vs. DWARF – my verdict
#5Nice analysis. It would be interesting to see comparison to Intel LBR. Also would be nice to know how profiling unwinding is done on Windows (maybe someone knows how to summon Bruce Dawson).
AIUI the problems with LBR are two-fold. It only works on newish CPUs, and it only handles a limited number of stack frames (I heard 8, but maybe more on recent CPUs).
Re: Frame pointers vs. DWARF – my verdict
#6Big advantage for DWARF2 over frame pointers is that it actually works for unwinding on aarch64.
Re: Frame pointers vs. DWARF – my verdict
#7> Frame pointers have some corner cases which they don’t handle well (certain leaf and most inlined functions aren’t collected), but these don’t matter a great deal in reality. > DWARF unwinding can show inlined functions as if they are separate stack frames. (Opinions differ as to whether or not this is an advantage.) This conflates unwinding and symbolization. Unwinding collects the list of frames, which by definit…
> To summarize, unwinding via frame pointers does not miss any information that would be collected with DWARF unwinding. Everything can be recovered later at symbolization time.
The real issue with DWARF based unwinding/symbolication is that it's really complex on the user experience around it. We at Sentry support stack walking from minidumps, yet we often cannot unwind on Linux platforms on the server because executables and object files are typically not available.
Despite us supporting debuginfod (we're hitting the canonical service today only) we are unable to collect executables/object files from there, making it completely impossible to produce proper stack traces if frame pointers are omitted.
In a world were DWARF unwinding is a thing and people want to use, there has to be an ecosystem of sharing binaries too. This issue is particular bad on Android where there are millions of devices out there with tons of different proprietary system libraries linked in, destroying stacks.
Re: Frame pointers vs. DWARF – my verdict
#8Big advantage for DWARF2 over frame pointers is that it actually works for unwinding on aarch64.
Apple's version of AARCH64 requires that the frame pointer register frame record. DWARF Information is not necessary to unwind there. It's lovely and I applaud Apple for that decision.
Re: Frame pointers vs. DWARF – my verdict
#9Relevant discussion on profiling without frame pointers: https://twitter.com/halvarflake/status/1577644229853151233 / https://prodfiler.com/blog/introducing-prodfiler/ via the "prodfiler" tool.
Re: Frame pointers vs. DWARF – my verdict
#10> Frame pointers have some corner cases which they don’t handle well (certain leaf and most inlined functions aren’t collected), but these don’t matter a great deal in reality. > DWARF unwinding can show inlined functions as if they are separate stack frames. (Opinions differ as to whether or not this is an advantage.) This conflates unwinding and symbolization. Unwinding collects the list of frames, which by definit…
> To summarize, unwinding via frame pointers does not miss any information that would be collected with DWARF unwinding. Everything can be recovered later at symbolization time. The real issue with DWARF based unwinding/symbolication is that it's really complex on the user experience around it. We at Sentry support stack walking from minidumps, yet we often cannot unwind on Linux platforms on the server because execu…
Couldn't this be solved by uploading the binaries along with the crash dumps, if you don't already have a copy of it, as determined by checking hashes or something?