Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Frame pointers vs. DWARF – my verdict

rwmj.wordpress.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Frame pointers vs. DWARF – my verdict

#51

Post by irogers »

tdullien wrote: post #40

What everybody in this discussion seems to miss is that you don't need to unwind the DWARF data structures during profiling time , you are free to convert DWARF to a fast-lookup data structure on the machine. DWARF needs to support every CPU under the sun. Every unwinder on the other hand is CPU-specific. For prodfiler.com's continuous in-production unwinding, we convert DWARF into something compact and fast-to-looku…

prodfiler clearly has a market. It would be interesting to see the approach as something standard in the kernel tree, perhaps it can be added to perf's synthesis, etc. There is already BPF based profiling within perf to avoid file descriptor overheads. If engineering resources are the issue then this could be a good GSoC project: https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf

Re: Frame pointers vs. DWARF – my verdict

#52

Post by irogers »

zznzz wrote: post #50
irogers wrote: post #48

Earlier quoted context omitted.

Allowing processes to sniff each others stacks has some fairly obvious security issues.

I don’t understand your concern - what about this would involve one process sniffing another process’s memory? The kernel would still be doing the unwinding, just not in the NMI handler.

Wouldn't all your kernel stacks then end up in whatever this handler is? Why not implement your approach and mail it to LKML :-)

Re: Frame pointers vs. DWARF – my verdict

#53

Post by gpderetta »

remexre wrote: post #39
yvdriess wrote: post #34

Earlier quoted context omitted.

nitpick: it's the number of physical registers that matter more here, not ISA registers

Why wouldn't the lower ISA register count lead to more spills, regardless of the physical count? Is store forwarding really that effective?

It isn't.

Re: Frame pointers vs. DWARF – my verdict

#54

Post by zznzz »

irogers wrote: post #52
zznzz wrote: post #50

Earlier quoted context omitted.

I don’t understand your concern - what about this would involve one process sniffing another process’s memory? The kernel would still be doing the unwinding, just not in the NMI handler.

Wouldn't all your kernel stacks then end up in whatever this handler is? Why not implement your approach and mail it to LKML :-)

Yes, this only works for user space stacks, but that is sufficient since with ORC kernel stacks are solved (IMO) and it avoids all the issues with trying to mlock debuginfo of all processes that you mentioned. The NMI handler would still unwind the kernel stack.

> Why not implement your approach and mail it to LKML :-)

because this would still be an in-kernel dwarf unwinder and I would expect an instant reject, and because I am lazy and/or don’t care enough about this problem or linux to work on it. Even if people could be persuaded, I don’t have the interest or temperance to debate this with LKML.

Re: Frame pointers vs. DWARF – my verdict

#55

Post by javierhonduco »

The pervasive lack of frame pointers is the reason why we've developed a custom format derived from DWARF unwind information thanks to some insights: DWARF unwind information is incredibly flexible, it supports many architecture and allows restoring any arbitrary register. But we only need 3: the frame pointer, the stack pointer, and in non-x86 the return address.

While DWARF unwind info doesn't use that many bytes, but unfortunately reading and parsing that information is quite expensive.

For that reason I've developed a new unwinder that uses custom unwind information derived from DWARF (https://www.polarsignals.com/blog/posts/2022/11/29/profiling..., previously discussed in https://news.ycombinator.com/item?id=33788794) that runs in BPF. This new compact representation can be binary searched easily and each unwind row has a size of 16 bytes. I am currently working on reducing it down to ~10 bytes.

All the code is fully OSS (Apache 2.0 for userspace and GPL for BPF), and part of the Parca project (https://github.com/parca-dev/parca-agent). We've also given a talk this year in FOSDEM going deeper into how we made it scale for many big processes.

Re: Frame pointers vs. DWARF – my verdict

#56

Post by javierhonduco »

irogers wrote: post #51
tdullien wrote: post #40

What everybody in this discussion seems to miss is that you don't need to unwind the DWARF data structures during profiling time , you are free to convert DWARF to a fast-lookup data structure on the machine. DWARF needs to support every CPU under the sun. Every unwinder on the other hand is CPU-specific. For prodfiler.com's continuous in-production unwinding, we convert DWARF into something compact and fast-to-looku…

prodfiler clearly has a market. It would be interesting to see the approach as something standard in the kernel tree, perhaps it can be added to perf's synthesis, etc. There is already BPF based profiling within perf to avoid file descriptor overheads. If engineering resources are the issue then this could be a good GSoC project: https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf

This would be ideal. There's some great work by folks at Oracle in this space: SFrame (https://www.phoronix.com/news/GNU-Binutils-SFrame) née ctf_frame that I hope will be integrated in the kernel.

As this will take few years, in the meantime I've developed a DWARF-based unwinder in BPF [0]. Some perf maintainers showed interest in this, so thanks for bringing up the GSoC project idea, didn't occur to me!

[0]: https://news.ycombinator.com/item?id=33788794

Re: Frame pointers vs. DWARF – my verdict

#57

Post by javierhonduco »

irogers wrote: post #45
pjc50 wrote: post #44

Earlier quoted context omitted.

Someone from prodfiler appears to be explaining in this thread https://news.ycombinator.com/item?id=34806693

For every running application turn DWARF data into BPF maps. Does this scale?

As surprising as it seems, it does! My colleague and I spoke about this in our project in FOSDEM https://fosdem.org/2023/schedule/event/walking_stack_without.... Let us know if you have any questions or feedback :)

Re: Frame pointers vs. DWARF – my verdict

#58

Post by js2 »

Not having frame pointers is a complete PITA on Android.

I built my company's in-house mobile crash reporter ~ 2013, based on experience building a similar system at an earlier startup. At the startup I used Google breakpad on both Android and iOS, doing all the unwinding and symbolication on the backend. iOS at least made this easy because Apple makes dSYMs readily available. On Android, you simply can't get system symbols. So you essentially can't unwind OS symbols on Android.

At current company I used PLCrashReporter on iOS. Unwinding occurs on device. Symbolication on the backend.

I tried everything with Android for native code crashes, starting with breakpad minidumps, then using every available unwinder option on Android: corkscrew, the Android fork of libunwind, the official libunwind, whatever custom unwinder Android eventually wrote. None of them work reliably for native code crashes. And good luck tracing from native code back into the ART frames.

In the end what ending up being most reliable was including the last few thousand lines of logcat (grabbed upon the app restarting after a crash since you can't reliably grab it inside the crash handler). Android's OOB crash handler for some crashes (with recent Android versions) dumps full stacks of every thread including native code and ART frames to logcat. So the crash SDK looks for that in the logcat output and includes it in the report. That at least provides a stack. Symbolicating anything but application frames is still impossible though.

And this isn't even going into esoteric things Android has done over the years just for Chrome like relocation packing:

https://android.googlesource.com/platform/bionic/+/f5e0ba94d...

To this day, I don't understand why both Apple and Google make it so difficult for an application to get access to the stack traces of its own crashes. And no, the reporting built-in to Google Play and iTunes Connect (and Xcode) are not sufficient for large usage apps or companies like mine that have lots of apps with shared SDKs and need to correlate crashes across apps.

Re: Frame pointers vs. DWARF – my verdict

#59

Post by sitkack »

ishitatsuyuki wrote: post #4

IMHO, perf's decision to write whole stacks directly to the disk and unwinding them as a post-process is a really bad design. It wastes disk space, and as the author pointed out, it also has a lot of IO overhead. As an alternative approach, https://github.com/mstange/samply processes data streamed from perf and unwinds it in realtime. The unwinding overhead is surprisingly low: it only takes around 1% of (single) CPU…

Are you saying that Dwarf information should be unwound in realtime or that it should use framepointers and debug information to trivially sample the stacks and record the symbols?

If you have framepointers and debug information, it is both high resolution and fast. DWARF is a fallback for not having framepointers.

If you are saying the DWARF information should be processed at the point of use and not copied and processed later, then I concur. But we should also encourage folks to compiled WITH `-fno-omit-frame-pointer` and `-g`

Re: Frame pointers vs. DWARF – my verdict

#60

Post by sitkack »

DaveFlater wrote: post #32
zznzz wrote: post #29

Earlier quoted context omitted.

I cannot comment on whether “everyone” is oblivious but yes, this is still the case - frame pointer based unwinding sometimes skips the caller when the IP is sampled before the callee sets up a frame. This is also common for samples in leaf functions. compiler & tool chain folks tend to think (quite justifiably imo) that this and similar stuff is fine because dwarf allows reconstructing everything perfectly. The prob…

AFAIK, the optimization undermined the only use case that -fno-omit-frame-pointer actually had on x86_64. Is there a real use case that benefitted from allowing the frame pointer push to wander? Why why why

That is too bad, it does mention `-fno-schedule-insns2` in the comments. I'll have to see how well that works in practice.

Having high quality low cost stack straces is important for continuous profiling. ref Knuth.

Post reply on HN