Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Frame pointers vs. DWARF – my verdict

rwmj.wordpress.com

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Frame pointers vs. DWARF – my verdict

#61

Post by amluto »

javierhonduco wrote: post #55

The pervasive lack of frame pointers is the reason why we've developed a custom format derived from DWARF unwind information thanks to some insights: DWARF unwind information is incredibly flexible, it supports many architecture and allows restoring any arbitrary register. But we only need 3: the frame pointer, the stack pointer, and in non-x86 the return address. While DWARF unwind info doesn't use that many bytes,…

Nice!

One suggestion: binary search has extremely poor cache behavior, and early versions of the ORC in winder (IIRC) spent considerably more time binary searching the table than actually unwinding.

There are many solutions to this. ORC (IIRC) uses a flat hint table mapping PC -> offset in the table. It’s sparse, so you look up hint[(ip-base) / divisor] and its successor to find a small range of the table to search. (Divisor is set to keep the hint table compact but still limit the main search to something small.). This gives essentially linear time lookups with simple code.

You can also use a B-tree or a similar structure. B-trees are pretty straightforward if you don’t ever need to modify them.

IIRC this gave a substantial speedup.

Re: Frame pointers vs. DWARF – my verdict

#62

Post by javierhonduco »

amluto wrote: post #61
javierhonduco wrote: post #55

The pervasive lack of frame pointers is the reason why we've developed a custom format derived from DWARF unwind information thanks to some insights: DWARF unwind information is incredibly flexible, it supports many architecture and allows restoring any arbitrary register. But we only need 3: the frame pointer, the stack pointer, and in non-x86 the return address. While DWARF unwind info doesn't use that many bytes,…

Nice! One suggestion: binary search has extremely poor cache behavior, and early versions of the ORC in winder (IIRC) spent considerably more time binary searching the table than actually unwinding. There are many solutions to this. ORC (IIRC) uses a flat hint table mapping PC -> offset in the table. It’s sparse, so you look up hint[(ip-base) / divisor] and its successor to find a small range of the table to search.…

This is a very good point. Thanks for the context on how ORC does it, I was not familiar with how it evolved.

In our case this is not an issue yet. We split up the unwind tables in chunks of up to ~1MB so we typically have in very few L2 cache misses.

This has to be done once per frame so in my 4 year old i7 processor. There are roughly `2 L2 cache misses (+ few other misses from reading ancillary data structures) * frames`.

We have more work to do regarding benchmarks but I collected some numbers the other day and 90 frames can be walked in less than 500ns (slide 50 https://fosdem.org/2023/schedule/event/walking_stack_without...)

There are more optimisations we have in the works to make our unwinder more efficient, mostly related to fitting more data in the CPU cache and reducing cache misses.

Re: Frame pointers vs. DWARF – my verdict

#63

Post by fooblaster »

irogers wrote: post #43
fooblaster wrote: post #22

There is an option far better than either suggested. In my experience using the --callgraph=lbr option produces far more reliable callstacks than relying on frame pointers. Sadly it's only available on Intel cpus at the moment.

AMD will have support in Zen4 and Linux 6.1 (which is LTS): https://lore.kernel.org/lkml/Yz%2FcpNTSacRMh1FK@gmail.com/ Further, precise events are fixed in Linux 6.2: https://lore.kernel.org/lkml/Y5eQeR2tpZ%2FBos49@gmail.com/

What does precise events in perf help with?

Re: Frame pointers vs. DWARF – my verdict

#64

Post by tdullien »

irogers wrote: post #51
tdullien wrote: post #40

What everybody in this discussion seems to miss is that you don't need to unwind the DWARF data structures during profiling time , you are free to convert DWARF to a fast-lookup data structure on the machine. DWARF needs to support every CPU under the sun. Every unwinder on the other hand is CPU-specific. For prodfiler.com's continuous in-production unwinding, we convert DWARF into something compact and fast-to-looku…

prodfiler clearly has a market. It would be interesting to see the approach as something standard in the kernel tree, perhaps it can be added to perf's synthesis, etc. There is already BPF based profiling within perf to avoid file descriptor overheads. If engineering resources are the issue then this could be a good GSoC project: https://wiki.linuxfoundation.org/gsoc/2023-gsoc-perf

Yeah. I really like the ideas proposed by Brendan Gregg -- essentially encouraging every HLL runtime to embed an eBPF-based unwinder in it's own executable. The upshot of that would be "generic, in-production unwinding of native code and HLL code", similar to what prodfiler is doing, but inside the main kernel tree...

Re: Frame pointers vs. DWARF – my verdict

#65

Post by lathiat »

ishitatsuyuki wrote: post #4

IMHO, perf's decision to write whole stacks directly to the disk and unwinding them as a post-process is a really bad design. It wastes disk space, and as the author pointed out, it also has a lot of IO overhead. As an alternative approach, https://github.com/mstange/samply processes data streamed from perf and unwinds it in realtime. The unwinding overhead is surprisingly low: it only takes around 1% of (single) CPU…

Parca also have done work to unwind DWARF in kernel with eBPF: https://www.polarsignals.com/blog/posts/2022/11/29/profiling...

Edit: refer to another comment in this thread: https://news.ycombinator.com/item?id=34809265

Re: Frame pointers vs. DWARF – my verdict

#66

Post by lathiat »

the_mitsuhiko wrote: post #14
conconpon wrote: post #10

Earlier quoted context omitted.

Couldn't this be solved by uploading the binaries along with the crash dumps, if you don't already have a copy of it, as determined by checking hashes or something?

Exactly, except debuginfod from Canonical and some others misses the executables.

I would expect them to be there but debuginfod for Ubuntu is very new and did not (at least currently) import older releases (including the current LTS) or older package versions. Which is maybe what you’re seeing. Should work better with the most recent release only.

Do you have a specific example of which executables are missing?

There are also other ways to get all the debug symbols but it’s much fiddlier and not nearly as nice as debuginfod. Data’s all there, finding it is harder. You can query the apt index for buildids but only for the currently released package version and not superseded ones.

There were discussions about importing all the history and a way to do it was determined but not sure where the implementation of that has gotten too.

Re: Frame pointers vs. DWARF – my verdict

#67

Post by titzer »

zznzz wrote: post #49
titzer wrote: post #33

Earlier quoted context omitted.

Why is profiling done in the kernel for userspace stacks?

because this is about PMU based sampling, which involves triggering interrupts at some interval and doing the sampling while handling the interrupt

Other than overhead, what is the advantage as opposed to handling the interrupt in the kernel and then delivering a signal to userspace? After all, isn't this the role of SIGPROF?
Post reply on HN