For C++ applications, nothing beats Tracy’s flame graphs in value per dollar. We are using it at Adobe with great effect. It can handle mutex lock contention, runtime values and notes, arbitrary value graphs, and more. Highly recommended: https://github.com/wolfpld/tracy
Flame Graphs: Making the opaque obvious (2017)
51–58 of 58 posts
Re: Flame Graphs: Making the opaque obvious (2017)
#52Back in 2018, I built a flame graph generator for AppDynamics APM-captured snapshots which are traces of call stacks captured over execution time with stack frame timing. Those would typically come from java or clr, although other types like node and ruby and php were also supported.
My tool could handle one snapshot or hundreds of thousands of them and it was more useful to aggregate many snapshots I used on Brandan Gregg perl script implementation as reference but wrote all the code myself from scratch to produce just what I wanted. One of the more key things I felt was different was was adding color coding of the TYPE of the call frame based off the code namespace - like green for Oracle jdbc driver or blue for websphere entry or light azure for the clr internals. With multiple contrasting colors one could see the important transitions from own code to library code and back to own code.
It was one of the most fun coding challenges I've ever done. Pretty much the only time I had to refactor recursion into stack because some of the call graphs (looking at you java) were so ridiculously deep
https://github.com/Appdynamics/AppDynamics.DEXTER/wiki/Flame...
Re: Flame Graphs: Making the opaque obvious (2017)
#53Earlier quoted context omitted.
As an example, imagine you sampled a program and got 5 CPU call stack samples. c c b b d a a a a a main main main main main In a flamegraph, you would see: [c ] [b ][d ] [a ] [main ]
Yeah, I imagine it, and still don't see how the flame graph would help? Shown as a hierarchical bar chart, this would suggest 'b' is problematic. Where, color-wise (because peak-wise, 'c' would be the culprit here) do I see this issue in a flame graph? Because I fear that either 'main' or 'a' would have the most dominant shade of red here?
> Shown as a hierarchical bar chart, this would suggest 'b' is problematic.
But it's not necessarily `b` that's problematic:
- it may be `a` because it does a lot of stuff on its own, and depending on what `a` is, it might not be expected
- it could be `d` if it's supposed to be super fast (e.g. a logging method)
- it could be `c` because it takes a long time
- it could be `b` if `c` is external code or if calling `c` from `b` is not appropriate for what `b` does
- it might be nothing because there's nothing to optimize anymore, things just take this long
in fact, `b` is the last method I'd look at here
Re: Flame Graphs: Making the opaque obvious (2017)
#54Earlier quoted context omitted.
> How do flame graphs handle the case where most of the time is spent in some leaf function that is called from all over the program? In my experience, they don't really. They're very good for finding easy wins where a high/medium-level operation takes much longer than you'd expect it to, not so much for finding low-level hotspots in the code. > such a view would [...] subsume the usual use cases for a flame graph, w…
OTOH, hitting Ctrl-C in gdb and looking at a backtrace a few times would show you the leaf function
Doesn’t perf also support reverse callgraphs? I seem to remember doing this pretty easily with that.
Re: Flame Graphs: Making the opaque obvious (2017)
#55How do flame graphs handle the case where most of the time is spent in some leaf function that is called from all over the program? In this case, each individual stack would not take much time but in aggregate, a lot of time is spent in the function at the top of all of the call stacks. This should not be that uncommon to have hotspots in things like copying routines, compression, encryption etc that are not associat…
I like speedscope.app for viewing flamegraphs. It is more interactive than traditional SVG flamegraphs, and what is relevant here is a "sandwich" view - basically a sorted list of all functions, you see what function was spent the most time in, click on it and see all calling stack traces like a mini flamegraph, filtered and centered on this function. Speedscope supports several popular trace formats, really useful.
I like perfetto
Re: Flame Graphs: Making the opaque obvious (2017)
#56Earlier quoted context omitted.
> I look at the reddest part of the chart, I look at the peaks Neither of these are really the places I look at when examining flame graphs. I tend to look at the bottom, and work my way up. The key thing (imo) to look for are wide pieces that are not a core part of what the code you're profiling is supposed to do. In the first example of your first link, you have a flame graph of code that seems to draw an image. Th…
> I think flame graphs, like all graphs, are more helpful when the reader has a lot of context about what's supposed to happen, or some intuition about how the chart is supposed to look. Yeah, and then your comment... just ends? So, what I get here is that, in a flame graph, the reddest part isn't the most interesting, and neither is the widest part, nor the part with the most peaks. So, what, exactly , am I looking…
? So, what, exactly, am I looking for?
You're looking for a wide part for a function that you don't expect to have a wide part.
So start at the widest part and work your way up each hill. If you see a function that's still relatively wide that you think should not be wide, that's when you go look into why it's wide.
Re: Flame Graphs: Making the opaque obvious (2017)
#57OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
I was confused by them in the past, too.
My problem was that I had used tools that use the x-axis for time. So you can follow program execution over time by reading the graph from left to right.
However, that is NOT how flame graphs work. At each level of stack depth the called functions are sorted alphabetically and the cumulated time spent in the function is represented as the length of the bar in x-direction. So yes, x has a notion of time or duration, but what you see is not chronologically ordered data.
Understanding that brought the breakthrough for me.
Re: Flame Graphs: Making the opaque obvious (2017)
#58OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
> I look at the reddest part of the chart,
At least in the original form colors are randomly assigned just for visual separation. So looking for a certain color is not helpful. You should look for long bars.
(There are obviously also enhancements where colors have some semantics. I am not familiar with those.)