Flame Graphs: Making the opaque obvious (2017)
tech.popdata.org
Flame Graphs: Making the opaque obvious (2017)
1–10 of 58 posts
Re: Flame Graphs: Making the opaque obvious (2017)
#2OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!)
I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, and none of that matches the conclusion reached in the blog post?
And then I tried to grok all the knowledge conveyed in https://www.brendangregg.com/flamegraphs.html and... came away similarly confused? Sure, I see patterns, but when asked to identify which of those patterns indicate problems, I would still score And, I mean, I do memory/CPU traces like all day every day, and I fix code all the time based on that, but that's all just based on hierarchical bar charts, which are NOT the same as inverted flame graphs, as far as I can tell?
Anyway, thanks for coming to my therapy session, and feel free to leave your helpful-comment-that-will-finally-enlighten-me and/or downvote at the door.
Re: Flame Graphs: Making the opaque obvious (2017)
#3OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
I think I'm the same as you, I much prefer the style where you get an interactive profile and can hit "show me the hot path" and it'll expand down the hot path.
The overall flame charts just don't provide the immediate "Here's what's wrong" you need from a chart.
Charts should be simple enough to quickly get an idea of whether the thing you're looking at is "Good" or "bad" and if "Bad", quickly see where the problem is.
Re: Flame Graphs: Making the opaque obvious (2017)
#4OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
I think I'm the same as you, I much prefer the style where you get an interactive profile and can hit "show me the hot path" and it'll expand down the hot path. The overall flame charts just don't provide the immediate "Here's what's wrong" you need from a chart. Charts should be simple enough to quickly get an idea of whether the thing you're looking at is "Good" or "bad" and if "Bad", quickly see where the problem…
Well, thank you for pointing out that hierarchical bar charts do a pretty good job at showing the hot path.
So my (and I guess your) workflow is then to fix the top-X hot paths, and then re-profile, etc.
The attraction of flame graphs, as I understand it, is that you can get the same information without the need to iterate. Which is obviously very valuable if you're operating on single-shot bug reports, but... how exactly? (Not a question directed at you, obviously, but more towards the general discussion)
Re: Flame Graphs: Making the opaque obvious (2017)
#5OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
> And then I tried to grok all the knowledge conveyed in https://www.brendangregg.com/flamegraphs.html
Have you watched the LISA 2013 presentation video?
* https://www.youtube.com/watch?v=nZfNehCzGdw
* https://www.usenix.org/conference/lisa13/technical-sessions/...
Some follow-on presentations from him, I found the first one useful:
Re: Flame Graphs: Making the opaque obvious (2017)
#6Earlier quoted context omitted.
I think I'm the same as you, I much prefer the style where you get an interactive profile and can hit "show me the hot path" and it'll expand down the hot path. The overall flame charts just don't provide the immediate "Here's what's wrong" you need from a chart. Charts should be simple enough to quickly get an idea of whether the thing you're looking at is "Good" or "bad" and if "Bad", quickly see where the problem…
Well, thank you for pointing out that hierarchical bar charts do a pretty good job at showing the hot path. So my (and I guess your) workflow is then to fix the top-X hot paths, and then re-profile, etc. The attraction of flame graphs, as I understand it, is that you can get the same information without the need to iterate. Which is obviously very valuable if you're operating on single-shot bug reports, but... how ex…
I recommend writing your own flame graph generator, the syntax is super simple and it will help you understand the span/subspan nesting.
In some cases it's not that helpful if you don't have the right type of problem, eg I was troubleshooting some higher level stuff (SQL) and looking at stored procedure invocations and a two depth invocation doesn't tell you very much.
For me a FG is more about sounding out the problem and understanding what the scope of what is possible more than fixing an immediate and obvious issue.
Re: Flame Graphs: Making the opaque obvious (2017)
#7OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
> I look at the reddest part of the chart, I look at the peaks
Neither of these are really the places I look at when examining flame graphs. I tend to look at the bottom, and work my way up.
The key thing (imo) to look for are wide pieces that are not a core part of what the code you're profiling is supposed to do.
In the first example of your first link, you have a flame graph of code that seems to draw an image. There are a bunch of functions like 'CopyPixels' and 'RasterizeEdges'. Those seem to be a core part of drawing an image. In contrast, there's a 'KiPageFault' function that's pretty wide that does not seem to be important to the code's overall goal. I think a bit more context is needed to arrive at the author's conclusion, though.
I think flame graphs, like all graphs, are more helpful when the reader has a lot of context about what's supposed to happen, or some intuition about how the chart is supposed to look.
Re: Flame Graphs: Making the opaque obvious (2017)
#8OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
> I do memory/CPU traces like all day every day, and I fix code all the time based on that
So I take it you understand one of the standard visualizations produced by a CPU profiling tool, e.g. it takes a call stack sample every millisecond. The x-axis is time (one sample per ms, if you have only one CPU), and the y-axis is the call stack.
Now for a flamegraph, you basically have the same visualization, but you sort the samples across the x-axis so that callstacks that start with the same prefixes are grouped together.
Incidentally, the sorting of samples across the x-axis destroys the time information which is often critical. I constantly see engineers who don't understand flamegraphs looking at the entire flamegraph of a say 3 second trace, and then trying to use that whole flamegraph to optimize a 100 ms critical path within the trace, which is totally nonsensical.
Re: Flame Graphs: Making the opaque obvious (2017)
#9Earlier quoted context omitted.
I think I'm the same as you, I much prefer the style where you get an interactive profile and can hit "show me the hot path" and it'll expand down the hot path. The overall flame charts just don't provide the immediate "Here's what's wrong" you need from a chart. Charts should be simple enough to quickly get an idea of whether the thing you're looking at is "Good" or "bad" and if "Bad", quickly see where the problem…
Well, thank you for pointing out that hierarchical bar charts do a pretty good job at showing the hot path. So my (and I guess your) workflow is then to fix the top-X hot paths, and then re-profile, etc. The attraction of flame graphs, as I understand it, is that you can get the same information without the need to iterate. Which is obviously very valuable if you're operating on single-shot bug reports, but... how ex…
The jetbrains profiler lets you quickly "Scope to X", so you can scope to the second or third highest ranked functions and see their breakdown too, so you don't need to fix the highest to get a full understanding.
Re: Flame Graphs: Making the opaque obvious (2017)
#10OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
> I look at the reddest part of the chart, I look at the peaks Neither of these are really the places I look at when examining flame graphs. I tend to look at the bottom, and work my way up. The key thing (imo) to look for are wide pieces that are not a core part of what the code you're profiling is supposed to do. In the first example of your first link, you have a flame graph of code that seems to draw an image. Th…
> I think flame graphs, like all graphs, are more helpful when the reader has a lot of context about what's supposed to happen, or some intuition about how the chart is supposed to look.
Yeah, and then your comment... just ends? So, what I get here is that, in a flame graph, the reddest part isn't the most interesting, and neither is the widest part, nor the part with the most peaks.
So, what, exactly, am I looking for? "You know what this graph should look like in case all was OK" is not exactly helpful, right, because I might just lack that baseline?