Just to advertise the perf tool has inbuilt flamegraph generation code these days (well leaning on D3.js). So `perf script report flamegraph` will convert a perf.data file into a flamegraph.html. Similarly there is `perf script report gecko` to write out the firefox profiler's json format.
Flame Graphs: Making the opaque obvious (2017)
31–40 of 58 posts
Re: Flame Graphs: Making the opaque obvious (2017)
#32Earlier quoted context omitted.
Peaks don't matter, they just correspond to the depth of the call stack. Probably the simplest way to use the flame graph is work from the bottom of the flamegraph and walk upwards until you find something interesting you optimize. Ideally you find something wide to optimize that makes sense. (The widest thing here is "main" which is obviously probably not the interesting thing to optimize, so you would work upwards…
> Probably the simplest way to use the flame graph is work from the bottom of the flamegraph and walk upwards until you find something interesting you optimize OK, so going by what is apparently the 'simple example' in the linked article: https://tech.popdata.org/images/cps1970_before_fix_dwarf_gcc... I work my way up. First thing that is really red is Conversion::Process::Run, but that probably wraps a lot of things…
Other user here: Confession - I don't actually know what, if anything, the colors mean in a flamegraph. They seem random to me.
The way I'd personally hone in on Record::hasVariable is that it's a relatively-simple sounding function (from the name) that is taking a large portion of the X-axis. Starting at the bottom, I'd go "main -> editInOrder -> relateEdits -> countPeopleMatching -> getSourceDataAsLong -> hasVariable." Then I'd be like "we really spend 47% of our time in this simple-sounding function? What's it doing?"
Basically, I look for the functions that have an outsized complexity/time ratio. A function with a simple task is usually easier to optimize, and a function that only runs for 2% of your program isn't worth spending the time to optimize.
Re: Flame Graphs: Making the opaque obvious (2017)
#33Earlier quoted context omitted.
Peaks don't matter, they just correspond to the depth of the call stack. Probably the simplest way to use the flame graph is work from the bottom of the flamegraph and walk upwards until you find something interesting you optimize. Ideally you find something wide to optimize that makes sense. (The widest thing here is "main" which is obviously probably not the interesting thing to optimize, so you would work upwards…
> Probably the simplest way to use the flame graph is work from the bottom of the flamegraph and walk upwards until you find something interesting you optimize OK, so going by what is apparently the 'simple example' in the linked article: https://tech.popdata.org/images/cps1970_before_fix_dwarf_gcc... I work my way up. First thing that is really red is Conversion::Process::Run, but that probably wraps a lot of things…
Indeed, the flame chart can't tell you that.
The solution provided in the article seems to rip out `Metadata::Cache::getVarsByName` entirely. If it were easy to optimise `Metadata::Cache::getVarsByName` instead, then that would also have been a suitable optimisation.
I guess domain knowledge and experience let them know which optimisation was more suitable here.
Re: Flame Graphs: Making the opaque obvious (2017)
#34Earlier quoted context omitted.
> I look at the reddest part of the chart, I look at the peaks Neither of these are really the places I look at when examining flame graphs. I tend to look at the bottom, and work my way up. The key thing (imo) to look for are wide pieces that are not a core part of what the code you're profiling is supposed to do. In the first example of your first link, you have a flame graph of code that seems to draw an image. Th…
> I think flame graphs, like all graphs, are more helpful when the reader has a lot of context about what's supposed to happen, or some intuition about how the chart is supposed to look. Yeah, and then your comment... just ends? So, what I get here is that, in a flame graph, the reddest part isn't the most interesting, and neither is the widest part, nor the part with the most peaks. So, what, exactly , am I looking…
> So, what, exactly, am I looking for?
This can’t be answered in general. Flamegraphs are measurements of what happened. But just like a ruler doesn’t tell you whether a given human is atypically short or tall for its species, a flamegraph can’t tell you which portion of the program takes too long a time. You need to have prior knowledge about data structures, algorithms, memory bandwidth etc in order to confront your justified expectations with the reality and be surprised with something. And it will all depend on the particular program you profile.
Re: Flame Graphs: Making the opaque obvious (2017)
#35OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
You often want to look at the widest stuff as close to the bottom as you have appropriate context for, then follow up by working upward from that to try to arrive at an explanation.
Some recent-ish examples:
- I was optimizing some code whose main tasks should have been networking and ML. There was a suspiciously wide chunk with a name indicating something about date times. An bit later we had a solid 10% improvement win.
- I had some code with strange behavior under load, undergoing some kind of a performance phase transition before the CPU (or other normal resources) were anywhere near maxed out. I grabbed a flamegraph under normal conditions and under load. The `main` loop was wider, but that's not helpful. Walking up the graph a little bit, next to some function the code was supposed to be calling there was a block named sched_yield which was huge and didn't exist in the normal trace. The root cause was just a strange (broken) concurrency mechanism in some underlying logging code, causing logging to pile up and hog all the resources past a certain request threshold.
The colors are a red herring. They exist just to make it easier to keep track of where you are in the graph, much like how ragged text is easier to read than justified.
Height is sometimes interesting. It represents a deep call stack. I find that happens most frequently in error handling code, and if something is called enough to make its way into a sampling-based flamegraph it's often worth taking a peek at for other reasons. Runtime is a function of width though, and height doesn't play a role.
Another point worth keeping in mind is that a lot of the benefit is in being able to quickly fine a plausible explanation for the performance issue. If you find that your application is spending most of its time in `read` calls, perhaps you'll want to do less of that, perhaps you'll want to submit a perf improvement to the kernel (unlikely?), but what you definitely want to do is look at the next level above the offending code, then the level above that, .... What you'll find depends on your application, but the difference between being slow because of locking and synchronization is very different from error handling or just executing the happy path a ton of times, and that insight will help govern your next actions.
Re: Flame Graphs: Making the opaque obvious (2017)
#36OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
You can use them pretty much the same as hierarchical bar charts - at least I do. The two things to look for are a) entries with a big flat surface on top - these are functions with a lot of intrinsic time (i.e. stuff that would show up as "most time in function" in a traditional profiler) b) entries that are wider than they should be - the things that you'd look into by repeatedly expanding the hierarchical bar chart, you can look into by going from the bottom of the flamegraph up.
For me when using a traditional profiler I often find myself switching back and forth between those two views (functions with most intrinstic time versus tree of function calls by time) and having to expand and contract different parts of that tree breakdown. So the advantage of a flamegraph is putting all that information on one page.
Re: Flame Graphs: Making the opaque obvious (2017)
#37How do flame graphs handle the case where most of the time is spent in some leaf function that is called from all over the program? In this case, each individual stack would not take much time but in aggregate, a lot of time is spent in the function at the top of all of the call stacks. This should not be that uncommon to have hotspots in things like copying routines, compression, encryption etc that are not associat…
They don't. The best way to visualize that, that I've seen is the DHAT tool of Valgrind, that basically builds a trie from the roots based on how much allocations happens in them (the tool measures allocations, but the visualization could just as well be used for time spent).
Re: Flame Graphs: Making the opaque obvious (2017)
#38Earlier quoted context omitted.
> I think flame graphs, like all graphs, are more helpful when the reader has a lot of context about what's supposed to happen, or some intuition about how the chart is supposed to look. Yeah, and then your comment... just ends? So, what I get here is that, in a flame graph, the reddest part isn't the most interesting, and neither is the widest part, nor the part with the most peaks. So, what, exactly , am I looking…
As I understand it, flame graphs add to normal x-axis-as-time ones by merging repeated calls to the same function, so if one function is called a lot it shows up as one wide chunk and not many small chunks spread out. So yes, height doesn't matter much, you read bottom to top to get context, and width is what you're looking for. But like any chart, it can't tell you what to optimise, or what Can be optimized. It just…
> As I understand it, flame graphs add to normal x-axis-as-time ones by merging repeated calls to the same function, so if one function is called a lot it shows up as one wide chunk and not many small chunks spread out.
Not if the graph is sorted on time axis. Not if the callstacks look different in different cases. Not if the program is recursive and thus have different depth all over the place. There is a lot of places that merging can fall over and not happen.
Re: Flame Graphs: Making the opaque obvious (2017)
#39How do flame graphs handle the case where most of the time is spent in some leaf function that is called from all over the program? In this case, each individual stack would not take much time but in aggregate, a lot of time is spent in the function at the top of all of the call stacks. This should not be that uncommon to have hotspots in things like copying routines, compression, encryption etc that are not associat…
Not sure it's a common feature, but I've seen a flame graph tool that you can pivot around some function, so that it filters to only stacks involving that function, and then goes both up and down from there.
Re: Flame Graphs: Making the opaque obvious (2017)
#40OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…
I have found the best way to learn new visualizations, is to try them out on very simple data sets that I know very well.
Write a couple of loops with threads or something and build it up from there.