Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Flame Graphs: Making the opaque obvious (2017)

tech.popdata.org

Post reply on HN
Print view View original

41–50 of 58 posts

Re: Flame Graphs: Making the opaque obvious (2017)

#41

Post by sebastianmestre »

PreInternet01 wrote: post #10
rkallos wrote: post #7

Earlier quoted context omitted.

> I look at the reddest part of the chart, I look at the peaks Neither of these are really the places I look at when examining flame graphs. I tend to look at the bottom, and work my way up. The key thing (imo) to look for are wide pieces that are not a core part of what the code you're profiling is supposed to do. In the first example of your first link, you have a flame graph of code that seems to draw an image. Th…

> I think flame graphs, like all graphs, are more helpful when the reader has a lot of context about what's supposed to happen, or some intuition about how the chart is supposed to look. Yeah, and then your comment... just ends? So, what I get here is that, in a flame graph, the reddest part isn't the most interesting, and neither is the widest part, nor the part with the most peaks. So, what, exactly , am I looking…

Width in a flame graph is directly proportional to runtime. Optimizing a block that covers x% of the graph will only speed up the program by x% or less, so probably dont bother with blocks less than 0.5% wide.

This by itself should already tell you what NOT to optimize.

But really, you should be looking for operations that take a long time but shouldn't (wide blocks that should be thin). To find it you need to have an intuitive idea of how fast things should be beforehand.

If you have no idea how fast things should be, no amount of graphs will help you with this.

Re: Flame Graphs: Making the opaque obvious (2017)

#42

Post by sgerenser »

yxhuvud wrote: post #38
spencerflem wrote: post #11

Earlier quoted context omitted.

As I understand it, flame graphs add to normal x-axis-as-time ones by merging repeated calls to the same function, so if one function is called a lot it shows up as one wide chunk and not many small chunks spread out. So yes, height doesn't matter much, you read bottom to top to get context, and width is what you're looking for. But like any chart, it can't tell you what to optimise, or what Can be optimized. It just…

> As I understand it, flame graphs add to normal x-axis-as-time ones by merging repeated calls to the same function, so if one function is called a lot it shows up as one wide chunk and not many small chunks spread out. Not if the graph is sorted on time axis. Not if the callstacks look different in different cases. Not if the program is recursive and thus have different depth all over the place. There is a lot of pl…

If you sort it on a time axis, its not really a flame graph any more. The whole point of a flame graph is to group same stacks together, even if they happen at multiple disparate times.

Re: Flame Graphs: Making the opaque obvious (2017)

#43

Post by sgerenser »

PreInternet01 wrote: post #24
simscitizen wrote: post #22

Earlier quoted context omitted.

Peaks don't matter, they just correspond to the depth of the call stack. Probably the simplest way to use the flame graph is work from the bottom of the flamegraph and walk upwards until you find something interesting you optimize. Ideally you find something wide to optimize that makes sense. (The widest thing here is "main" which is obviously probably not the interesting thing to optimize, so you would work upwards…

> Probably the simplest way to use the flame graph is work from the bottom of the flamegraph and walk upwards until you find something interesting you optimize OK, so going by what is apparently the 'simple example' in the linked article: https://tech.popdata.org/images/cps1970_before_fix_dwarf_gcc... I work my way up. First thing that is really red is Conversion::Process::Run, but that probably wraps a lot of things…

The colors are completely arbitrary! They’re just used to make it easier to see the difference between one stack and the next. They could just as easily be all the same color, it would just be harder to see the edges.

Re: Flame Graphs: Making the opaque obvious (2017)

#45

Post by Aponax »

zubspace wrote: post #44

Do colors have any significance in those flame graphs? It's unfortunate that a post about them does not mention anything about colors. If you look at at the examples, there are bars, which have the same length, but the colors look random to me.

Yes they are random. See the blog about the inventor of the flame graph, how we know them today

> Neelakanth and Roch's visualizations used completely random colors to differentiate frames. I thought it looked nicer to narrow the color palette, and picked just warm colors initially as it explained why the CPUs were "hot" (busy). Since it resembled flames, it quickly became known as flame graphs.

https://www.brendangregg.com/flamegraphs.html

Re: Flame Graphs: Making the opaque obvious (2017)

#46

Post by vient »

zlurkerz wrote: post #14

How do flame graphs handle the case where most of the time is spent in some leaf function that is called from all over the program? In this case, each individual stack would not take much time but in aggregate, a lot of time is spent in the function at the top of all of the call stacks. This should not be that uncommon to have hotspots in things like copying routines, compression, encryption etc that are not associat…

I like speedscope.app for viewing flamegraphs. It is more interactive than traditional SVG flamegraphs, and what is relevant here is a "sandwich" view - basically a sorted list of all functions, you see what function was spent the most time in, click on it and see all calling stack traces like a mini flamegraph, filtered and centered on this function. Speedscope supports several popular trace formats, really useful.

Re: Flame Graphs: Making the opaque obvious (2017)

#47

Post by yosefk »

nemetroid wrote: post #20
zlurkerz wrote: post #14

How do flame graphs handle the case where most of the time is spent in some leaf function that is called from all over the program? In this case, each individual stack would not take much time but in aggregate, a lot of time is spent in the function at the top of all of the call stacks. This should not be that uncommon to have hotspots in things like copying routines, compression, encryption etc that are not associat…

> How do flame graphs handle the case where most of the time is spent in some leaf function that is called from all over the program? In my experience, they don't really. They're very good for finding easy wins where a high/medium-level operation takes much longer than you'd expect it to, not so much for finding low-level hotspots in the code. > such a view would [...] subsume the usual use cases for a flame graph, w…

OTOH, hitting Ctrl-C in gdb and looking at a backtrace a few times would show you the leaf function

Re: Flame Graphs: Making the opaque obvious (2017)

#48

Post by dieortin »

PreInternet01 wrote: post #17
simscitizen wrote: post #13

Earlier quoted context omitted.

As an example, imagine you sampled a program and got 5 CPU call stack samples. c c b b d a a a a a main main main main main In a flamegraph, you would see: [c ] [b ][d ] [a ] [main ]

Yeah, I imagine it, and still don't see how the flame graph would help? Shown as a hierarchical bar chart, this would suggest 'b' is problematic. Where, color-wise (because peak-wise, 'c' would be the culprit here) do I see this issue in a flame graph? Because I fear that either 'main' or 'a' would have the most dominant shade of red here?

If you only had to look at a specific position or color in the graph, then a graph wouldn’t be needed at all. The offending function would just be printed.

But the flamegraph can’t color the offending function, or show it in a specific position, because what the offending function is depends on the context of what the program is trying to do and what the expected CPU usage for each function is. That’s information only you have.

So you only need to look at wide chunks and see if it’s expected for them to take so much share of the execution time or not, and if they can be optimized.

Re: Flame Graphs: Making the opaque obvious (2017)

#49

Post by dangets »

zlurkerz wrote: post #14

How do flame graphs handle the case where most of the time is spent in some leaf function that is called from all over the program? In this case, each individual stack would not take much time but in aggregate, a lot of time is spent in the function at the top of all of the call stacks. This should not be that uncommon to have hotspots in things like copying routines, compression, encryption etc that are not associat…

Java Mission Control [0] has a button to toggle for displaying the profile as thread roots or method roots for this purpose. I am not imaginative enough to come up with a visualization that shows both (maybe utilize background color or another indicator to show the leaf function's relative frequency in the other direction?). Either way, both directions have their use case when investigating.

[0] https://adoptium.net/jmc/

Re: Flame Graphs: Making the opaque obvious (2017)

#50

Post by twodave »

PreInternet01 wrote: post #2

OK, shameful confession time here: I just cannot grasp flame charts, no matter how hard I try. And yes: that's just me, I'm dumb, etc. etc. (and I freely admit all of that, including the et-ceteras!) I tried to follow along with things that are relevant to my job, like https://randomascii.wordpress.com/2016/09/05/etw-flame-graph... ...And totally failed? I look at the reddest part of the chart, I look at the peaks, a…

Alright, so first of all, the colors don't matter, they're just for contrast/legend purposes. Flame graphs come in all sorts of color schemes, not even necessarily yellow/orange/red.

I tend to look at flame graphs in terms of % of the overall process. They're good for finding that one part of a routine that is taking up a decent % of processing, and if that part of the routine is being hung up on some mundane task.

For example, if I see 3/4 stacks directly on top of each other, then I know I've got a call stack a few levels deep that is overwhelmingly waiting on some low level thing to finish. If it's something that should be really fast (like a cache lookup), then I know something really stupid is happening.

Some flame graphs will tie in e.g. network requests and DB queries as their own traces, which will also give you a clue sometimes. Like, oh, this function is waiting 10s for a query to complete? Let's see what that is actually doing, maybe we can speed it up.

I used flame graphs (among other things) this year to take a 30 minute long payroll process down to about 3 minutes. Much of this was just scanning the flame graph for things high in the call stack that were taking up noticeable % of the processing time. This is easier if you know the codebase, but for example, I could see within the "load a bunch of data" phase of our processing that there were a few tax-related things taking up most of the overall time. We managed to trace those to a few calls to a third party library that we couldn't make any faster, but we could cache the results to mitigate the issue.

Another place we found expensive queries being repeated in different functions, which was obvious because we had 2 calls to the same function from both places. We ended up just raising those shared calls up a level and patching the data directly into the two functions that needed it.

Other places were less obvious. We could see a lot of time being spent, but we couldn't tell from the flame graph what was happening. We'd go look at some code and find some n^2 aggregation function that we'd need to simplify.

Overall, flame graphs are just one tool. They might not even be the best tool. In our case (heavy data driven web application) I would place DB observability at least as high in importance as good tracing, and flame graphs are just one way of visualizing traces.

Post reply on HN