Saving 13M Computational Minutes per Day with Flame Graphs
techblog.netflix.com
Saving 13M Computational Minutes per Day with Flame Graphs
1–10 of 23 posts
Re: Saving 13M Computational Minutes per Day with Flame Graphs
#2I saw a youtube talk on this one ... I think it was this one: https://www.youtube.com/watch?v=O1YP8QP9gLA
Really great stuff. The spot where he gets to a pretty good description of how he uses his flame graph is roughly here: https://youtu.be/O1YP8QP9gLA?t=611
With respect to that blog-post the bit about the truncated towers is a bit of a red herring if you're 100% new to flame graphs.
The real meaty bits are the wide sections.
Re: Saving 13M Computational Minutes per Day with Flame Graphs
#3I saw a youtube talk on this one ... I think it was this one: https://www.youtube.com/watch?v=O1YP8QP9gLA Really great stuff. The spot where he gets to a pretty good description of how he uses his flame graph is roughly here: https://youtu.be/O1YP8QP9gLA?t=611 With respect to that blog-post the bit about the truncated towers is a bit of a red herring if you're 100% new to flame graphs. The real meaty bits are the wid…
Generally you're right that the wide sections are where you want to focus your attention when looking for optimizations. The point I was trying to make in the blog post is that we had to take the flame graph visualization a step further to eliminate the noise obscuring a major hot spot. The large number of broken stacks was one of the first hurdles we had to cross to improve the clarity of the visualization.
BTW, this is a different flame graph and optimization than the one discussed in the YouTube video. We use flame graphs extensively throughout Netflix.
Re: Saving 13M Computational Minutes per Day with Flame Graphs
#4"Middle-Out" approach...wonder where they got that from?
Re: Saving 13M Computational Minutes per Day with Flame Graphs
#5Wouldn't such an insanely big call stack be a performance issue in itself?
Re: Saving 13M Computational Minutes per Day with Flame Graphs
#6Wouldn't such an insanely big call stack be a performance issue in itself?
Well it depends. Stack is just a data structure. I'd say that the fact alone that you go from a deep call stack to "iterative" version where you can clearly see the stack in the code doesn't automatically make it much better.
Re: Saving 13M Computational Minutes per Day with Flame Graphs
#7"Middle-Out" approach...wonder where they got that from?
That's when I checked to make sure it wasn't an April Fools joke.
Re: Saving 13M Computational Minutes per Day with Flame Graphs
#8I saw a youtube talk on this one ... I think it was this one: https://www.youtube.com/watch?v=O1YP8QP9gLA Really great stuff. The spot where he gets to a pretty good description of how he uses his flame graph is roughly here: https://youtu.be/O1YP8QP9gLA?t=611 With respect to that blog-post the bit about the truncated towers is a bit of a red herring if you're 100% new to flame graphs. The real meaty bits are the wid…
Generally you're right that the wide sections are where you want to focus your attention when looking for optimizations. The point I was trying to make in the blog post is that we had to take the flame graph visualization a step further to eliminate the noise obscuring a major hot spot. The large number of broken stacks was one of the first hurdles we had to cross to improve the clarity of the visualization. BTW, thi…
Can you write up how you fixed the broken call stacks? I've used Brendan's tools (with java-perf-map, also an awesome tool) to generate flame graphs for Scala code and had no idea I could only see 127 frames.
Re: Saving 13M Computational Minutes per Day with Flame Graphs
#9Wouldn't such an insanely big call stack be a performance issue in itself?
Recursive call would have huge call stack.
Re: Saving 13M Computational Minutes per Day with Flame Graphs
#10Earlier quoted context omitted.
Generally you're right that the wide sections are where you want to focus your attention when looking for optimizations. The point I was trying to make in the blog post is that we had to take the flame graph visualization a step further to eliminate the noise obscuring a major hot spot. The large number of broken stacks was one of the first hurdles we had to cross to improve the clarity of the visualization. BTW, thi…
Can you write up how you fixed the broken call stacks? I've used Brendan's tools (with java-perf-map, also an awesome tool) to generate flame graphs for Scala code and had no idea I could only see 127 frames.
We ultimately should be fixing this with BPF, which we'll certainly post instructions for.