Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving 13M Computational Minutes per Day with Flame Graphs

techblog.netflix.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Saving 13M Computational Minutes per Day with Flame Graphs

#11

Post by azinman2 »

What I think is interesting about this is that they weren't able to easily measure or find using existing tools these hotspots -- they needed a combination of visualization and data munging to do so.

Visualization is an often overlooked tool in CS -- for example IDEs do little to zero visualization... only LightTable is starting to break out of the traditional text document. It also shows that depending on the problem visualization & data can be morphed and stretched to provide new insights when others might have walked away.

So why isn't this something that's a part of job interviewing or a bigger part of our normal toolbox as engineers?

Re: Saving 13M Computational Minutes per Day with Flame Graphs

#12

Post by f_ »

Very interesting indeed; but somehow I was even more baffled at the package names they seem to be using:

  com.netflix.vulturemonkey.cow.iguana.MacawSquirrel

  com.netflix.ape.serpent.vulture.ApeVultureMantis

  com.netflix.iguanas.monkey.insect.IguanaRabbit
Any idea what's up with that?

Re: Saving 13M Computational Minutes per Day with Flame Graphs

#13

Post by mspier »

f_ wrote: post #12

Very interesting indeed; but somehow I was even more baffled at the package names they seem to be using: com.netflix.vulturemonkey.cow.iguana.MacawSquirrel com.netflix.ape.serpent.vulture.ApeVultureMantis com.netflix.iguanas.monkey.insect.IguanaRabbit Any idea what's up with that?

We really love animals! :-) JK. Just obfuscating class names with animal names before publishing the blog post.

Re: Saving 13M Computational Minutes per Day with Flame Graphs

#15

Post by f_ »

mspier wrote: post #13
f_ wrote: post #12

Very interesting indeed; but somehow I was even more baffled at the package names they seem to be using: com.netflix.vulturemonkey.cow.iguana.MacawSquirrel com.netflix.ape.serpent.vulture.ApeVultureMantis com.netflix.iguanas.monkey.insect.IguanaRabbit Any idea what's up with that?

We really love animals! :-) JK. Just obfuscating class names with animal names before publishing the blog post.

Hehe! Thanks for the clarification -- I wondered if everyone was going bananas over at your company! This explains it (:

Re: Saving 13M Computational Minutes per Day with Flame Graphs

#16

Post by [deleted] »

mspier wrote: post #13
f_ wrote: post #12

Very interesting indeed; but somehow I was even more baffled at the package names they seem to be using: com.netflix.vulturemonkey.cow.iguana.MacawSquirrel com.netflix.ape.serpent.vulture.ApeVultureMantis com.netflix.iguanas.monkey.insect.IguanaRabbit Any idea what's up with that?

We really love animals! :-) JK. Just obfuscating class names with animal names before publishing the blog post.

[deleted]

Re: Saving 13M Computational Minutes per Day with Flame Graphs

#17

Post by mspier »

f_ wrote: post #15
mspier wrote: post #13

Earlier quoted context omitted.

We really love animals! :-) JK. Just obfuscating class names with animal names before publishing the blog post.

Hehe! Thanks for the clarification -- I wondered if everyone was going bananas over at your company! This explains it (:

Still better than some unpronounceable old-norse names we had on a few projects. :-)

Re: Saving 13M Computational Minutes per Day with Flame Graphs

#18

Post by topspin »

geodel wrote: post #14
Illniyar wrote: post #5

Wouldn't such an insanely big call stack be a performance issue in itself?

These call stacks are normal for typical Java enterprise application.

Indeed this is normal. Apache Camel produced such huge stack traces they refactored the routing system specifically to reduce AsyncCallback usage and shorten stack traces; at one time Camel would dump traces thousands of lines long. However, pointing this out doesn't actually address the question; is there a performance issue indicated by these huge call stacks?

I've wondered about the question myself when encountering incredibly long stack traces while troubleshooting Java systems. I've also wondered if there is some more general dysfunction indicated. I've see impressive stack traces in C and C++, but nothing quite like what I've found in Java. What is the experience of C# programmers?

Re: Saving 13M Computational Minutes per Day with Flame Graphs

#19

Post by sdesol »

azinman2 wrote: post #11

What I think is interesting about this is that they weren't able to easily measure or find using existing tools these hotspots -- they needed a combination of visualization and data munging to do so. Visualization is an often overlooked tool in CS -- for example IDEs do little to zero visualization... only LightTable is starting to break out of the traditional text document. It also shows that depending on the proble…

> Visualization is an often overlooked tool in CS

It's often overlooked, because generating meaningful data, that can provide visual insight, is usually very difficult. Right now I'm working on a blog post that goes over how you can use motion bubble charts to track code changes and I use GitLab as an example. You can find a draft of the blog at:

http://gitsense.github.io/blog/motion-bubble-charts.html

Note the blog post is still in DRAFT state, so there are broken links and grammatical errors and what not.

Capturing meaningful data at the Enterprise scale, requires a lot of effort. There is a reason why I ended up creating my own real-time process monitoring system:

http://gitsense.github.io/blog/realtime-process-monitoring.h...

What I'm ultimately hoping to do with the metrics, is create a new way to visual Git logs and improve how we approach complex code reviews and diffs.

Re: Saving 13M Computational Minutes per Day with Flame Graphs

#20

Post by surrealvortex »

I'm currently using flame graphs at work. If your application hasn't been profiled recently, you'll usually get lots of improvement for very little effort.

Some 15 minutes of work improved CPU usage of my team's biggest fleet by ~40%. Considering we scaled up to 1500 c3.4xlarge hosts at peak in NA alone on that fleet, those 15 minutes kinda made my month :)

One thing to note once you eliminate the easy pickings is that as you go higher up the call graph, the profiler visualization is often misleading. There may be sections of code without safe-points, and stuff that appears wide on the flame graph may just be getting blamed for adjacent code that doesn't have safe points.

Post reply on HN