Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Loupe: Etsy's New Monitoring Stack

codeascraft.com

Post reply on HN
Print view View original

11–20 of 30 posts

Re: Loupe: Etsy's New Monitoring Stack

#11

Post by philsnow »

I'm confused on one point: does the anomaly correlation find other metrics that look "similar" or other metrics that also have anomalies in the same time span ? The latter seems like it would be very useful.

You mention elsewhere that statsd lets you do complicated aggregations over time. If you have a moving average of errors over 10 minutes or something, that's potentially not going to show up when you do anomaly correlation, since a spike is smeared across 20 minutes. Do you account for that? It would require knowing which metrics are aggregated across time and by how much, etc, I guess.

Re: Loupe: Etsy's New Monitoring Stack

#12

Post by jonlives »

philsnow wrote: post #11

I'm confused on one point: does the anomaly correlation find other metrics that look "similar" or other metrics that also have anomalies in the same time span ? The latter seems like it would be very useful. You mention elsewhere that statsd lets you do complicated aggregations over time. If you have a moving average of errors over 10 minutes or something, that's potentially not going to show up when you do anomaly c…

Oculus author here - Oculus detects other metrics that look similar, ie that have a simlar anomaly or shape in the same time span. It doesn't pick up metrics that have other, "dissimilar" anomalies - that part is left to Skyline

Oculus treats all metrics that it gets from Skyline equally at the moment, ie it doesn't know if what it's looking at is an aggregation, or a single set of data points. It just takes the data as it's presented. It would be totally possible, however, to add 10 and 20 minute averages (for example) for the same metric into Skyline so that Oculus would treat them separately.

Re: Loupe: Etsy's New Monitoring Stack

#13

Post by plasma »

Anyone recommend an easy way to get started with StatsD?

I've tried to configure/install/setup StatsD etc in the past but hit so many problems with dependencies, undocumented software needing to be installed, etc.

Any tutorial or something to get stats being tracked and graphed beautifully would be awesome.

Re: Loupe: Etsy's New Monitoring Stack

#14

Post by tel »

DTW is quadratic. While in grad school I worked for a bit with a team interested in doing massive speech recognition using DTW so they did some work speeding up the algorithm using a technical called Locality Sensitive Hashing [1], [2], [3]. It might be worth a look in order to speed your algorithms.

[1] http://www.academia.edu/2600658/Indexing_Raw_Acoustic_Featur... [2] http://old-site.clsp.jhu.edu/~ajansen/papers/IS2012a.pdf [3] http://www.cs.jhu.edu/~vandurme/papers/JansenVanDurmeASRU11....

Re: Loupe: Etsy's New Monitoring Stack

#18

Post by boothead »

aba_sababa wrote: post #8
noelwelsh wrote: post #6

Interesting stuff! I've actually been working on the same idea recently, starting with reading about anomaly detection. In particular, this survey: http://www-users.cs.umn.edu/~kumar/papers/anomaly-survey.php I would like to know more about the performance of Skyline in practice: - what are the accuracy and recall like? - what is CPU consumption like? Regarding the latter, I had a quick look at the implemented algori…

Author here. Accuracy is okay - we err on the side of noise, but it does routinely pick up anomalies. It doesn't currently account for seasonal trends, though. We aim for 100% CPU consumption. Analyzing is very CPU intensive process, and there are two parts in particular that are expensive: decoding the Redis string from MessagePack to Python, and running the algorithms. As for the algorithm inefficiencies, pull requ…

If you're analyzing streams of data Haskell would be a good choice.*

*assumes that you've already put in the requisite couple of years in monastic contemplation to learn Haskell :-)

Re: Loupe: Etsy's New Monitoring Stack

#19

Post by contingencies »

Skyline and oculus both look interesting and this is definitely a solid direction to be heading in.

However, I wonder if some form of topology knowledge, operations dependency tree or similar could further inform this type of root cause analytics.

Without a declarative style "here is how thing should be" model of adequate accuracy, it seems like the analytics will be stuck at the "these things are strange and happened at once, what does human think?" level of sophistication.

Re: Loupe: Etsy's New Monitoring Stack

#20

Post by russgray »

plasma wrote: post #13

Anyone recommend an easy way to get started with StatsD? I've tried to configure/install/setup StatsD etc in the past but hit so many problems with dependencies, undocumented software needing to be installed, etc. Any tutorial or something to get stats being tracked and graphed beautifully would be awesome.

This blog post helped me when I took a look last year: http://geek.michaelgrace.org/2011/09/installing-statsd-on-ub...
Post reply on HN