Loupe: Etsy's New Monitoring Stack
codeascraft.com
Loupe: Etsy's New Monitoring Stack
1–10 of 30 posts
Re: Loupe: Etsy's New Monitoring Stack
#2This is really interesting. I can't even imagine measuring 250k different metrics without a tool like this. It's just so much data to assess.
Granted it would be extremely useful for post-mortems, but looking at it real time is a bit like the library of Babel [1].
Re: Loupe: Etsy's New Monitoring Stack
#3Very nifty. The automatic selection is a great innovation.
I built a somewhat similar system a while ago on-top of statsd/graphite. Mine was not designed for production deployment though, just as a test platform (I was basically using graphite to store and query metric data. Not optimal, but that problem was out of scope and it was easy to abuse like that.) This tool allowed a user to manually select a set of metrics and create a fault classifiers with those metrics.
These classifiers were able to detect not only the presence of faults but also classify what type of faults they were (provided sufficient training data. Of course you could train new classifiers with data you collected in production so training new classifiers becomes an ongoing activity.). We were only testing geometric classification, but using any sort of classifier to identify complex fault types seems to be an idea with promise.
Re: Loupe: Etsy's New Monitoring Stack
#4Always fun to read these Etsy ops posts. I'm very curious to know what their practical architecture looks like that allows them to capture 250k unique metrics and also run skyline against them all. It seems like each new algorithm would add a ton of processing requirements when you're at that scale.
Also, it seems like this would be really useful with the addition of metrics grouping and group specific algorithms as right now it looks like their 250k metrics all pop up in the same anomalous bucket with all metrics getting the same algorithms applied to them.
Re: Loupe: Etsy's New Monitoring Stack
#5Abe gave a talk at OpenVisConf about Skyline and how to make use of 250k metrics http://www.youtube.com/watch?v=Rij604NBXqk
Re: Loupe: Etsy's New Monitoring Stack
#6Interesting stuff! I've actually been working on the same idea recently, starting with reading about anomaly detection. In particular, this survey: http://www-users.cs.umn.edu/~kumar/papers/anomaly-survey.php
I would like to know more about the performance of Skyline in practice:
- what are the accuracy and recall like?
- what is CPU consumption like?
Regarding the latter, I had a quick look at the implemented algorithms and them seemed very inefficient. Basically recomputing over the entire series at every change. I think with a bit of work most of the algorithms could be reimplemented in an incremental way. I also wouldn't use Python for something that is going to be CPU bound. (I await the "We rewrote in Go and it's 10x faster!" blog post ;-)
Re: Loupe: Etsy's New Monitoring Stack
#7They might want to reconsider the name:
http://www.gibraltarsoftware.com/
Their monitoring solution is also called Loupe.
Re: Loupe: Etsy's New Monitoring Stack
#8Interesting stuff! I've actually been working on the same idea recently, starting with reading about anomaly detection. In particular, this survey: http://www-users.cs.umn.edu/~kumar/papers/anomaly-survey.php I would like to know more about the performance of Skyline in practice: - what are the accuracy and recall like? - what is CPU consumption like? Regarding the latter, I had a quick look at the implemented algori…
Author here. Accuracy is okay - we err on the side of noise, but it does routinely pick up anomalies. It doesn't currently account for seasonal trends, though.
We aim for 100% CPU consumption. Analyzing is very CPU intensive process, and there are two parts in particular that are expensive: decoding the Redis string from MessagePack to Python, and running the algorithms.
As for the algorithm inefficiencies, pull requests encouraged :)
Rewriting it in Go is a plan for a rainy weekend :) The problem with Go is that it doesn't have as great statistics support as Python does.
Re: Loupe: Etsy's New Monitoring Stack
#9Interesting stuff! I've actually been working on the same idea recently, starting with reading about anomaly detection. In particular, this survey: http://www-users.cs.umn.edu/~kumar/papers/anomaly-survey.php I would like to know more about the performance of Skyline in practice: - what are the accuracy and recall like? - what is CPU consumption like? Regarding the latter, I had a quick look at the implemented algori…
Author here. Accuracy is okay - we err on the side of noise, but it does routinely pick up anomalies. It doesn't currently account for seasonal trends, though. We aim for 100% CPU consumption. Analyzing is very CPU intensive process, and there are two parts in particular that are expensive: decoding the Redis string from MessagePack to Python, and running the algorithms. As for the algorithm inefficiencies, pull requ…
Thanks for replying! A little while ago I watched part of your Bacon Conf talk (http://devslovebacon.com/conferences/bacon-2013/talks/bring-...) and read the slides.
There were a few things I thought were a bit odd about the architecture, as I recall it.
IIRC you poll Graphite for metrics. Why not push them from StatsD directly into Skyline? This would probably be more efficient. If you used incremental / online / streaming algorithms you'll have a compact summary at each time step, so you can throw away the raw data. 250K metrics would fit in memory quite easily (we're just talking approximately a number and a string each, right?) and you have 4000+ cycles per second to process them, which should be sufficient.
Python lack of good threading would possibly be a problem. I would use the JVM (Scala in my case). Apache Commons Math is pretty good (http://commons.apache.org/proper/commons-math/). Java's verbose interfaces are a bit annoying, but the JVM is damn efficient, and you can wrap the crap is something more aesthetic. It's a solid choice no matter what the hipsters say. ;-)
Re: Loupe: Etsy's New Monitoring Stack
#10Earlier quoted context omitted.
Author here. Accuracy is okay - we err on the side of noise, but it does routinely pick up anomalies. It doesn't currently account for seasonal trends, though. We aim for 100% CPU consumption. Analyzing is very CPU intensive process, and there are two parts in particular that are expensive: decoding the Redis string from MessagePack to Python, and running the algorithms. As for the algorithm inefficiencies, pull requ…
Thanks for replying! A little while ago I watched part of your Bacon Conf talk ( http://devslovebacon.com/conferences/bacon-2013/talks/bring-... ) and read the slides. There were a few things I thought were a bit odd about the architecture, as I recall it. IIRC you poll Graphite for metrics. Why not push them from StatsD directly into Skyline? This would probably be more efficient. If you used incremental / online /…
Ah! We could, but StatsD provides support for more complex metrics like aggregated sums over time. Not something that lends itself easily to a discrete datapoint.
It is. But we use multiprocessing, which is basically the same API. Still, you can't beat the awesome Python stats libraries: Numpy, SciPy, Statsmodels, Pandas..