Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Loupe: Etsy's New Monitoring Stack

codeascraft.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Loupe: Etsy's New Monitoring Stack

#21

Post by laichzeit0 »

I'm not really convinced that this is very useful. I've been in the application monitoring space for a few years now and I'm not sure that watching graphs is something Ops people should be doing.

There should be rules which notify them if something is anomalous, by email, SMS, or logging a problem on an incident management tool. e.g. "Java request foo.bar() on Managed Server 1 is throwing exceptions for 50% of invocations (20 requests, 10 exceptions) in the last 10 minutes. This affects the following services: Customer Login page on foo.bar." possibly even attaching some of the exception messages to the email, if sampled through instrumentation or correlating it back to the log files, automatically.

This type of monitoring is actually useful because Ops understand what is broken, what it effects and gives them enough detail to either fix it or pass the problem to someone else; and they're not wasting their time looking at graphs waiting for a problem to appear.

Re: Loupe: Etsy's New Monitoring Stack

#22

Post by kawsper »

russgray wrote: post #20
plasma wrote: post #13

Anyone recommend an easy way to get started with StatsD? I've tried to configure/install/setup StatsD etc in the past but hit so many problems with dependencies, undocumented software needing to be installed, etc. Any tutorial or something to get stats being tracked and graphed beautifully would be awesome.

This blog post helped me when I took a look last year: http://geek.michaelgrace.org/2011/09/installing-statsd-on-ub...

It just too bad there is a lot of moving parts in this setup.

Graphite consists of these three parts:

carbon - a daemon that listens for time-series data. whisper - a simple database library for storing time-series data. webapp - a (Django) webapp that renders graphs on demand.

And statsd is its own daemon.

That means 3 daemons needs to run to make stat aggregating.

Re: Loupe: Etsy's New Monitoring Stack

#23

Post by jqueryin »

I was very intrigued until I poked around the github repos and noticed the server specs Etsy was using to analyze the 250k metrics.

Oculus recommended setup (found at https://github.com/etsy/oculus):

  * ElasticSearch
    * At least 8GB RAM
    * Quad Core  Xeon 5620 CPU or comparable
    * 1GB disk space
    * two ElasticSearch servers in separate clusters
  * a cluster of Worker boxes running Resque 
    * worker master runs redis
    * additional resque worker boxes (and potentially slaves)
    * At least 12GB RAM
    * Quad Core Xeon 5620 CPU or comparable
    * 1GB disk space
It'd be nice if there was a more established baseline set of server specs to get up and running. While many of us aspire to be at Etsy level monitoring, we're just not there.

Re: Loupe: Etsy's New Monitoring Stack

#24

Post by jonlives »

jqueryin wrote: post #23

I was very intrigued until I poked around the github repos and noticed the server specs Etsy was using to analyze the 250k metrics. Oculus recommended setup (found at https://github.com/etsy/oculus ): * ElasticSearch * At least 8GB RAM * Quad Core Xeon 5620 CPU or comparable * 1GB disk space * two ElasticSearch servers in separate clusters * a cluster of Worker boxes running Resque * worker master runs redis * additi…

I'll definitely have a look at doing that - the initial specs were designed around the metric volumes we use the tools for, but I realise that might not be practical for smaller workloads :)

Re: Loupe: Etsy's New Monitoring Stack

#25

Post by thibaut_barrere »

laichzeit0 wrote: post #21

I'm not really convinced that this is very useful. I've been in the application monitoring space for a few years now and I'm not sure that watching graphs is something Ops people should be doing. There should be rules which notify them if something is anomalous, by email, SMS, or logging a problem on an incident management tool. e.g. "Java request foo.bar() on Managed Server 1 is throwing exceptions for 50% of invoca…

The thing is that at a given scale (and it comes early, actually), pushes do not scale.

I still use pushes for clear-cut things that require paging, but having graphes of a lot of things and just noticing changes or anomaly on the overall patterns will help spot a lot of issues, including things you haven't yet planned paging for :-)

Re: Loupe: Etsy's New Monitoring Stack

#26

Post by thibaut_barrere »

plasma wrote: post #13

Anyone recommend an easy way to get started with StatsD? I've tried to configure/install/setup StatsD etc in the past but hit so many problems with dependencies, undocumented software needing to be installed, etc. Any tutorial or something to get stats being tracked and graphed beautifully would be awesome.

Batsd [1] is a stripped down version of (StatsD + Graphite) that works well in my opinion. You won't have the full graphite functions etc, but it's easier to get started.

[1] https://github.com/noahhl/batsd

Re: Loupe: Etsy's New Monitoring Stack

#27

Post by ghotli »

kawsper wrote: post #22
russgray wrote: post #20

Earlier quoted context omitted.

This blog post helped me when I took a look last year: http://geek.michaelgrace.org/2011/09/installing-statsd-on-ub...

It just too bad there is a lot of moving parts in this setup. Graphite consists of these three parts: carbon - a daemon that listens for time-series data. whisper - a simple database library for storing time-series data. webapp - a (Django) webapp that renders graphs on demand. And statsd is its own daemon. That means 3 daemons needs to run to make stat aggregating.

Just use the existing chef recipies out there for setting it up. Why would you try to get all the moving parts integrated properly when the work has already been done? Stand on the shoulders of giants.

Re: Loupe: Etsy's New Monitoring Stack

#28

Post by methehack »

methehack wrote: post #15

> That’s far too many graphs for a team of 150 engineers to watch all day long! Does etsy have 150 engineers? Is that even possible?

I'll take the downvote as a "yes" :)

It's true the "Is that even possible?" was out of line and I should have tempered it -- but, I am truly surprised.

Re: Loupe: Etsy's New Monitoring Stack

#29

Post by kawsper »

ghotli wrote: post #27
kawsper wrote: post #22

Earlier quoted context omitted.

It just too bad there is a lot of moving parts in this setup. Graphite consists of these three parts: carbon - a daemon that listens for time-series data. whisper - a simple database library for storing time-series data. webapp - a (Django) webapp that renders graphs on demand. And statsd is its own daemon. That means 3 daemons needs to run to make stat aggregating.

Just use the existing chef recipies out there for setting it up. Why would you try to get all the moving parts integrated properly when the work has already been done? Stand on the shoulders of giants.

> when the work has already been done?

I am very wary about introducing new software into our stack, if I doesn't understand it. Bad configured software could cause problems down the road.

Last time I tried out a recipe that installed Redis, it didn't version lock the Redis-server which meant that the daemon couldn't start because they had deprecated some configs.

The recent DDOS DNS attacks was possible because people have setup wrongly configured DNS servers.

Re: Loupe: Etsy's New Monitoring Stack

#30

Post by sokoloff »

thibaut_barrere wrote: post #25
laichzeit0 wrote: post #21

I'm not really convinced that this is very useful. I've been in the application monitoring space for a few years now and I'm not sure that watching graphs is something Ops people should be doing. There should be rules which notify them if something is anomalous, by email, SMS, or logging a problem on an incident management tool. e.g. "Java request foo.bar() on Managed Server 1 is throwing exceptions for 50% of invoca…

The thing is that at a given scale (and it comes early, actually), pushes do not scale. I still use pushes for clear-cut things that require paging, but having graphes of a lot of things and just noticing changes or anomaly on the overall patterns will help spot a lot of issues, including things you haven't yet planned paging for :-)

Totally agree. We have a mix of algorithmic/automated monitor and visual monitors. I can tell you a lot more about how healthy our site is from looking at two screens in our NOC than I can from all the pages sent over the last .

Computers are great at executing repetitive, specified tasks. Use them for that.

Humans are great at pattern recognition and flexibly adapting. Use them for that, IMO.

Post reply on HN