Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-Sourcing Rearview: Real-Time Monitoring With Graphite

techblog.livingsocial.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Open-Sourcing Rearview: Real-Time Monitoring With Graphite

#11

Post by sakers »

dekz wrote: post #6

This looks really polished and definitely a great idea. I can see why you chose Ruby for the scripting of the monitors, being able to evaluate that code in a predefined binding can be quite powerful, especially with the aid of helpers being pre-defined as well. Why not a full ruby stack, or was the "live" scripting done after the initial inception?

We have always used Ruby for the scripting (we're predominately a Ruby shop so this was key for future adoption.) The very first mvp for this tool was individual Ruby scripts running against Graphite and being scheduled via cron. The first real backend scheduler was built in Scala, but for various reasons we've converted to Rails/Puma/Celluloid running in a VM using Jruby. The monitors themselves run in an MRI sandbox for security purposes.

Re: Open-Sourcing Rearview: Real-Time Monitoring With Graphite

#12

Post by sakers »

jwatte wrote: post #7

Server side graphs didn't work out for all our monitoring use, so we don't use graphite. You should make a version that works with istatd :-) https://github.com/imvu-open/istatd

I'll definitely check that out! I'm also thinking we may need to add support for a time-series database as Graphite does have its limitations.

Re: Open-Sourcing Rearview: Real-Time Monitoring With Graphite

#13

Post by talbright »

fit2rule wrote: post #8

I'm not sure I'm ready to abandon a custom monitoring environment consisting of a shell environment, screen, ssh certs, lugubrious quantities of /proc/, and a fair bit of gnuplot. Seems to me thats all you need? Why commit to a Ruby install for an operator console?

Actually, Rearview started out similar to that. We wanted something more accessible to our large engineering team. It has all the advantages of a web user interface coupled with the powerful capabilities you get with a scripting language.

Re: Open-Sourcing Rearview: Real-Time Monitoring With Graphite

#14

Post by talbright »

foz wrote: post #5

At my company we've been using Graphite and StatsD for nearly two years now, we rely on it heavily for tracking performance and troubleshooting issues. We rely on Icinga, Pingdom, NewRelic and other tools to alert of us of problems. Often, when things have gone really wrong (DoS, internal network issues, app errors, disk full) the affected machine(s) stop reporting to graphite (or under-report data). We get alerted b…

Rearview compliments these services, and is not intended as a replacement for them. While there is overlap, the scope is different.

Pingdom will tell you that your engine just threw a rod. Rearview will tell you your rods are knocking before that happens.

Re: Open-Sourcing Rearview: Real-Time Monitoring With Graphite

#15

Post by SEJeff »

foz wrote: post #5

At my company we've been using Graphite and StatsD for nearly two years now, we rely on it heavily for tracking performance and troubleshooting issues. We rely on Icinga, Pingdom, NewRelic and other tools to alert of us of problems. Often, when things have gone really wrong (DoS, internal network issues, app errors, disk full) the affected machine(s) stop reporting to graphite (or under-report data). We get alerted b…

In my currently non-existent freetime, I'm a Graphite co-maintainer (check github). If you have any improvements or suggestions, please feel free to send us pull requests. The current pull requests are a bit of a mess, but I blame myself and will be getting around to merging a ton of them "real soon now TM".

Re: Open-Sourcing Rearview: Real-Time Monitoring With Graphite

#16

Post by fit2rule »

sakers wrote: post #10
fit2rule wrote: post #8

I'm not sure I'm ready to abandon a custom monitoring environment consisting of a shell environment, screen, ssh certs, lugubrious quantities of /proc/, and a fair bit of gnuplot. Seems to me thats all you need? Why commit to a Ruby install for an operator console?

I'm not sure lugubrious means what you intended it to mean. :) At any rate, see my reply here https://news.ycombinator.com/item?id=6646402 for a sampling of things Rearview brings to the table. The tl;dr is that it's not a NOC tool, it's more for process monitoring whether that be application processes, engineering processes, or business processes. It also does provide a central location for anyone to see the state a…

>The tl;dr is that it's not a NOC tool, it's more for process monitoring whether that be application processes, engineering processes, or business processes. It also does provide a central location for anyone to see the state and history of an application or business unit.

Ah. I've usually just used email for that. :)

Re: Open-Sourcing Rearview: Real-Time Monitoring With Graphite

#17

Post by foz »

SEJeff wrote: post #15
foz wrote: post #5

At my company we've been using Graphite and StatsD for nearly two years now, we rely on it heavily for tracking performance and troubleshooting issues. We rely on Icinga, Pingdom, NewRelic and other tools to alert of us of problems. Often, when things have gone really wrong (DoS, internal network issues, app errors, disk full) the affected machine(s) stop reporting to graphite (or under-report data). We get alerted b…

In my currently non-existent freetime, I'm a Graphite co-maintainer (check github). If you have any improvements or suggestions, please feel free to send us pull requests. The current pull requests are a bit of a mess, but I blame myself and will be getting around to merging a ton of them "real soon now TM".

Thank you for your work on Graphite. For all it's UI strangeness and quirks, it is a great solution that a lot of people love (myself included).

I'll peek at the pull requests and see if my company might be able to contribute some help.

Post reply on HN