My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

Post by thelamest » Wed, Oct 15, 2014, 2:12 PM UTC

sleepydog wrote: post #91

jldugger wrote: post #55

Earlier quoted context omitted.

Logging is super fucking noisy and generally not structured for operations support. The state of the art at my not-google employers is basically Nagios scripts. Everyone has these scripts that check various components: - is the webserver running - is it responding on port 443 - does it return HTML' and maybe - 'If I submit a search, do I get a result back?' Nagios scripts are responsible for everything: opening netwo…

> As an example, imagine you're monitoring /. Nagios runs NRPE, NRPE looks at df /, and if it's 85 percent full (by default), sends a warning page. At 90 percent it sends a critical page. A smarter system collects the df / results, delivers it to a central timeseries database. The new data point is used to create a new projection, and the new projection is used to determine the time to an actual failure. The system a…

I too would be very interested to see your scripts!

Post by nucleardog » Wed, Oct 15, 2014, 8:02 PM UTC

rhizome wrote: post #94

beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

Another thing about car alarms: how many people even know what their own car alarm sounds like?

Unless you have some fancy/aftermarket thing, it sounds just like your horn which you probably hear every time you use the key fob to lock it.

Whether you have the ability to differentiate your particular tone of horn from all the others is a whole different story.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google