Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

docs.google.com

Post reply on HN
Print view View original

31–40 of 122 posts

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#31

Post by robewaschuk »

peterwwillis wrote: post #18
leef wrote: post #8

"If you want a quiet oncall rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical." This is an excellent point that is missed in most monitoring setups I've seen. A classic example is some request that kills your service process. You get paged for that so you wrap the service in a supervisor like daemon. The immediate issue is fixed and, typicall…

Re: this will be a problem soon? Metrics trending. Look for changes in your metrics to spot potential problems and plan for the future. This is done quite often for example in QA to look for issues between releases, and can be done both macro and micro in terms of continuous delivery services' metrics.

I think anything that requires "spotting potential problems" is only a partial solution. I've never seen a compelling system that can look at all the metrics and (with reasonable precision and recall) spot and summarize changes that are actually problematic and surprising to humans. It's definitely a necessary part of observing what's going on (and quickly eliminating hypotheses like "maybe we're out of CPU!"), for sure.

The subcritical alerts I think of are more things like "Well, the database is _getting_ full, but it's not full yet." Or to borrow someone else's example, "we put in this daemon restarter when it was dying once a week; now it's dying every few minutes and we're only surviving because our proxy is masking the problem but soon it's going to take the whole site down."

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#32

Post by tonfa »

Someone1234 wrote: post #6

Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…

What makes you think this isn't the case? I don't see anything to the contrary in the document.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#33

Post by robewaschuk »

0xbadcafebee wrote: post #22

Most of this appears to be just end-to-end testing, and whether you're alerting on a failure of testing the entire application stack or just individual components. He probably got paged by too many individual alerts versus an actual failure to serve data, which I agree would be annoying. In a previous position, we had a custom ticketing system that was designed to also be our monitoring dashboard. Alerts that were du…

I don't think it's end-to-end testing because "testing" to me implies a synthetic environment. This is about instrumenting and monitoring the production system at scale, and learning about the right things at the right time.

It certainly shares some things with end-to-end testing, and blackbox monitoring is very useful for finding high level problems with any complex networked system.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#34

Post by blackaspen »

Someone1234 wrote: post #6

Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…

I think it's simple: Developers who are on-call write better code.

As soon as you take the view of "This is crappy, but keeping it running is someone else's problem" then everything suffers (product quality, engineering quality, and reliability).

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#35

Post by learningram »

beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

Link I found of the lecture

http://vimeo.com/95073903

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#36

Post by x1798DE »

larrys wrote: post #15
beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

Would add another dimension to car vs. smoke alarms. [1] Smoke alarms are loud and typically near you (think about the one that goes off in the kitchen because of burning pizza in the oven). So you have to react or you are paralyzed. You fan it almost immediately. You have to. Car alarm typically you can't do anything (unless it's your car) and the noise, while annoying, is not as close or as loud as a smoke alarm. […

Yeah, I think this actually may explain things better than specificity. Even if the false alarm rate is low, the rate of actual, life-threatening fires is incredibly low in modern day, so chances are almost every time you hear a fire alarm go off it's alerting you essentially erroneously. I get the impression that in a home the "standard procedure" for dealing with fire alarms is to open windows, fan the smoke away from the thing and/or take the batteries out of the alarm.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#37

Post by praptak »

dchichkov wrote: post #20
jakozaur wrote: post #2

That's harder problem than I originally realized. It's easy to write noisy alerts, super easy to not have them (or not catching some issues). It's hard to tune them so signal to noise ratio will be high.

Yep. Extracting meaningfull information out of logs automatically is probably an AI-complete problem... Correct me if I wrong, but AFAIK the current state of the art solution to the alerts/log-filtering problem is: "log everything & feed these logs into a real time search engine that produces dashboard/alerts". Like elasticsearch/kibana . No? Curious, is that the approach that is being used internally at Google right…

Alerting and monitoring is not about logs. Applications export interesting signals directly in a way understood by monitoring service like Nagios. It stores the samples, draws nice graphs and supports flexible alert definition logic.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#38

Post by blackaspen »

sarahj wrote: post #26
Someone1234 wrote: post #16

Earlier quoted context omitted.

> Having the team feel direct pain is a great motivator for building robust applications But the OP is a dedicated reliability engineer, they don't build the actual applications and couldn't given the size of the company. Essentially you're being punished for other team's generated issues. You getting punished when your own stuff breaks is more a small business issue, not something corporations have.

There are definitely corporations where engineers who build the applications take responsibility for maintaining and running the applications - I happen to work for one.

This should be all corporations.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#39

Post by Moru »

beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

We just got new fire alarms installed in our building. They go off 2-4 times per day and everyone just ignores them. It's enough to toast your bread lightly brown to go off. You can easily hear the neighbours fire alarms. There is no way of removing the battery in the new alarms, the only thing you can do is take it down and stuff it in a drawer somewhere. Not even pushing the button helps. This makes them totally useless, even more so than a car alarm. We got ours exchanged and they go off much less now luckily.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#40

Post by bokchoi »

larrys wrote: post #15
beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

Would add another dimension to car vs. smoke alarms. [1] Smoke alarms are loud and typically near you (think about the one that goes off in the kitchen because of burning pizza in the oven). So you have to react or you are paralyzed. You fan it almost immediately. You have to. Car alarm typically you can't do anything (unless it's your car) and the noise, while annoying, is not as close or as loud as a smoke alarm. […

So all we need to do is have car alarms be sent directly to you (to your phone or your house) and be loud.
Post reply on HN