Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

docs.google.com

Post reply on HN
Print view View original

11–20 of 122 posts

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#11

Post by falcolas »

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring.

Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict:

  - Pages should be [...] actionable
  - Symptoms should be monitored, not causes
The problem is that can't act on symptoms, only research them and then act on the causes. If you get an alert that says the DB is down, that's an actionable page - start the DB back up. Whereas, being paged that the connections to the DB are failing is far less actionable - you have to spend precious downtime researching the actual cause first. It could be the network, it could be an intermediary proxy, or it could be the DB itself.

Now granted, if you're only catching causes, there is the possibility you might miss something with your monitoring, and if you double up on your monitoring (that is, checking symptoms as well as causes), you could get noise. That said, most monitoring solutions (such as Nagios) include dependency chains, so you get alerted on the cause, and the symptom is silenced while the cause is in an error condition. And if you missed a cause, you still get the symptom alert and can fill your monitoring gaps from there.

Leave your research for the RCA and following development to prevent future downtime. When stuff is down, a SA's job is to get it back up.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#12

Post by sarahj »

Someone1234 wrote: post #6

Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…

As someone who is on a regular rotation - being oncall sucks. But there are definite advantages.

Having the team feel direct pain is a great motivator for building robust applications - if you know that that quick hack could lead to you getting paged at 3 in the morning you are far more likely to seek out additional solutions(anecdotally speaking) - it means you also have weight within the team to make the right engineering calls.

The team can bond over operations - it sucks being oncall, everyone knows that so everyone tries to make it less sucky, by clearing ops queues before handing off or making ticket messages a bit nicer.

When you start having ticket-free weeks or months it is an awesome feeling, the service works and is robust and your team can spend that time writing new stuff.

Additionally, when you have a small team building/maintaining a service it makes far more sense to rotate the oncall responsibility between them rather than an external engineer.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#13

Post by beat »

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing (http://en.wikipedia.org/wiki/Sensitivity_and_specificity). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms.

Think about how people react to smoke alarms versus car alarms. When the smoke alarm goes off, people mostly follow the official procedure. When car alarms go off, people ignore them. Why? Car alarms have very poor specificity.

I'd add another layer of car alarms are Not My Problem, but that's just me and not part of Dan's excellent original talk.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#14

Post by ultrasaurus »

leef wrote: post #8

"If you want a quiet oncall rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical." This is an excellent point that is missed in most monitoring setups I've seen. A classic example is some request that kills your service process. You get paged for that so you wrap the service in a supervisor like daemon. The immediate issue is fixed and, typicall…

I don't want to threadjack, but you don't have an email in your profile.

If you (or anyone else reading) ever want to talk about what "this will be a problem soon" might look like in the future drop me a line: dave@pagerduty.com

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#15

Post by larrys »

beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

Would add another dimension to car vs. smoke alarms. [1]

Smoke alarms are loud and typically near you (think about the one that goes off in the kitchen because of burning pizza in the oven). So you have to react or you are paralyzed. You fan it almost immediately. You have to.

Car alarm typically you can't do anything (unless it's your car) and the noise, while annoying, is not as close or as loud as a smoke alarm.

[1] Edit: Distance from and the decibels of the alarm and what you can do about the alarm. Of course there are also building alarms (follow procedure) and alarms in your own house "get the noise to stop!"

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#16

Post by Someone1234 »

sarahj wrote: post #12
Someone1234 wrote: post #6

Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…

As someone who is on a regular rotation - being oncall sucks. But there are definite advantages. Having the team feel direct pain is a great motivator for building robust applications - if you know that that quick hack could lead to you getting paged at 3 in the morning you are far more likely to seek out additional solutions(anecdotally speaking) - it means you also have weight within the team to make the right engi…

> Having the team feel direct pain is a great motivator for building robust applications

But the OP is a dedicated reliability engineer, they don't build the actual applications and couldn't given the size of the company. Essentially you're being punished for other team's generated issues.

You getting punished when your own stuff breaks is more a small business issue, not something corporations have.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#17

Post by wereHamster »

falcolas wrote: post #11

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…

> The problem is that can't act on symptoms, only research them and then act on the causes.

Have a look at the example at the top of page 5.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#18

Post by peterwwillis »

leef wrote: post #8

"If you want a quiet oncall rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical." This is an excellent point that is missed in most monitoring setups I've seen. A classic example is some request that kills your service process. You get paged for that so you wrap the service in a supervisor like daemon. The immediate issue is fixed and, typicall…

Re: this will be a problem soon? Metrics trending. Look for changes in your metrics to spot potential problems and plan for the future. This is done quite often for example in QA to look for issues between releases, and can be done both macro and micro in terms of continuous delivery services' metrics.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#20

Post by dchichkov »

jakozaur wrote: post #2

That's harder problem than I originally realized. It's easy to write noisy alerts, super easy to not have them (or not catching some issues). It's hard to tune them so signal to noise ratio will be high.

Yep. Extracting meaningfull information out of logs automatically is probably an AI-complete problem...

Correct me if I wrong, but AFAIK the current state of the art solution to the alerts/log-filtering problem is: "log everything & feed these logs into a real time search engine that produces dashboard/alerts". Like elasticsearch/kibana. No? Curious, is that the approach that is being used internally at Google right now? BTW, the article stated the problem/and desired outcome, but not the solution. (?)

Post reply on HN