Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

docs.google.com

Post reply on HN
Print view View original

101–110 of 122 posts

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#101

Post by retroencabulato »

twp wrote: post #27
Someone1234 wrote: post #6

Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…

Google does spread oncall rotations across multiple timezones. Most SREs are oncall only during the day, with the local nightshift being somebody else's dayshift. For a more detailed look at Google's SRE operations, watch Ben Traynor's excellent talk "Keys to SRE": https://www.usenix.org/conference/srecon14/technical-session...

That was an insightful talk, thanks.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#102

Post by jon-wood »

Moru wrote: post #39
beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

We just got new fire alarms installed in our building. They go off 2-4 times per day and everyone just ignores them. It's enough to toast your bread lightly brown to go off. You can easily hear the neighbours fire alarms. There is no way of removing the battery in the new alarms, the only thing you can do is take it down and stuff it in a drawer somewhere. Not even pushing the button helps. This makes them totally us…

Somehow our rented house has three separate smoke detectors clustered around the kitchen, all of which have been renamed cooking detectors, since the slightest hint of heat seems to set them off, triggering every other detector in the house to alarm as well.

They do at least have a snooze button on them, but I was pretty close to buying a set of the Nest alarms out of my own pocket and replacing them until I saw the price tag.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#104

Post by never_snapped »

thrownaway2424 wrote: post #52
Someone1234 wrote: post #6

Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…

That's exactly what Google does except some teams have triple rotations like Mountain View/Sydney/Zurich. I'm not sure why you think the word "rotation" means something else.

I've worked for companies in the past where a "rotation" was you were on call 24/7 for a week or longer. Worked all night? Still expected at the office no later than 9 a.m.

It was exhausting as the people on call didn't have the power to actually fix the system. Instead, they would have to walk someone else through the steps over the phone. If it took a code change to fix the system, too bad that was at least two weeks of red tape, and every single time the error occurred they had to page the person on call to walk the person through over the phone to verify it was the same error and nothing could be done.

Think "big business," "division of responsibilities," "accountability," inept management who never had to suffer under the policies they demanded, and a toxic culture which was proud to "give everything they have for the product."

Luckily, I don't work for those shitty companies anymore.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#105

Post by x1798DE »

dllthomas wrote: post #46
x1798DE wrote: post #36

Earlier quoted context omitted.

Yeah, I think this actually may explain things better than specificity. Even if the false alarm rate is low, the rate of actual, life-threatening fires is incredibly low in modern day, so chances are almost every time you hear a fire alarm go off it's alerting you essentially erroneously. I get the impression that in a home the "standard procedure" for dealing with fire alarms is to open windows, fan the smoke away f…

It depends. The rate of smoke alarms going off inadvertently while someone is cooking is high compared to the actual incidence of fires. The rate of smoke alarms going of when no one is doing anything heat-related is clearly lower. If you are a person who just burned some toast, that is almost certainly the reason the smoke alarm is going off, and it's entirely reasonable to address the symptoms. If you are a person…

I don't know if I've ever experienced personal (home) fire alarms going off for no reason, and I've never had them go off because of an actual fire, but I have had them go off when someone else is cooking (alerting me that someone is cooking, whether or not I knew that), or due to steam from a shower, if they are placed wrong - again, whether or not I knew someone was taking a shower.

In corporate/organizational environments - often in buildings made entirely of concrete - I've had them go off very frequently, but almost never as a result of a dangerous condition - usually either a drill, a small fire that was immediately put out, someone pulling the fire alarm as a prank/protest or something else like someone burning popcorn or something.

I still think the point stands.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#106

Post by luch »

beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

On the same topic, I've heard that every supertanker has a proximity radar in order to avoid to collide with small boats.

However, in some heavy-traffic routes like the Straight of Dover or Gibraltar the proximity alarm goes off constantly. So the commanding officers usually disable them, leading to a good number of accidents with fishermen boats.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#107

Post by dllthomas »

x1798DE wrote: post #105
dllthomas wrote: post #46

Earlier quoted context omitted.

It depends. The rate of smoke alarms going off inadvertently while someone is cooking is high compared to the actual incidence of fires. The rate of smoke alarms going of when no one is doing anything heat-related is clearly lower. If you are a person who just burned some toast, that is almost certainly the reason the smoke alarm is going off, and it's entirely reasonable to address the symptoms. If you are a person…

I don't know if I've ever experienced personal (home) fire alarms going off for no reason, and I've never had them go off because of an actual fire, but I have had them go off when someone else is cooking (alerting me that someone is cooking, whether or not I knew that), or due to steam from a shower, if they are placed wrong - again, whether or not I knew someone was taking a shower. In corporate/organizational envi…

Which point?

My point was that in a (nonadversarial) context where you have known triggers for false alarms, P(problem|alarm) falls dramatically once you have identified the presence of such a trigger, and it's entirely reasonable for our actions to reflect this.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#108

Post by dhpe »

My startup http://usetrace.com is a web monitoring (+regression testing) tool with the "monitor for your users" philosophy mentioned in Rob's article. Monitoring is done on the application/feature level -> alerts are always about a feature visible to the users.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#109

Post by AloisReitbauer »

Good article. Alerting system unfortunately are still at the same level they where decades ago. Today we work in highly distributed environments that scale dynamically and we finding symptoms is a key problem. That is why a lot of people alert on causes or anomalies. In reality they should just detect them and log them for further dependency analysis once a real problem is found. We for example differentiate between three levels of alerts: infrastructure only, application services and users. Our approach to have NO alerts at all but monitor a ton of potential anomalies. Once these anomalies have user impact we report back problem dependencies.

If you are interested you can also get my point of view from my Velocity talk on Monitoring without alerts. https://www.youtube.com/watch?v=Gqqb8zEU66s. If you are interested also check out www.ruxit.com and let me know what you think of our approach.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#110

Post by gfodor »

ithkuil wrote: post #82
falcolas wrote: post #11

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…

"Every page should require intelligence to deal with: no robotic, scriptable responses." This omits the implicit "because robotic, scriptable responses should be dealt by robots and scripts". You should have monitoring for DB is down. But you should page on "product is not working". Having a monitoring system that lets you quickly find what's wrong is extremely important, but you shouldn't be woken up / or distracted…

can't upvote enough. not every monitoring check has to page. this is the key insight -- more visibility is always better, more alerting is not.
Post reply on HN