Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

docs.google.com

Post reply on HN
Print view View original

41–50 of 122 posts

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#41

Post by breischl »

falcolas wrote: post #11

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…

I don't know about that, frequently the "cause" that you're alerting on is also a symptom. "DB down" is causing your webapp to fail, but why is the DB down? Misconfiguration, hardware failure, power outage? Heck, maybe the DB is not actually down but there was a network failure that made it unreachable from the monitoring server.

My point is that alerting on a "cause" may not actually get you to the root cause, and maybe not even all that much closer than a symtpom.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#42

Post by r00fus »

bokchoi wrote: post #40
larrys wrote: post #15

Earlier quoted context omitted.

Would add another dimension to car vs. smoke alarms. [1] Smoke alarms are loud and typically near you (think about the one that goes off in the kitchen because of burning pizza in the oven). So you have to react or you are paralyzed. You fan it almost immediately. You have to. Car alarm typically you can't do anything (unless it's your car) and the noise, while annoying, is not as close or as loud as a smoke alarm. […

So all we need to do is have car alarms be sent directly to you (to your phone or your house) and be loud.

That'd be nice. Make it a loud part of the key fob? Maybe then the owner would be less inclined to accept a car alarm with a hair-trigger.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#43

Post by praneshp »

Moru wrote: post #39
beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

We just got new fire alarms installed in our building. They go off 2-4 times per day and everyone just ignores them. It's enough to toast your bread lightly brown to go off. You can easily hear the neighbours fire alarms. There is no way of removing the battery in the new alarms, the only thing you can do is take it down and stuff it in a drawer somewhere. Not even pushing the button helps. This makes them totally us…

If you're in the US, watch out. I think there are laws against tampering with fire alarms.

(Though I think you're in a ridiculous situation)

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#44

Post by dchichkov »

praptak wrote: post #37
dchichkov wrote: post #20

Earlier quoted context omitted.

Yep. Extracting meaningfull information out of logs automatically is probably an AI-complete problem... Correct me if I wrong, but AFAIK the current state of the art solution to the alerts/log-filtering problem is: "log everything & feed these logs into a real time search engine that produces dashboard/alerts". Like elasticsearch/kibana . No? Curious, is that the approach that is being used internally at Google right…

Alerting and monitoring is not about logs. Applications export interesting signals directly in a way understood by monitoring service like Nagios. It stores the samples, draws nice graphs and supports flexible alert definition logic.

Well, to me "applications export interesting signals directly in a way understood by monitoring service" feels like a legacy approach. It places the burden of decision "what is an interesting alert signal" and burden of structuring the log file output on the software developer! And it places that burden at an inconvinient time, when the system is still in the making.

On the other hand, by logging everything as text, and then running intellegent/structurizing real time search engine over logs one can make/modify these decisions at a later time. And it can be done both by devs/ops, without touching the source code!

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#45

Post by dllthomas »

r00fus wrote: post #42
bokchoi wrote: post #40

Earlier quoted context omitted.

So all we need to do is have car alarms be sent directly to you (to your phone or your house) and be loud.

That'd be nice. Make it a loud part of the key fob? Maybe then the owner would be less inclined to accept a car alarm with a hair-trigger.

And more likely to notice, and more able to find his keys... seems like a win all 'round.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#46

Post by dllthomas »

x1798DE wrote: post #36
larrys wrote: post #15

Earlier quoted context omitted.

Would add another dimension to car vs. smoke alarms. [1] Smoke alarms are loud and typically near you (think about the one that goes off in the kitchen because of burning pizza in the oven). So you have to react or you are paralyzed. You fan it almost immediately. You have to. Car alarm typically you can't do anything (unless it's your car) and the noise, while annoying, is not as close or as loud as a smoke alarm. […

Yeah, I think this actually may explain things better than specificity. Even if the false alarm rate is low, the rate of actual, life-threatening fires is incredibly low in modern day, so chances are almost every time you hear a fire alarm go off it's alerting you essentially erroneously. I get the impression that in a home the "standard procedure" for dealing with fire alarms is to open windows, fan the smoke away f…

It depends. The rate of smoke alarms going off inadvertently while someone is cooking is high compared to the actual incidence of fires. The rate of smoke alarms going of when no one is doing anything heat-related is clearly lower. If you are a person who just burned some toast, that is almost certainly the reason the smoke alarm is going off, and it's entirely reasonable to address the symptoms. If you are a person who woke up to a smoke alarm, and no one else was cooking anything, things are a little different.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#47

Post by rco8786 »

Someone1234 wrote: post #6

Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…

There's a lot to be said for having a domain expert be on call. Most issues can be fixed really quickly, but if it's your first time ever seeing an alert or working on a service, you're pretty much boned and end up paging the domain expert anyway.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#49

Post by scoot »

Moru wrote: post #39
beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

We just got new fire alarms installed in our building. They go off 2-4 times per day and everyone just ignores them. It's enough to toast your bread lightly brown to go off. You can easily hear the neighbours fire alarms. There is no way of removing the battery in the new alarms, the only thing you can do is take it down and stuff it in a drawer somewhere. Not even pushing the button helps. This makes them totally us…

> Not even pushing the button helps.

If it's like every other smoke detector I've encountered, the bottom is to test the alarm, not to cancel it.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#50

Post by thrownaway2424 »

dchichkov wrote: post #44
praptak wrote: post #37

Earlier quoted context omitted.

Alerting and monitoring is not about logs. Applications export interesting signals directly in a way understood by monitoring service like Nagios. It stores the samples, draws nice graphs and supports flexible alert definition logic.

Well, to me "applications export interesting signals directly in a way understood by monitoring service" feels like a legacy approach. It places the burden of decision "what is an interesting alert signal" and burden of structuring the log file output on the software developer! And it places that burden at an inconvinient time, when the system is still in the making. On the other hand, by logging everything as text,…

That seems silly though. I can replace stats on a thing that normally takes 50 usecs in a log line because it will take more than that long just to log the fact and an insane amount of cpu to analyze such a thing. The large scale systems that I personally operate produce a few KB per minute in structured stats, a few MB per second in structured logs, and hundreds of MB per second in unstructured text logs. I know which of these I'd rather use for monitoring.
Post reply on HN