Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

docs.google.com

Post reply on HN
Print view View original

61–70 of 122 posts

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#61

Post by meowface »

dchichkov wrote: post #20
jakozaur wrote: post #2

That's harder problem than I originally realized. It's easy to write noisy alerts, super easy to not have them (or not catching some issues). It's hard to tune them so signal to noise ratio will be high.

Yep. Extracting meaningfull information out of logs automatically is probably an AI-complete problem... Correct me if I wrong, but AFAIK the current state of the art solution to the alerts/log-filtering problem is: "log everything & feed these logs into a real time search engine that produces dashboard/alerts". Like elasticsearch/kibana . No? Curious, is that the approach that is being used internally at Google right…

We use Splunk at my organization to handle alerting and paging (and lots of other things).

Generally speaking, it works out pretty well for us. If you're a Splunk query guru you can also correlate and/or combine multiple disparate logs in elaborate ways to create more complex alert conditions.

The same can presumably be done with Elasticsearch/Logstash/Kibana.

We're actually security incident response, not reliability incident response, so our goals and methods differ a bit but the core concepts are all the same.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#62

Post by dchichkov »

thrownaway2424 wrote: post #50
dchichkov wrote: post #44

Earlier quoted context omitted.

Well, to me "applications export interesting signals directly in a way understood by monitoring service" feels like a legacy approach. It places the burden of decision "what is an interesting alert signal" and burden of structuring the log file output on the software developer! And it places that burden at an inconvinient time, when the system is still in the making. On the other hand, by logging everything as text,…

That seems silly though. I can replace stats on a thing that normally takes 50 usecs in a log line because it will take more than that long just to log the fact and an insane amount of cpu to analyze such a thing. The large scale systems that I personally operate produce a few KB per minute in structured stats, a few MB per second in structured logs, and hundreds of MB per second in unstructured text logs. I know whi…

To thrownaway2424. What seems silly is that processing of a few of MB per second of unstructured text logs by a real time search engine seems impossible to you. Think web-crawlers. Search engines are efficient....

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#63

Post by bane »

beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

A similar lesson most of us are probably familiar with. http://en.m.wikipedia.org/wiki/Boy_who_cried_wolf

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#64

Post by lamontcg »

praptak wrote: post #21

Having your application reviewed by SREs who are going to support it is a legendary experience. They have no motivation to be gentle. It changes the mindset from "Failure? Just log an error, restore some 'good'-ish state and move on to the next cool feature." towards "New cool feature? What possible failures will it cause? How about improving logging and monitoring on our existing code instead?"

I like making the devs who write the application the SREs who are going to be supporting it and making them carry a pager.

If its worth it to the team to write code with edge conditions that will errantly wake them up in the middle of night occasionally they can make that decision to put that tax on their own lives, not someone else's.

Then the SREs are in every single code review.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#65

Post by bane »

beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

A similar lesson most of us are probably familiar with. http://en.m.wikipedia.org/wiki/Boy_who_cried_wolf

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#66

Post by kevinbowman »

scoot wrote: post #49
Moru wrote: post #39

Earlier quoted context omitted.

We just got new fire alarms installed in our building. They go off 2-4 times per day and everyone just ignores them. It's enough to toast your bread lightly brown to go off. You can easily hear the neighbours fire alarms. There is no way of removing the battery in the new alarms, the only thing you can do is take it down and stuff it in a drawer somewhere. Not even pushing the button helps. This makes them totally us…

> Not even pushing the button helps. If it's like every other smoke detector I've encountered, the bottom is to test the alarm, not to cancel it.

I've got a mains-attached one (ie not solely battery powered), and the button is "hush" which silences it for 5 mins (although it chirps once per minute to remind you that it's hushed). If you hold the button whilst it's not going off, it tests the alarm.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#67

Post by cdr »

scoot wrote: post #49
Moru wrote: post #39

Earlier quoted context omitted.

We just got new fire alarms installed in our building. They go off 2-4 times per day and everyone just ignores them. It's enough to toast your bread lightly brown to go off. You can easily hear the neighbours fire alarms. There is no way of removing the battery in the new alarms, the only thing you can do is take it down and stuff it in a drawer somewhere. Not even pushing the button helps. This makes them totally us…

> Not even pushing the button helps. If it's like every other smoke detector I've encountered, the bottom is to test the alarm, not to cancel it.

All the storebought smoke alarms I've seen in the past few years have two buttons (or one button with two functions) - one to test, one to shut it off for a few minutes when something besides a housefire is making it go off.

It may vary by country I suppose.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#68

Post by falcolas »

breischl wrote: post #41
falcolas wrote: post #11

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…

I don't know about that, frequently the "cause" that you're alerting on is also a symptom. "DB down" is causing your webapp to fail, but why is the DB down? Misconfiguration, hardware failure, power outage? Heck, maybe the DB is not actually down but there was a network failure that made it unreachable from the monitoring server. My point is that alerting on a "cause" may not actually get you to the root cause, and m…

> Misconfiguration, hardware failure, power outage? Heck, maybe the DB is not actually down but there was a network failure that made it unreachable from the monitoring server.

Sorry, I should have been more specific. DB daemon being down is a cause, and should be monitored. Hardware down is a cause, and should be monitored. Network availability is a cause and should be monitored. Power outage... you get my drift.

I think the root cause of my disagreement with this document is the lack of a proper dependency tree in the alerting tool. Their tool appears to want to alert on any and every monitored problem, which necessitates limiting what you monitor for fear of a pager flood. A proper tool can address this problem correctly.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#69

Post by falcolas »

wereHamster wrote: post #17
falcolas wrote: post #11

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…

> The problem is that can't act on symptoms, only research them and then act on the causes. Have a look at the example at the top of page 5.

I did. And I see the words "A human just has to parse through a few lines of text ...".

Why is a human doing this? Why isn't the tool identifying the cause through a pre-determined dependency tree and only alerting on the salient points instead of relying on a human intelligently parsing through this data when they've been woken up by a page at 3am?

My experience has shown me that this kind of parsing can be avoided by using the right tools.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#70

Post by sysworld »

Moru wrote: post #39
beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

We just got new fire alarms installed in our building. They go off 2-4 times per day and everyone just ignores them. It's enough to toast your bread lightly brown to go off. You can easily hear the neighbours fire alarms. There is no way of removing the battery in the new alarms, the only thing you can do is take it down and stuff it in a drawer somewhere. Not even pushing the button helps. This makes them totally us…

Your lucky, when someone in my building triggers the fire alarm for there flat, the really loud alarm sounds in all the flats in the building, with an audio note saying that someone has triggered the fire alarm and you can leave if you want, but it's not required. I guess it will tell you when they have confirmed it's a real fire, and you really have to leave.
Post reply on HN