Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

docs.google.com

Post reply on HN
Print view View original

71–80 of 122 posts

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#71

Post by falcolas »

robewaschuk wrote: post #23
falcolas wrote: post #11

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…

I don't think there's a contradiction there. "Research them and act on the causes" is _exactly_ the action you should take on a symptom-based alert. Yes, the ultimate action is always to try to address and eliminate the "rootest root cause", but if you start from the symptom you will potentially see more ways to mitigate it then if you start from "database server disk is bad."

Perhaps not the best choice of problem to use as an example - a Database server disk is bad problem has one real solution, and will cause a whole flood of seemingly unrelated problems.

Knowing immediately that it's the server disk (more realistically a raid array going into recovery mode) will save you a lot of time and effort troubleshooting what would appear as a sporadic slow response issue. There's dozens of potential causes for poor response times, of which a raid array in recovery mode is just one.

And once you know it's a raid array in recovery mode, you can then take immediate action, something you can't do if you are still busy troubleshooting a sporadic response slowness issue.

I feel that ultimately, there's no problem with monitoring for high level symptoms, but they should not be the goal state of monitoring. The goal should be to monitor all possible causes of problems to limit the troubleshooting the SA has to do at 3am when woken by a page.

Plus, you should be using a tool which properly silences high level symptoms if there's a problem with a system which is clearly identified as a parent. That is to say, a server being down will silence "db is not responding" alerts.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#72

Post by falcolas »

jakozaur wrote: post #2

That's harder problem than I originally realized. It's easy to write noisy alerts, super easy to not have them (or not catching some issues). It's hard to tune them so signal to noise ratio will be high.

The real trick, in my experience, is to treat every page as an actionable item, even if, no, especially if the action is to change alerting thresholds.

Doing this has taken us, in the past, from 400 pages a day to under 100 a day, over the course of a week's worth of effort.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#73

Post by apposite »

0xdeadbeefbabe wrote: post #51

So I guess the author uses a smart phone as a pager, but given his passion for uptime, reliability, latency etc. I wonder if he has experimented with an actual pager.

We use a variety of escalation techniques. As mentioned down thread, pagers are actually very unreliable. Some SREs carry pagers and mobiles. Most SREs carry phones with escalation via SMS, an actual telephone call from an automated system (after a delay), and/or escalation via a network connection over the data network. Phone calls are way way more reliable than pagers.

Unacknowledged pages escalate to a secondary oncaller (e.g. if the oncall is out of range/in a tunnel, under a bus) and tertiary depending on configuration (and then it loops, or falls to another rotation, again depending on configuration). The code and services that do escalations is deliberately and carefully vetted to have minimal overlap with production systems (who's failure they might be alerting people to).

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#74

Post by apposite »

twic wrote: post #53
praptak wrote: post #37

Earlier quoted context omitted.

Alerting and monitoring is not about logs. Applications export interesting signals directly in a way understood by monitoring service like Nagios. It stores the samples, draws nice graphs and supports flexible alert definition logic.

You can do alerting and monitoring through logs. I've done it myself. You can reduce the complexity of your infrastructure by converging those functions into a single set of tools. I would absolutely agree that the state of the art is capturing the evolving state of the system as a stream of events, and deriving monitoring and alerting from that stream.

Logs are typically fairly unstructured and complex to parse. For whitebox monitoring (i.e. where you have access to the code and the code can report state) you are far better off exporting state in a very well defined format to minimise parsing overhead. It also tends to make you a bit more focused on defining the characteristics of the parameter you are monitoring.

You want blackbox monitoring (for close-to-user experience) AND whitebox monitoring (which provide diagnostics of internal state for debugging). True blackbox monitoring is often pretty unreliable so you are usually better of alerting on whitebox reported state of end user perceivable variables, e.g. HTTP error codes, latency and so on.

State of the art is to report a staggering amount of data about the internal state of a server. I mean a lot. 10s to 100s of times the number of parameters you are probably used to seeing.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#75

Post by orangesareok »

ChuckMcM wrote: post #7

Great writeup. Should be in any operations handbook. One of the challenges I've found has been dynamic urgency, which is to say something is urgent when it first comes up, but now that its known and being addressed it isn't urgent anymore, unless there is something else going on we don't know about. Example you get a server failure which affects a service, and you begin working on replacing that server with a backup,…

This is definitely the biggest problem in where I'm working now. We have a lot of monitoring via Wily Introscope, but the biggest thing is relating failures of different components together. E.g. one service layer fails so some queue gets backed up so some application server starts timing out.

The amount of noise that starts coming in when there is some major outage (say some mainframe system fails) is ridiculous.

Right now where I work they solve it by throwing manpower at the problem tbh.

It takes a lot of work by the application owners all working together to really get a coherent picture of how the services are interdependant, but the applications are so large, old code, etc - normal problems I guess a lot of companies face, that its almost impossible to find people who have a complete end to end understanding of most transcations.

Side note: my only monitoring experience is with Wily - anyone have opinions to hare on it?

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#76

Post by leesfer »

jsmeaton wrote: post #54
leesfer wrote: post #4

Great write up! I think alerts are some of the most important things you can set up on a website in 2014. I've been working in conversion optimization for the past 7 years and over that time I've discovered one thing: conversion optimization is dead. Just doesn't work anymore. I recently started telling people that conversions 2.0 is all about knowing what is going on with your site and reacting as fast as possible t…

Stop it. Your spam is obvious.

You win some you lose some. Hacker News is literally 100% spam, that's what a link aggregator is by definition. Some stick and some get called out.

Something gets called spam just because you post it yourself, but if "someone else" posts it then it isn't spam anymore. Magic! Just like your past submissions

I've been on these boards lurking for many many years and I've seen posts come and go, all of them are what you would call spam. Just people pitching their products

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#77

Post by jrochkind1 »

falcolas wrote: post #11

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…

Yep, I thought of that when I was reading it too, but he also talked about having a 'dashboard' system that lets engineers easily determine causes for symptoms. His argument is _heavily_ dependent on having such a system -- and his argument about symptoms-not-causes _only_ about _alerts_ (specifically pages that go directly and immediately to engineers).

In that context, I find it persuasive based on my own experience. (And I wish I had the kind of dashboard he described available; I'd love to see a presentation or blog post about _that_!)

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#78

Post by yuliyp »

dllthomas wrote: post #45
r00fus wrote: post #42

Earlier quoted context omitted.

That'd be nice. Make it a loud part of the key fob? Maybe then the owner would be less inclined to accept a car alarm with a hair-trigger.

And more likely to notice, and more able to find his keys... seems like a win all 'round.

And if someone parks like an asshole, you trigger their car alarm and embarrass him in front of everyone they're around. bonus win.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#79

Post by grayclhn »

Moru wrote: post #39
beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

We just got new fire alarms installed in our building. They go off 2-4 times per day and everyone just ignores them. It's enough to toast your bread lightly brown to go off. You can easily hear the neighbours fire alarms. There is no way of removing the battery in the new alarms, the only thing you can do is take it down and stuff it in a drawer somewhere. Not even pushing the button helps. This makes them totally us…

Not necessarily recommended, but wrapping smoke alarms in plastic bags usually prevents them from going off. Of course...

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#80

Post by mkopinsky »

falcolas wrote: post #72
jakozaur wrote: post #2

That's harder problem than I originally realized. It's easy to write noisy alerts, super easy to not have them (or not catching some issues). It's hard to tune them so signal to noise ratio will be high.

The real trick, in my experience, is to treat every page as an actionable item, even if, no, especially if the action is to change alerting thresholds. Doing this has taken us, in the past, from 400 pages a day to under 100 a day, over the course of a week's worth of effort.

If something not-so-bad-this-time happens while I'm eating dinner, I am not taking time right then and there to figure out exactly when we should and shouldn't alert for that event. That demands a level of analytical thinking that can take place during work hours.
Post reply on HN