My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
1–10 of 122 posts
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#2That's harder problem than I originally realized. It's easy to write noisy alerts, super easy to not have them (or not catching some issues).
It's hard to tune them so signal to noise ratio will be high.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#3Thanks for posting this! I'm on the product team at PagerDuty, and this lines up with a lot of our thinking on how to effectively design alerting + incident response. I love the line "Pages should be urgent, important, actionable, and real."
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#4Great write up! I think alerts are some of the most important things you can set up on a website in 2014.
I've been working in conversion optimization for the past 7 years and over that time I've discovered one thing: conversion optimization is dead. Just doesn't work anymore. I recently started telling people that conversions 2.0 is all about knowing what is going on with your site and reacting as fast as possible to it.
That's where alerting comes in, and you NEED this, and the big players are taking note of that. Pingdom (of course) and Google Analytics has some simple alerts built in.
I have a strange feeling alerts are going to be one of the most talked about web things in the next year or so. Especially with all these little guys I've seen popping up in the space lately. Smaller guys like https://www.gosquared.com/ and https://chartbeat.com/ are starting to build alerting tools and I've seen one or two startups dedicated to alerting like http://www.pagerduty.com/ and http://www.voodooalerts.com/
Big things are coming in this space for sure. Keep and eye on it. I think some big players will start emerging and some good money will be thrown into this. Developers, start your engines.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#5This is huge. One of the big things dev teams benefit from bringing an SRE team onto their project is learning things like this and how to run a sustainable oncall rotation.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#6Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed?
I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they could likely use the same wage-hours to just hire someone else for a shift. Heck with an international company like Google they could have UK-based staff monitoring US-based sites overnight and visa-versa. Keep everyone on 9-5 and still get 24 hour engineers at their desks.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#7Great writeup. Should be in any operations handbook. One of the challenges I've found has been dynamic urgency, which is to say something is urgent when it first comes up, but now that its known and being addressed it isn't urgent anymore, unless there is something else going on we don't know about.
Example you get a server failure which affects a service, and you begin working on replacing that server with a backup, but a switch is also dropping packets and so you are getting alerts on degraded service (symptom) but believe you are fixing that cause (down server) when in fact you will still have a problem after the server is restored. So my challenge is figuring out how to alert on that additional input in a way that folks won't just say "oh yeah, this service, we're working on it already."
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#8"If you want a quiet oncall rotation, it's imperative to have a system for dealing with things that need timely response, but are not imminently critical."
This is an excellent point that is missed in most monitoring setups I've seen. A classic example is some request that kills your service process. You get paged for that so you wrap the service in a supervisor like daemon. The immediate issue is fixed and, typically, any future causes of the service process dying are hidden unless someone happens to be looking at the logs one day.
I would love to see smart ways to surface "this will be a problem soon" on alerting systems.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#9In todays world, 90% of bloggers rely on google for their living
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#10Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…
The only people who can support or fix an application is often the team developing the application itself.