So I guess the author uses a smart phone as a pager, but given his passion for uptime, reliability, latency etc. I wonder if he has experimented with an actual pager.
My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
51–60 of 122 posts
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#52Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…
That's exactly what Google does except some teams have triple rotations like Mountain View/Sydney/Zurich. I'm not sure why you think the word "rotation" means something else.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#53Earlier quoted context omitted.
Yep. Extracting meaningfull information out of logs automatically is probably an AI-complete problem... Correct me if I wrong, but AFAIK the current state of the art solution to the alerts/log-filtering problem is: "log everything & feed these logs into a real time search engine that produces dashboard/alerts". Like elasticsearch/kibana . No? Curious, is that the approach that is being used internally at Google right…
Alerting and monitoring is not about logs. Applications export interesting signals directly in a way understood by monitoring service like Nagios. It stores the samples, draws nice graphs and supports flexible alert definition logic.
You can do alerting and monitoring through logs. I've done it myself. You can reduce the complexity of your infrastructure by converging those functions into a single set of tools. I would absolutely agree that the state of the art is capturing the evolving state of the system as a stream of events, and deriving monitoring and alerting from that stream.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#54Great write up! I think alerts are some of the most important things you can set up on a website in 2014. I've been working in conversion optimization for the past 7 years and over that time I've discovered one thing: conversion optimization is dead. Just doesn't work anymore. I recently started telling people that conversions 2.0 is all about knowing what is going on with your site and reacting as fast as possible t…
Stop it. Your spam is obvious.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#55That's harder problem than I originally realized. It's easy to write noisy alerts, super easy to not have them (or not catching some issues). It's hard to tune them so signal to noise ratio will be high.
Yep. Extracting meaningfull information out of logs automatically is probably an AI-complete problem... Correct me if I wrong, but AFAIK the current state of the art solution to the alerts/log-filtering problem is: "log everything & feed these logs into a real time search engine that produces dashboard/alerts". Like elasticsearch/kibana . No? Curious, is that the approach that is being used internally at Google right…
Logging is super fucking noisy and generally not structured for operations support. The state of the art at my not-google employers is basically Nagios scripts. Everyone has these scripts that check various components:
- is the webserver running - is it responding on port 443 - does it return HTML' and maybe - 'If I submit a search, do I get a result back?'
Nagios scripts are responsible for everything: opening network connections, querying system internals, collecting metrics, interpreting results, and boiling it down to a number between 0 and 3 and an unstructured text output to stdout.
A few of us understand that what we need is a more structured, data driven approach. Collect base metrics first, build a time series, apply a projection, and feed that projection in to a system that understands the actual failure condition.
As an example, imagine you're monitoring /. Nagios runs NRPE, NRPE looks at df /, and if it's 85 percent full (by default), sends a warning page. At 90 percent it sends a critical page. A smarter system collects the df / results, delivers it to a central timeseries database. The new data point is used to create a new projection, and the new projection is used to determine the time to an actual failure. The system above might have an idea of how long it takes to respond, repair, and resolve and issue a page when the disk will fill up if not responded to within 4 hours. That's the ideal solution, IMO.
It doesn't exist, AFAIK. There's a massive backlog of scripts that were written in the monolothic Nagios model that need to be rewritten, and thus this newer better version is always imaginary.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#56So I guess the author uses a smart phone as a pager, but given his passion for uptime, reliability, latency etc. I wonder if he has experimented with an actual pager.
It turns out that pagers are often unreliable; while I've never had a message not get through eventually, I have frequently seen messages get delayed for hours or even until the pager was power-cycled.
On the other hand, if you set up an IMAPS account for each person's alert address, and then have their smartphone use that account to check email, reliability is quite high. (Strangely, sysadmins tend to have wifi in their houses and good data plans, especially when the company pays for it.)
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#57Great article @robewaschuk :)
-- Marcin, former Google SRE
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#58So I guess the author uses a smart phone as a pager, but given his passion for uptime, reliability, latency etc. I wonder if he has experimented with an actual pager.
It turns out that pagers are often unreliable; while I've never had a message not get through eventually, I have frequently seen messages get delayed for hours or even until the pager was power-cycled. On the other hand, if you set up an IMAPS account for each person's alert address, and then have their smartphone use that account to check email, reliability is quite high. (Strangely, sysadmins tend to have wifi in t…
Ahhh I thought the medical profession had it figured out, but these unreliable pagers could be why the doctor is always late for the delivery.
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#59Very cool
Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google
#60Earlier quoted context omitted.
Re: this will be a problem soon? Metrics trending. Look for changes in your metrics to spot potential problems and plan for the future. This is done quite often for example in QA to look for issues between releases, and can be done both macro and micro in terms of continuous delivery services' metrics.
I think anything that requires "spotting potential problems" is only a partial solution. I've never seen a compelling system that can look at all the metrics and (with reasonable precision and recall) spot and summarize changes that are actually problematic and surprising to humans. It's definitely a necessary part of observing what's going on (and quickly eliminating hypotheses like "maybe we're out of CPU!"), for s…
These subcritical alerts deserve better but different handling: they can almost always be delivered to a non-paging email address, either a relevant internal mailing list or a ticket queue, where they can be investigated during normal office hours.
The other useful tip I have is to put URLs to internal wikis and/or tickets in the alert body. We write documentation for these to a 3AM standard: if I can't understand it immediately after being woken up at 3AM, it's not clear or actionable enough.