Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

docs.google.com

Post reply on HN
Print view View original

91–100 of 122 posts

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#91

Post by sleepydog »

jldugger wrote: post #55
dchichkov wrote: post #20

Earlier quoted context omitted.

Yep. Extracting meaningfull information out of logs automatically is probably an AI-complete problem... Correct me if I wrong, but AFAIK the current state of the art solution to the alerts/log-filtering problem is: "log everything & feed these logs into a real time search engine that produces dashboard/alerts". Like elasticsearch/kibana . No? Curious, is that the approach that is being used internally at Google right…

Logging is super fucking noisy and generally not structured for operations support. The state of the art at my not-google employers is basically Nagios scripts. Everyone has these scripts that check various components: - is the webserver running - is it responding on port 443 - does it return HTML' and maybe - 'If I submit a search, do I get a result back?' Nagios scripts are responsible for everything: opening netwo…

> As an example, imagine you're monitoring /. Nagios runs NRPE, NRPE looks at df /, and if it's 85 percent full (by default), sends a warning page. At 90 percent it sends a critical page. A smarter system collects the df / results, delivers it to a central timeseries database. The new data point is used to create a new projection, and the new projection is used to determine the time to an actual failure. The system above might have an idea of how long it takes to respond, repair, and resolve and issue a page when the disk will fill up if not responded to within 4 hours. That's the ideal solution, IMO.

We've actually implemented exactly this at my current workplace. We have a nagios check that queries graphite and calculates a "days until full" value, and alert based on that. We have similar checks for monitoring other infrastructure. These checks take a lot of work to get the right calculation and threshold values, but once they work, it's pretty great.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#92

Post by jonbarker »

Where I work, at a mobile ad network, they put everyone on call on a rotating basis even if they are not devops or server engineers. We use Pager Duty and it works well. Since there is always a primary and secondary on call person, and the company is pretty small and technical, everyone feels "responsible" during their shifts, and at least one person is capable of handling rare, catastrophic events. I often wonder which is more important: good docs on procedures for failure modes or a heightened sense of responsibility. A good analogy may be the use of commercial airline pilots. They can override autopilot, but I am told rarely do. The safest airlines are good at maintaining their heightened sense of vigilance despite the lack of the need for it 99.999% of the time.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#93

Post by philsnow »

praptak wrote: post #21

Having your application reviewed by SREs who are going to support it is a legendary experience. They have no motivation to be gentle. It changes the mindset from "Failure? Just log an error, restore some 'good'-ish state and move on to the next cool feature." towards "New cool feature? What possible failures will it cause? How about improving logging and monitoring on our existing code instead?"

In a past life, we had good luck with obligating dev teams to be (co-)oncall for their own systems until it conformed to to an agreed-upon SLA for N months, before taking on oncall for the system.

That aligned everybody's interests: devs came to SREs for design review and code review, and were incentivized to not just throw code over the wall.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#94

Post by rhizome »

beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

Another thing about car alarms: how many people even know what their own car alarm sounds like?

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#95

Post by rhizome »

falcolas wrote: post #68
breischl wrote: post #41

Earlier quoted context omitted.

I don't know about that, frequently the "cause" that you're alerting on is also a symptom. "DB down" is causing your webapp to fail, but why is the DB down? Misconfiguration, hardware failure, power outage? Heck, maybe the DB is not actually down but there was a network failure that made it unreachable from the monitoring server. My point is that alerting on a "cause" may not actually get you to the root cause, and m…

> Misconfiguration, hardware failure, power outage? Heck, maybe the DB is not actually down but there was a network failure that made it unreachable from the monitoring server. Sorry, I should have been more specific. DB daemon being down is a cause, and should be monitored. Hardware down is a cause, and should be monitored. Network availability is a cause and should be monitored. Power outage... you get my drift. I…

911: PING TO GOOGLE >100ms

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#96

Post by donavanm »

dchichkov wrote: post #62
thrownaway2424 wrote: post #50

Earlier quoted context omitted.

That seems silly though. I can replace stats on a thing that normally takes 50 usecs in a log line because it will take more than that long just to log the fact and an insane amount of cpu to analyze such a thing. The large scale systems that I personally operate produce a few KB per minute in structured stats, a few MB per second in structured logs, and hundreds of MB per second in unstructured text logs. I know whi…

To thrownaway2424. What seems silly is that processing of a few of MB per second of unstructured text logs by a real time search engine seems impossible to you. Think web-crawlers. Search engines are efficient....

youre thinking too small. Try hundreds of KB to a couple MB per second per host. And tens of thousands of hosts. Data streams at (tens of) gigabits per second are not trivial.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#97

Post by pja »

dredmorbius wrote: post #90
pja wrote: post #24

Earlier quoted context omitted.

Yeah, when I interviewe at Google, this was pretty much what I was told - one of the prices of writing "new, cool stuff" is that you get to support it, because no one else is going to be able to fix it when things go wrong.

Sounds as if that requires retiring systems when the developer leaves the company.

I would imagine that if the new thing makes the transition to being a successful, widely used service then putting a support team in place is a part of that. At the beginning though, when no-one knows whether your new thing is going to be a success or not, the dev team is it.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#98

Post by fh973 »

falcolas wrote: post #11

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…

On he topic cause vs symptom-based alerting there is one aspect missing: decoupling.

The systems that Rob is responsible for are decoupled from individual pieces of hardware (redundancy/fault-tolerance) and degrade gracefully (if one part fails only a fraction of the working set is affected). Otherwise the huge scale would not be manageable as some piece is always breaking.

So yes: if you run systems that are not somewhat decoupled from their base, cause==symptom and cause-based alerting is indeed the way to go. And you will be woken up by every single piece breaking. The way to improve this is decoupling, and then you'll also switch to symptom-based alerting.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#99

Post by Rapzid »

apposite wrote: post #74
twic wrote: post #53

Earlier quoted context omitted.

You can do alerting and monitoring through logs. I've done it myself. You can reduce the complexity of your infrastructure by converging those functions into a single set of tools. I would absolutely agree that the state of the art is capturing the evolving state of the system as a stream of events, and deriving monitoring and alerting from that stream.

Logs are typically fairly unstructured and complex to parse. For whitebox monitoring (i.e. where you have access to the code and the code can report state) you are far better off exporting state in a very well defined format to minimise parsing overhead. It also tends to make you a bit more focused on defining the characteristics of the parameter you are monitoring. You want blackbox monitoring (for close-to-user exp…

Typically yes. But I'm predicting a huge shift towards structured logging. Take a look at Serilog. It's a .Net logger I've been using recently that just has some fantastic concepts. Worth reading into even if you don't use .Net . I believe that's the direction "logging" will go... It's more eventing now I guess.

Both metrics and "logs" can be expressed as events. Those are like points in space. An incident could be like a line; a 2d event with a start and duration.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#100

Post by wereHamster »

falcolas wrote: post #69
wereHamster wrote: post #17

Earlier quoted context omitted.

> The problem is that can't act on symptoms, only research them and then act on the causes. Have a look at the example at the top of page 5.

I did. And I see the words "A human just has to parse through a few lines of text ...". Why is a human doing this? Why isn't the tool identifying the cause through a pre-determined dependency tree and only alerting on the salient points instead of relying on a human intelligently parsing through this data when they've been woken up by a page at 3am? My experience has shown me that this kind of parsing can be avoided…

Of course if the system can automatically detect the problem and fix it then it should (this is also mentioned somewhere in the document). But at some point it will have to bail and then a human has to have a look.
Post reply on HN