Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

docs.google.com

Post reply on HN
Print view View original

81–90 of 122 posts

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#81

Post by raldi »

Moru wrote: post #39
beat wrote: post #13

This reminds me of an excellent talk my friend Dan Slimmon gave called "Car Alarms and Smoke Alarms". He relates monitoring to the concepts of sensitivity and specificity in medical testing ( http://en.wikipedia.org/wiki/Sensitivity_and_specificity ). Sensitivity is about the likelihood that your monitor will detect the error condition. Specificity is about the likelihood that it will not create false alarms. Think a…

We just got new fire alarms installed in our building. They go off 2-4 times per day and everyone just ignores them. It's enough to toast your bread lightly brown to go off. You can easily hear the neighbours fire alarms. There is no way of removing the battery in the new alarms, the only thing you can do is take it down and stuff it in a drawer somewhere. Not even pushing the button helps. This makes them totally us…

Near a kitchen, avoid ionization-based smoke alarms like the plague. Get photoelectric instead.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#82

Post by ithkuil »

falcolas wrote: post #11

> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…

"Every page should require intelligence to deal with: no robotic, scriptable responses."

This omits the implicit "because robotic, scriptable responses should be dealt by robots and scripts".

You should have monitoring for DB is down. But you should page on "product is not working". Having a monitoring system that lets you quickly find what's wrong is extremely important, but you shouldn't be woken up / or distracted for unactionable alerts.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#83

Post by eikenberry »

Someone1234 wrote: post #6

Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…

IMO the difference is ownership. That you own your code from the inception through production and that there is never a hand-off. It leads to better code and better moral.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#84

Post by shackattack »

apposite wrote: post #73
0xdeadbeefbabe wrote: post #51

So I guess the author uses a smart phone as a pager, but given his passion for uptime, reliability, latency etc. I wonder if he has experimented with an actual pager.

We use a variety of escalation techniques. As mentioned down thread, pagers are actually very unreliable. Some SREs carry pagers and mobiles. Most SREs carry phones with escalation via SMS, an actual telephone call from an automated system (after a delay), and/or escalation via a network connection over the data network. Phone calls are way way more reliable than pagers. Unacknowledged pages escalate to a secondary o…

This sort of mirrors how we do things at PagerDuty. Phone may be more reliable than SMS, but every telephony/messaging gateway fails sometimes. We use something like a dozen different phone/SMS gateways to prevent single points of failure, and do end-to-end testing of our SMS providers to check their uptime and latency.

Responders can customize their notification methods (push, SMS, phone, and email) and rules, so you can do things like get a lightweight push notification when an alert happens, and then a phone call 2 minutes later if you haven't acknowledged the incident. Teams get escalation timeouts that forward alerts up the chain if the primary hasn't responded after a period of time.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#85

Post by nostrademons »

Someone1234 wrote: post #16
sarahj wrote: post #12

Earlier quoted context omitted.

As someone who is on a regular rotation - being oncall sucks. But there are definite advantages. Having the team feel direct pain is a great motivator for building robust applications - if you know that that quick hack could lead to you getting paged at 3 in the morning you are far more likely to seek out additional solutions(anecdotally speaking) - it means you also have weight within the team to make the right engi…

> Having the team feel direct pain is a great motivator for building robust applications But the OP is a dedicated reliability engineer, they don't build the actual applications and couldn't given the size of the company. Essentially you're being punished for other team's generated issues. You getting punished when your own stuff breaks is more a small business issue, not something corporations have.

SRE has veto power over many of the design choices that go into the actual application.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#87

Post by quanticle »

lamontcg wrote: post #64
praptak wrote: post #21

Having your application reviewed by SREs who are going to support it is a legendary experience. They have no motivation to be gentle. It changes the mindset from "Failure? Just log an error, restore some 'good'-ish state and move on to the next cool feature." towards "New cool feature? What possible failures will it cause? How about improving logging and monitoring on our existing code instead?"

I like making the devs who write the application the SREs who are going to be supporting it and making them carry a pager. If its worth it to the team to write code with edge conditions that will errantly wake them up in the middle of night occasionally they can make that decision to put that tax on their own lives, not someone else's. Then the SREs are in every single code review.

This works only as long as you have a single application, or a bunch of independent applications, each with their own team. With any kind of "platform" or "framework" or shared service architecture, the incentive will be for the devs on the application teams to do as little logging as possible, because what will inevitably happen is that failures and errors with "unknown" causes will be marked as platform failures and will wake the platform team in the middle of the night. At that point, you're back to where you started. I've been in that scenario multiple times, and believe me, there are few things worse than trying to debug someone else's code at three in the morning to try to determine if the page that woke me up was a legitimate platform issue or if it's due to application code that's misbehaving or misusing the platform.

Making devs carry pagers certainly helps, but it's a mistake to think that it's a panacea, and that forcing devs to carry pagers will suddenly make them write code with perfect logging and perfect error handling.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#88

Post by jschwartzi »

bokchoi wrote: post #40
larrys wrote: post #15

Earlier quoted context omitted.

Would add another dimension to car vs. smoke alarms. [1] Smoke alarms are loud and typically near you (think about the one that goes off in the kitchen because of burning pizza in the oven). So you have to react or you are paralyzed. You fan it almost immediately. You have to. Car alarm typically you can't do anything (unless it's your car) and the noise, while annoying, is not as close or as loud as a smoke alarm. […

So all we need to do is have car alarms be sent directly to you (to your phone or your house) and be loud.

Just put a cell radio in it. As a bonus it can transmit GPS coordinates to you so you can watch it get stolen in real time.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#89

Post by eropple »

leesfer wrote: post #76
jsmeaton wrote: post #54

Earlier quoted context omitted.

Stop it. Your spam is obvious.

You win some you lose some. Hacker News is literally 100% spam, that's what a link aggregator is by definition. Some stick and some get called out. Something gets called spam just because you post it yourself, but if "someone else" posts it then it isn't spam anymore. Magic! Just like your past submissions I've been on these boards lurking for many many years and I've seen posts come and go, all of them are what you…

The idea that a link aggregator is "100% spam" may help you feel better, but has no bearing on reality. You tried to promote something you work on (without even being decent enough to admit it) and got called for it.

Don't be a jerk.

Re: My Philosophy on Alerting: Observations of a Site Reliability Engineer at Google

#90

Post by dredmorbius »

pja wrote: post #24
herge wrote: post #10

Earlier quoted context omitted.

The only people who can support or fix an application is often the team developing the application itself.

Yeah, when I interviewe at Google, this was pretty much what I was told - one of the prices of writing "new, cool stuff" is that you get to support it, because no one else is going to be able to fix it when things go wrong.

Sounds as if that requires retiring systems when the developer leaves the company.
Post reply on HN