21–30 of 122 posts
It changes the mindset from "Failure? Just log an error, restore some 'good'-ish state and move on to the next cool feature." towards "New cool feature? What possible failures will it cause? How about improving logging and monitoring on our existing code instead?"
In a previous position, we had a custom ticketing system that was designed to also be our monitoring dashboard. Alerts that were duplicates would become part of a thread, and each was either it's own ticket or part of a parent ticket. Custom rules would highlight or reassign parts of the dashboard, so critical recurrent alerts were promoted higher than urgent recurrent alerts, and none would go away until they had been addressed and closed with a specific resolution log. The whole thing was designed so a single noc engineer at 3am could close thousands of alerts per minute while logging the reason why, and keep them from recurring if it was a known issue. The noc guys even created a realtime console version so they could use a keyboard to close tickets with predefined responses just seconds after they were opened.
The only paging we had was when our end-to-end tests showed downtime for a user, which were alerts generated by some paid service providers who test your site around the globe. We caught issues before they happened by having rigorous metric trending tools.
> Err on the side of removing noisy alerts – over-monitoring is a harder problem to solve than under-monitoring. Absolutely this. Our team is having more problems with this issue than anything else. However, there are two points which seem to contradict: - Pages should be [...] actionable - Symptoms should be monitored, not causes The problem is that can't act on symptoms, only research them and then act on the cause…
Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…
The only people who can support or fix an application is often the team developing the application itself.
Earlier quoted context omitted.
As someone who is on a regular rotation - being oncall sucks. But there are definite advantages. Having the team feel direct pain is a great motivator for building robust applications - if you know that that quick hack could lead to you getting paged at 3 in the morning you are far more likely to seek out additional solutions(anecdotally speaking) - it means you also have weight within the team to make the right engi…
> Having the team feel direct pain is a great motivator for building robust applications But the OP is a dedicated reliability engineer, they don't build the actual applications and couldn't given the size of the company. Essentially you're being punished for other team's generated issues. You getting punished when your own stuff breaks is more a small business issue, not something corporations have.
Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…
For a more detailed look at Google's SRE operations, watch Ben Traynor's excellent talk "Keys to SRE": https://www.usenix.org/conference/srecon14/technical-session...
Earlier quoted context omitted.
As someone who is on a regular rotation - being oncall sucks. But there are definite advantages. Having the team feel direct pain is a great motivator for building robust applications - if you know that that quick hack could lead to you getting paged at 3 in the morning you are far more likely to seek out additional solutions(anecdotally speaking) - it means you also have weight within the team to make the right engi…
> Having the team feel direct pain is a great motivator for building robust applications But the OP is a dedicated reliability engineer, they don't build the actual applications and couldn't given the size of the company. Essentially you're being punished for other team's generated issues. You getting punished when your own stuff breaks is more a small business issue, not something corporations have.
why do you think that should be the case? I'd argue quite the opposite having done this in a large corporation with pretty decent success.
to be successful, it requires that your monitoring isn't noisy; most issues are repaired in an automated way; then when you get a "page" it is something that someone deeply familiar with the code / service is the best person to react to it. it will drive down the MTTR. and it typically gets the root cause fixed sooner in the code. these folks need to be engineers, but they don't all need to be just the devs, since internet scale services just are noisy and you need to have some randomization buffer. but those engineers need to be dedicated to the service and not a central org, where they aren't going to know how to look at logs, have depth on the intra-service interactions, the latest changes, etc. to me, this is also the best definition of "devops".
Why does a company the size of Google even have call rotations? Shouldn't they have 24/7 shifts of reliability engineers who can manually call in additional people as and when they're needed? I can totally understand why SMBs have rotations. They have less staff. But a monster corporation? This seems like lame penny pinching. Heck for the amount of effort they're clearly putting into automating these alerts, they cou…
Thanks for posting this! I'm on the product team at PagerDuty, and this lines up with a lot of our thinking on how to effectively design alerting + incident response. I love the line "Pages should be urgent, important, actionable, and real."