Ask HN: Best practices for DevOps pager/on-call schedule?

Post by notfunk » Tue, Aug 06, 2013, 11:12 PM UTC

makerops wrote: post #7

notfunk wrote: post #5

Earlier quoted context omitted.

We have not discussed this point, but I'm assuming it's off the table due to cost. And yes, this is for 24x7 rotations where everyone is in the same timezone...

I'd love to get feedback on pricing etc, to see how far off I am, if you have a second to email me? If not, nbd. http://blog.pagerduty.com/2011/03/on-call-best-practices-par... This is a series of posts that have pretty sane defaults; I personally would not do a daily rotation, but rather rotations of 5 days, and alternating weekends (one guy does M-F, one guy does Sat/Sund) and you switch off.

Nice link! And I agree the daily rotation is a bad idea, however we're leaning towards doing a Tuesday-to-Tuesday week long shift.

Post by caw » Wed, Aug 07, 2013, 1:57 AM UTC

notfunk wrote: post #10

caw wrote: post #8

Megacorp sysadmin here - we do on-call for weekly rotations, though technically anyone can get woken up for the service they own. Weekly is easy to schedule, and it lets our boss know who the contact is for the week (since the schedule is on the wiki). Never page if it's not an absolute dire emergency. One server out of a cluster - Next Business Day. Failed disk - NBD, unless you're out of hot spares. As much of your…

"During pager hand-off, last week's guy and this week's guy should talk about what happened and if there's anything they should know" Agreed, we were thinking of doing week long rotations (Tuesday - Tuesday) with a "hand off conversation" happening on Tuesdays.

Tuesday does solve the 3 day weekend problem. What do you do if Monday is a holiday? Trade on Monday morning and meet up outside of work, or just hold it till Tuesday. Most of the time we just hold it.

The reason for this discussion is because up until a certain seniority level, you get "hazard pay" for carrying the pager. You get paid 1 hour for every so many you're on call. A weekend/holiday is 24 hours instead of 8 on the day your receive it or 16 on a weekday.

You should also cover rules for holding the pager. Ours include no alcohol, and no more than 1 hour away from the site (certain emergencies may require on-site visits). You also need to respond within 20 minutes, otherwise it gets escalated, or in certain larger locations, sent to the backup on-call person.

Re: Ask HN: Best practices for DevOps pager/on-call schedule?

Re: Ask HN: Best practices for DevOps pager/on-call schedule?