Ask HN: What is your company's on-call rotation like?

For those doing on-call and required to be responsive to alerts at all times during your shift, one thing to think about is your pay for this burden. Some cardiologists get over $3K/day to just be on-call (whether or not anything comes up) which means keeping within range of the hospital, being sober, not being more than a few minutes away from their phone, and so on. While generally nobody's life is on the line should a server go down, if you're expected to go beyond "if I hear my phone alert and I'm sober and am near my laptop" level of effort, ensure you are compensated appropriately.

In Brazil the labor relations are heavily regulated. Each hour you are on-call you make 1/3 of a regular hour of work, and while you answer a call and work on weekends or at night you are paid twice the regular hourly wage.

You are expected to answer the phone promptly, be sober and do anything to remedy the problem including reaching for other people that can fix it (does not have to be a permanent solution).

I work for a major games developer/publisher. Our submission packages are being built this week and will be submitted next Monday. We are supposed to be on-call this weekend just in case any problems in any of the systems we were responsible arise. No, I am not paid for being on-call, just like I am not being paid overtime. From what I can tell people are going to come in this weekend just to play the game and make sure everything works even though they are not getting paid for it.

In a past life covering out of hours (typically 9pm through 6am) for a core application for an insurance company the set-up we had was that we had a ladder of 4-6 staff providing on-call support. Whom ever was top of the list was first paged and if they didn't acknowledge within x minutes we kept going down the list until somebody responded. Once you got called you owned all incidents for that night and the next morning you would be moved to the bottom of the list.

When we got called out we were well recompensed at minimum of two hours of 3x hourly rate. If you resolved incident in 10 minutes you still got 6 hours of pay. If you got another call within those first two hours then you didn't get anything extra until you were working for 2 hours and 1 minute , but if the 2nd call came in after 3 hours you started the clock again. During times of instability (new code release) we often had management agree to you working from first call-out until the day shift came in to order to minimise downtime. When we were working during the night the on-call always had priority, but if you were sat around waiting for the system to do something then we all did something else to keep us awake. But there was no expectation that it was other tickets or project work. We made personal choices of doing day work or it could be crossword puzzle etc.

We all had areas of expertise and if you caught something you didn't know how to resolve then we had the authority to ring anybody. And that person would also receive the same compensation. In 5 years doing this I never had a complaint from calling somebody else out. Did we get tired yes? Most of us had project roles or team lead roles during the day but we all knew that the production system had priority and we and our own line management would deal with the project work accordingly. Did we get burned out on this? No because the rota meant that you still had a life. If you were top of the list and had other commitments that night you'd negotiate coverage with somebody else further down the list to see who could go top for the night.

My employer did this because the cost of not doing and having downtime during the day was a lot higher. If the system failed out of hours then it could impact our business and we could lose c.8,000 hours of day shift working time that would still require salary payment. Plus reputation damage etc. Most I ever accumulated over a weekend was 32 hours at 3x (situation so bad required to restore database to last known good and then catch-up log files).

One night I also got it wrong and in trying to correct the incident forced us into a situation when we had to do a database restore. That caused about 4,000 man/hours of downtime. When my bosses boss came in at 6:30am (he got called out by my boss) I got sent home being told I was too tired to continue support, day shift staff would take over and the missive to ring him when I had slept. I made the phone call expecting a significant dressing down only to be told, everybody else who looked at the diagnostics said they would have done exactly the same, we've all learned, and you stay top of the list tonight so that we know you're not scared to get back on the horse. Truth be told it did knock my confidence and anything that was a little out of the ordinary after that I always ended up calling somebody else for a consult. After a while I engineered my self off the on-call list.

>Oh no... just no. Production is not where debugging and testing should ever happen.

Debugging sure. Testing in production however is incredibly valuable.