There's a big difference between having programmers on-call and actually having work/fires for the on-call programmers to solve. We only know about one of these for sure from this post.
Putting out fires at 37signals: The on-call programmer
11–20 of 48 posts
Re: Putting out fires at 37signals: The on-call programmer
#12I have to assume all of the other comments in this thread are from small shops that have never supported a live product.
We run a multi-hundred person team here for a live, 24/7 product, and as many as half of our developers have been scheduled as "on-call programmers," which we call our Live team. Their sole responsibility is the live, deployed product and customer-impacting issues.
They do no bug fixes outside of that. They do no feature development outside of that. There is an entire other team dedicated to those things, and like 37s, that team gets rotated through.
We also have QA dedicated to the live product, Operations dedicated to the live product, etc., etc., all separate from new feature development, because an immediate, customer-facing issue requires different prioritization than feature development.
Re: Putting out fires at 37signals: The on-call programmer
#13I have to assume all of the other comments in this thread are from small shops that have never supported a live product. We run a multi-hundred person team here for a live, 24/7 product, and as many as half of our developers have been scheduled as "on-call programmers," which we call our Live team. Their sole responsibility is the live, deployed product and customer-impacting issues. They do no bug fixes outside of t…
I've supported (what I would expect to be) an equivalently large deployment. If you truly have half of a multi-hundred person development team scheduled simply to respond to emergency on-call events, you very, very likely have fundamental issues in your development standards and processes leading to those events.
That's simply a tremendous percentage of your staff dedicated to putting out fires.
Re: Putting out fires at 37signals: The on-call programmer
#14More than whether or not they "should" or shouldn't need on-call programmers, I am curious what causes the majority of errors that are encountered. Is it mistakes the programmers have made? Unpredictable interactions that are caused by the complexity of the software? Unexpected user behavior or interactions with client software? Something else a novice like me can't anticipate?
Re: Putting out fires at 37signals: The on-call programmer
#15I have to assume all of the other comments in this thread are from small shops that have never supported a live product. We run a multi-hundred person team here for a live, 24/7 product, and as many as half of our developers have been scheduled as "on-call programmers," which we call our Live team. Their sole responsibility is the live, deployed product and customer-impacting issues. They do no bug fixes outside of t…
But root-cause analysis of the issues that arise is still important, right? Tracing customer-impacting issues back to the decisions that might have caused them? Perhaps not even particular bugs or parts of the product, but architectural decisions as a whole? Or even organizational processes? You can throw money at live support until the end of time, but the only way to reduce that cost is by addressing problems at the source, be it code or process, or something else... (right?)
Re: Putting out fires at 37signals: The on-call programmer
#16A requirement for 24/7 on-call programmers demonstrates a systemic organizational failure in the design and implementation of robust, well-architected software. 37Signals would see significant savings in development and maintenance costs -- and increased customer satisfaction -- if they approached this staffing requirement as a band-aid, not as a final solution, and took a long, considered look at the root cause of t…
What they're actually doing is conflating programmer and sysadmin here; the tasks they're assigning to the 'on-call programmer' are very similar to the ones I would expect as a sysadmin to get. Things like monitor the service, act as first responder, coordinate fixes; there's precious little that is actual programming there until you get into fixing it, and even there the goal is minimizing downtime, not debugging th…
It sounds like they have a dedicated customer support team and the on-call programmers are the second-level support. Take a look at the examples DHH gives of the work on-call programmers do-
"We spend time trying to figure out why emails weren’t delivered (often because they get caught in the client’s spam filter or their inbox is over capacity), or why an import of contacts from Excel is broken (because some formatting isn’t right), or any of the myriad of other issues that arises from having variable input and output from an application that’s been used by millions of people."
Re: Putting out fires at 37signals: The on-call programmer
#17I like how quite a number of peoples answers to the on-call programmer blog was "you need better tests"
here's a what if scenario:-
- you have a third party service your systems rely on
- at 4am on Sunday morning said 3rd party service upgrades their system, introducing a breaking change, having never bothered to notify users
- you get a call as the on-call person saying "application X is not longer working, please resolve"
How do tests stop that scenario from happening? Tests don't magically help you invent features/work around introduced issues in 3rd party systems.
Those are typically the on-call issues we deal with (we're on a weekly rotation)
Re: Putting out fires at 37signals: The on-call programmer
#18I like how quite a number of peoples answers to the on-call programmer blog was "you need better tests" here's a what if scenario:- - you have a third party service your systems rely on - at 4am on Sunday morning said 3rd party service upgrades their system, introducing a breaking change, having never bothered to notify users - you get a call as the on-call person saying "application X is not longer working, please r…
I saw only two comments regarding "more" or "better" tests. One that seemed like a sarcastic low-blow ("I’m glad to see 37signals post from last week about their minimal approach to testing is keeping the new hires busy and up late at night tracking down bugs"), and the other being "Alice" who quickly got labeled a troll.
Seems like both you and DHH are putting words in people's mouths.
Re: Putting out fires at 37signals: The on-call programmer
#19I like how quite a number of peoples answers to the on-call programmer blog was "you need better tests" here's a what if scenario:- - you have a third party service your systems rely on - at 4am on Sunday morning said 3rd party service upgrades their system, introducing a breaking change, having never bothered to notify users - you get a call as the on-call person saying "application X is not longer working, please r…
I saw only two comments regarding "more" or "better" tests. One that seemed like a sarcastic low-blow ("I’m glad to see 37signals post from last week about their minimal approach to testing is keeping the new hires busy and up late at night tracking down bugs"), and the other being "Alice" who quickly got labeled a troll. Seems like both you and DHH are putting words in people's mouths.
I'll wear that, but also I just happen to be on support this week so it's a raw subject right now ;)
Re: Putting out fires at 37signals: The on-call programmer
#20Is this seriously a post highlighting the heroics of being on-call?!
Wake up -- being on call sucks.
Being an on call programmer is even worse. All developers should have to work support sometime in their life to realize the pain of supporting software vs writing it. Only then will you realize why doing it "right" the first time really matters.
I kind of agree with the first comment on that post from Alice Young. Even though DHH just calls Alice out as trolling I know from experience that if you have on-call programmers it is a sign that your product is reaching a new level of complexity. Whether the complexity is coming from internal features or outside integrations it is probably time to take a second look at how you are handling your development processes.