Putting out fires at 37signals: The on-call programmer
1–10 of 48 posts
Re: Putting out fires at 37signals: The on-call programmer
#2A requirement for 24/7 on-call programmers demonstrates a systemic organizational failure in the design and implementation of robust, well-architected software.
37Signals would see significant savings in development and maintenance costs -- and increased customer satisfaction -- if they approached this staffing requirement as a band-aid, not as a final solution, and took a long, considered look at the root cause of this systemic failure.
Re: Putting out fires at 37signals: The on-call programmer
#3A requirement for 24/7 on-call programmers demonstrates a systemic organizational failure in the design and implementation of robust, well-architected software. 37Signals would see significant savings in development and maintenance costs -- and increased customer satisfaction -- if they approached this staffing requirement as a band-aid, not as a final solution, and took a long, considered look at the root cause of t…
What they're actually doing is conflating programmer and sysadmin here; the tasks they're assigning to the 'on-call programmer' are very similar to the ones I would expect as a sysadmin to get. Things like monitor the service, act as first responder, coordinate fixes; there's precious little that is actual programming there until you get into fixing it, and even there the goal is minimizing downtime, not debugging the change into working on the live system; typically "revert whatever changed" is one of the first tools for this work.
An actual need to do programming instantly and with no warning is quite a different proposition, not one that I'm aware of having ever been needed in any of the companies I've worked for.
Re: Putting out fires at 37signals: The on-call programmer
#4Don't be too quick to condemn 37signals for needing on-call programmers. For many startups, the process goes like this: all devs are always on-call. It seems that 37signals at least makes the requirements of the job clear. The fact is, running a live service almost always requires some degree of live support. (Even the most robust production software will experience the occasional hiccup.)
But it does seem like they're throwing money at the band-aids. Would love to see an article addressing how to fix the root of these sorts of problems, instead of just outlining how they put out all of their fires.
Re: Putting out fires at 37signals: The on-call programmer
#5A requirement for 24/7 on-call programmers demonstrates a systemic organizational failure in the design and implementation of robust, well-architected software. 37Signals would see significant savings in development and maintenance costs -- and increased customer satisfaction -- if they approached this staffing requirement as a band-aid, not as a final solution, and took a long, considered look at the root cause of t…
Someone made this point in a direct response to the post, and was quickly dismissed as a troll (as well as being treated to a particularly snarky response from DHH).
While I certainly have to think that 37signals knows what they are doing, the need for a 24/7 programmer does sound a little strange. Perhaps it's just a question of semantics, as the post describes dev/ops duties more than anything else.
Re: Putting out fires at 37signals: The on-call programmer
#6A requirement for 24/7 on-call programmers demonstrates a systemic organizational failure in the design and implementation of robust, well-architected software. 37Signals would see significant savings in development and maintenance costs -- and increased customer satisfaction -- if they approached this staffing requirement as a band-aid, not as a final solution, and took a long, considered look at the root cause of t…
Couldn't agree more. For a long time I was a developer at a financial company, where there was a similar rotation for level 2 support programmer. For a week every 5-6 weeks, you could expect phone calls at 3 AM when European users had a problem.
An absolute majority of these issues were caused by bad system architecture issues (which, to be fair, in financial company is usually not up to the technical people to solve).
And also, today a much better solution is available: instead of requiring people to be on call at nighttime, why not hire people in different time zones, across the world, specifically for the purpose of Level 2 support when your main devs are asleep?
Re: Putting out fires at 37signals: The on-call programmer
#7Honestly, this sounds like a nightmare. It brings me back to my sysadmin days when I was getting paid $10/hour.
I would need to get paid lots of money to do this (dig into my precious free time). Probably more than 37signals is ever willing to may me.
A buddy of mine is a sysadmin and told me that at his work, only the "best" techs get this duty. The company makes it sound like an honor to get pager duty and have to deal with putting out fires at 2am.
Re: Putting out fires at 37signals: The on-call programmer
#8Don't be too quick to condemn 37signals for needing on-call programmers. For many startups, the process goes like this: all devs are always on-call. It seems that 37signals at least makes the requirements of the job clear. The fact is, running a live service almost always requires some degree of live support. (Even the most robust production software will experience the occasional hiccup.) But it does seem like they'…
Unless it's major server failure, I really don't see a need to have immediate customer support. Most issues can be solved the next day/a few hours later.
Re: Putting out fires at 37signals: The on-call programmer
#9I guess that's what happens when you don't have enough tests...
Re: Putting out fires at 37signals: The on-call programmer
#10(I work at 37signals, though not as a sysadmin or developer)
Just to clarify - we do have a 24/7 on-call system administrator who is the first line of defense for when things go wrong. They're the ones who get phone calls when things do go 'bump' in the night, and they're fantastic in every way.
Our "on call" developers fix customer problems; rarely do these arise suddenly in the middle of the night, but our software has bugs (like most pieces of software) that impact customers immediately, and we've found it helpful to have a couple of developers at a time who focus on fixing those during business hours rather than working on a longer term project. Most companies probably don't call this "on call", but rather something like (as a commenter on the original post pointed out) "second level support". This is what Nick was describing in his post.
Of course, fixing root causes is the best way to solve bugs, and we do a lot of this too. We've taken a significant dent (>= 30% reduction) out of our "on call" developer load over the last 6-12 months by going after these root cause issues.
Hope that clarifies the situation some.