Putting out fires at 37signals: The on-call programmer • Hacker News Board

Re: Putting out fires at 37signals: The on-call programmer

#31

Post by anthonyb » Tue, Apr 17, 2012, 12:42 AM UTC

jedberg wrote: post #26

anthonyb wrote: post #24

Earlier quoted context omitted.

> Tests don't magically help you invent features/work around introduced issues in 3rd party systems. Uh, yes they do. You want a unit or system test which covers the case where an external system is down or returns something that you can't parse. Something like: # code to take third party thing down # eg. mock out lib and return nonsense (unit tests) # or add an /etc/hosts entry (system tests) assert "Sorry, but that…

(I work for Netflix) It's funny that you mention the Chaos Monkey, considering that Netflix has 24/7 on call programmers for tier 1 support. We do however also make great efforts to make sure that we are resilient as possible to failure of 3rd party services.

I suspect you also pay for your 3rd party services, which the GP's company doesn't seem to do.

Re: Putting out fires at 37signals: The on-call programmer

#32

Post by rdl » Tue, Apr 17, 2012, 12:45 AM UTC

paulhauggis wrote: post #7

Honestly, this sounds like a nightmare. It brings me back to my sysadmin days when I was getting paid $10/hour. I would need to get paid lots of money to do this (dig into my precious free time). Probably more than 37signals is ever willing to may me. A buddy of mine is a sysadmin and told me that at his work, only the "best" techs get this duty. The company makes it sound like an honor to get pager duty and have to…

There are ways to do ops which don't suck as much for the ops people -- for exempt/salaried, offering liberal comp time for taking on-call shifts, and even more if there are alerts, is pretty nice. And making sure all the tools for on-call people are as convenient as possible.

There surely is some price where being woken up is worth it to you. If it happens once a month and you get a day off the next week, I'm pretty happy.

Re: Putting out fires at 37signals: The on-call programmer

#33

Post by efsavage » Tue, Apr 17, 2012, 12:47 AM UTC

"We spend little time investigating crash bugs."

Isn't "not crashing" kind of an implicit responsibility of any programmer? There are some bugs that aren't worth fixing, but even the most rare set of circumstances shouldn't be causing a crash for very long.

Re: Putting out fires at 37signals: The on-call programmer

#34

Post by dmpk2k » Tue, Apr 17, 2012, 12:57 AM UTC

bryanl wrote: post #25

jtchang wrote: post #20

Is this seriously a post highlighting the heroics of being on-call?! Wake up -- being on call sucks. Being an on call programmer is even worse. All developers should have to work support sometime in their life to realize the pain of supporting software vs writing it. Only then will you realize why doing it "right" the first time really matters. I kind of agree with the first comment on that post from Alice Young. Eve…

I believe that programmers shouldn't have to work support "sometime in their life", they should be working it at their current position. Sometimes it is all too easy to throw the problems over the fence to a tech ops (fancy name for sysadmins?) or the even worse -- the dreaded app support team. Having to live with the decisions your code makes can hopefully only make it better.

In theory that sounds great. What I've seen and heard from practice (surprisingly) isn't.

I think a better idea is to have excellent communication between the two groups, whether on channels or in the same room. Both specialize, but both have the ready support of the other. It's unnecessary to go further.

That's what we do at my current gig. It has worked out well.

Re: Putting out fires at 37signals: The on-call programmer

#35

Post by biot » Tue, Apr 17, 2012, 1:05 AM UTC

I'm curious to know what compensation people receive for being on-call, either as a percentage of salary or flat rate.

(I'd submit a poll, but it appears from http://news.ycombinator.com/newpoll that polls are currently turned off.)

Re: Putting out fires at 37signals: The on-call programmer

#36

Post by johngalt » Tue, Apr 17, 2012, 1:15 AM UTC

Programmers shouldn't be on-call, but they should probably listen to the sysadmins who are.

I'll never understand why it's so common to use programmers as IT/Sysadmins. Operating a working system is fundamentally different than building it. No one would expect a ship designer to be a captain. Sure there is enough overlap to make it possible, but why not have them each handle their specialty?

If you've never experienced a good IT person backing you up I encourage you to try it. Detailed reports of failures/bottlenecks/repeatable issues. Problems already localized, and identified. No getting up at 2am!

Re: Putting out fires at 37signals: The on-call programmer

#37

Post by elliotanderson » Tue, Apr 17, 2012, 1:42 AM UTC

jtchang wrote: post #20

Is this seriously a post highlighting the heroics of being on-call?! Wake up -- being on call sucks. Being an on call programmer is even worse. All developers should have to work support sometime in their life to realize the pain of supporting software vs writing it. Only then will you realize why doing it "right" the first time really matters. I kind of agree with the first comment on that post from Alice Young. Eve…

You're equating complexity with support assurance.

From what they have written in the past, everyone has a share in providing customer support so that the developers don't become disconnected from the customers actual needs. Small annoyances are easy to ignore when there's a layer of support personnel insulating you.

I've been in a similar situation developing a SaaS ecommerce application, so from my point of view what they are saying makes perfect sense. It's an assurance to their customers that they have proper escalation and continuity plans in place, and should anything out of the ordinary arise there are developers rostered on. This kind of setup would be easy for them to implement considering their employee's are spread across many timezones.

Re: Putting out fires at 37signals: The on-call programmer

#38

Post by jacques_chester » Tue, Apr 17, 2012, 2:02 AM UTC

biot wrote: post #35

I'm curious to know what compensation people receive for being on-call, either as a percentage of salary or flat rate. (I'd submit a poll, but it appears from http://news.ycombinator.com/newpoll that polls are currently turned off.)

A friend of mine used to work in a dinosaur pen. He's moved up, but he is still rotated through on-call periods because of his familiarity with the particular outfit he works for.

He receives several hundred dollars over his base salary per week to be on-call; he then receives a minimum of three hours pay at the maximum penalty rate when he takes a phone call.

Given how stressful being on-call can be, I think he earns every dollar. His social life is constrained; getting a 2AM phone call and having to login or drive to the data centre to troubleshoot is hell on sleeping patterns.

The expense of calling him in also encourages the relevant shift managers to think carefully about whether they need to bump the issue up or to recheck it themselves.

If I was in the position of requiring on-call staff of any kind, I would endeavour to have a similar set of rules in place.

Re: Putting out fires at 37signals: The on-call programmer

#39

Post by elliotanderson » Tue, Apr 17, 2012, 2:07 AM UTC

They're a geographically spread out company with employees spanning multiple timezones. They work in small teams and cycle their programmers into the support teams to get them on the front lines. The programmers in the support teams are "on-call" for issues that come up, skipping the need to send the issue over the fence and take someone off application development.

Whats the controversy? Despite the name of the position, it sounds like its just the role they assume in day to day work rather than fighting fires every couple of days.

Re: Putting out fires at 37signals: The on-call programmer

#40

Post by wvenable » Tue, Apr 17, 2012, 2:16 AM UTC

nupark2 wrote: post #2

A requirement for 24/7 on-call programmers demonstrates a systemic organizational failure in the design and implementation of robust, well-architected software. 37Signals would see significant savings in development and maintenance costs -- and increased customer satisfaction -- if they approached this staffing requirement as a band-aid, not as a final solution, and took a long, considered look at the root cause of t…

I disagree. All software has bugs. I do this sort of support in my company although I don't have to: a ticket will come in or an automatic bug report and if it's not to onerous I'll fix it immediately and update the ticket. Could it wait till tomorrow? Perhaps. But I prefer the customer get back to their work as soon as possible.

Most problems of this sort are not very serious -- if you've got serious problems all night then that's a systemic organizational failure)