Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Putting out fires at 37signals: The on-call programmer

37signals.com

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Putting out fires at 37signals: The on-call programmer

#21

Post by vitovito »

nupark2 wrote: post #13
vitovito wrote: post #12

I have to assume all of the other comments in this thread are from small shops that have never supported a live product. We run a multi-hundred person team here for a live, 24/7 product, and as many as half of our developers have been scheduled as "on-call programmers," which we call our Live team. Their sole responsibility is the live, deployed product and customer-impacting issues. They do no bug fixes outside of t…

I've supported (what I would expect to be) an equivalently large deployment. If you truly have half of a multi-hundred person development team scheduled simply to respond to emergency on-call events, you very, very likely have fundamental issues in your development standards and processes leading to those events. That's simply a tremendous percentage of your staff dedicated to putting out fires.

Well, some of them are artists and designers, too, and this isn't just a web site, it's a desktop product and an online service, and the proportion changes depending on where features are in development and what sort of load we're seeing on customer-facing issues, but, yes, there have been occasions where half of our web and infrastructure staff have been doing "live" development and support.

And that's the thing: they're not "emergency on-call" events. They're simply "customer-facing issues." With a 24/7 product and 1.7M subscribers, things come up. They're not "fires." They're "live" issues. They're always there.

The 37s post is not about emergency staff, even if they're using those types of words. It's about having dedicated personnel to handle technical issues arising from a customer support ticket, so the "new feature" programmers don't have to get pulled away unless they have the only knowledge of that particular system (which doesn't happen too often here any more).

Re: Putting out fires at 37signals: The on-call programmer

#22

Post by joedev »

nupark2 wrote: post #2

A requirement for 24/7 on-call programmers demonstrates a systemic organizational failure in the design and implementation of robust, well-architected software. 37Signals would see significant savings in development and maintenance costs -- and increased customer satisfaction -- if they approached this staffing requirement as a band-aid, not as a final solution, and took a long, considered look at the root cause of t…

Couldn't agree more. I left an otherwise good job just for this reason: I was hired as a programmer, built a 15+ year career as a programmer, and while I love programming, I hate late-night systems support. Most companies do not staff this way so it was easy to find other jobs where programmers are not expected to be at the company's beck-and-call 24/7.

Re: Putting out fires at 37signals: The on-call programmer

#23

Post by vitovito »

Smudge wrote: post #15
vitovito wrote: post #12

I have to assume all of the other comments in this thread are from small shops that have never supported a live product. We run a multi-hundred person team here for a live, 24/7 product, and as many as half of our developers have been scheduled as "on-call programmers," which we call our Live team. Their sole responsibility is the live, deployed product and customer-impacting issues. They do no bug fixes outside of t…

But root-cause analysis of the issues that arise is still important, right? Tracing customer-impacting issues back to the decisions that might have caused them? Perhaps not even particular bugs or parts of the product, but architectural decisions as a whole? Or even organizational processes? You can throw money at live support until the end of time, but the only way to reduce that cost is by addressing problems at th…

Sure, and the "live" team is made up of people from the regular "new feature" team, rotated in and out so they all understand the impact their code has.

But, in a product that gets used by lots of real users, shit happens. You're never going to get everything right the first time.

Re: Putting out fires at 37signals: The on-call programmer

#24

Post by anthonyb »

malbs wrote: post #17

I like how quite a number of peoples answers to the on-call programmer blog was "you need better tests" here's a what if scenario:- - you have a third party service your systems rely on - at 4am on Sunday morning said 3rd party service upgrades their system, introducing a breaking change, having never bothered to notify users - you get a call as the on-call person saying "application X is not longer working, please r…

> Tests don't magically help you invent features/work around introduced issues in 3rd party systems.

Uh, yes they do. You want a unit or system test which covers the case where an external system is down or returns something that you can't parse. Something like:

  # code to take third party thing down 
  # eg. mock out lib and return nonsense (unit tests)
  # or add an /etc/hosts entry (system tests)
  assert "Sorry, but that feature is unavailable." in page.content
Now the entire app doesn't asplode, and you can wait until 9am to fix it. Follow up is to make sure that you're on whatever mailing list tells you when changes are coming.

The only case that this doesn't cover is when it's a) an essential part of your app, which b) you aren't paying for and c) they don't have a mailing list, in which case wtf? you need to find a better 3rd party library/service.

ps. Look up the "chaos monkey" - it's very enlightening :)

Re: Putting out fires at 37signals: The on-call programmer

#25

Post by bryanl »

jtchang wrote: post #20

Is this seriously a post highlighting the heroics of being on-call?! Wake up -- being on call sucks. Being an on call programmer is even worse. All developers should have to work support sometime in their life to realize the pain of supporting software vs writing it. Only then will you realize why doing it "right" the first time really matters. I kind of agree with the first comment on that post from Alice Young. Eve…

I believe that programmers shouldn't have to work support "sometime in their life", they should be working it at their current position. Sometimes it is all too easy to throw the problems over the fence to a tech ops (fancy name for sysadmins?) or the even worse -- the dreaded app support team. Having to live with the decisions your code makes can hopefully only make it better.

Re: Putting out fires at 37signals: The on-call programmer

#26

Post by jedberg »

anthonyb wrote: post #24
malbs wrote: post #17

I like how quite a number of peoples answers to the on-call programmer blog was "you need better tests" here's a what if scenario:- - you have a third party service your systems rely on - at 4am on Sunday morning said 3rd party service upgrades their system, introducing a breaking change, having never bothered to notify users - you get a call as the on-call person saying "application X is not longer working, please r…

> Tests don't magically help you invent features/work around introduced issues in 3rd party systems. Uh, yes they do. You want a unit or system test which covers the case where an external system is down or returns something that you can't parse. Something like: # code to take third party thing down # eg. mock out lib and return nonsense (unit tests) # or add an /etc/hosts entry (system tests) assert "Sorry, but that…

(I work for Netflix)

It's funny that you mention the Chaos Monkey, considering that Netflix has 24/7 on call programmers for tier 1 support.

We do however also make great efforts to make sure that we are resilient as possible to failure of 3rd party services.

Re: Putting out fires at 37signals: The on-call programmer

#27

Post by malbs »

anthonyb wrote: post #24
malbs wrote: post #17

I like how quite a number of peoples answers to the on-call programmer blog was "you need better tests" here's a what if scenario:- - you have a third party service your systems rely on - at 4am on Sunday morning said 3rd party service upgrades their system, introducing a breaking change, having never bothered to notify users - you get a call as the on-call person saying "application X is not longer working, please r…

> Tests don't magically help you invent features/work around introduced issues in 3rd party systems. Uh, yes they do. You want a unit or system test which covers the case where an external system is down or returns something that you can't parse. Something like: # code to take third party thing down # eg. mock out lib and return nonsense (unit tests) # or add an /etc/hosts entry (system tests) assert "Sorry, but that…

a, b, and c, and there is no other / better service, so it's a difficult one to solve.

also things can't wait until 9am or the selected waking hours, we have too many people/systems relying on working infrastructure, so an issue popping up at an ungodly hour is fixed there and then, even if it means calling other people.

Those are the worst support calls, 3am on some weeknight, and you can't actually fix the problem because you're not 100% certain, and you need to call a colleague and wake them up too. You feel like an asshole.

Re: Putting out fires at 37signals: The on-call programmer

#28

Post by shepbook »

"I spend one week every ten or so, on call. Then I spend the next nine weeks writing code to make my next on call shift better." - Tom Limoncelli

Sure, people may write off the fact that Tom found his niche in systems administration. He's currently at Google, as a "Site Reliability Engineer" which (in case you aren't familiar) is about 40% development work and 60% systems administration work. (Though his recent project, Ganeti, seems far more development work.)

I find it "amusing" how so many people are all "DevOps! DevOps! DevOps!" _until_ it causes some kind of inconvenience for the developer. (Pesky paying clients! Why must you want what you paid for, to work!) Then it's "Make the sysadmin's do it. That's Ops job. It's not my job, as a developer, to help fix the service when it breaks. I write the code... it's your job to make it work, sysadmins..." Operability is _everyone's_ responsibility. If your code fails, for whatever reason, it should fail gracefully. It should tell us why it failed. This is the basis of operable code. Of course, even with testing or the best, possible, operable code, shit will still happen.

I think the division of labor is simple. If the failure is clearly software related (you know this because you monitor your systems/software), the on call developer is paged. If the failure is hardware or core OS/system related, the sysadmin is paged. If shit's on fire, both are paged.

Yes, we all know "Well Designed Systems and Software" shouldn't experience catastrophic failure. Guess what, it happens, no matter how well you prepare. So, you prepare for the worst case and have processes in place on how to deal with such issues. Drill your developers and sysadmins. Preparation is key.

Ultimately, _everyone_ on your team should carry the title of "Chief Make It Fucking Work Officer". If you don't get this, don't sit here and gripe about "Not being DevOps-y enough" as is so prevalent in what I read and hear these days. When the Sysadmin says, "No, you aren't pushing code today.", don't bitch. Perhaps if developers accepted responsibility for helping support the systems and software they write, the Sysadmins would be more open to working with the developers.

DevOps Motherfucker. Do You (do more than just) Speak It?

Re: Putting out fires at 37signals: The on-call programmer

#29

Post by anthonyb »

malbs wrote: post #27
anthonyb wrote: post #24

Earlier quoted context omitted.

> Tests don't magically help you invent features/work around introduced issues in 3rd party systems. Uh, yes they do. You want a unit or system test which covers the case where an external system is down or returns something that you can't parse. Something like: # code to take third party thing down # eg. mock out lib and return nonsense (unit tests) # or add an /etc/hosts entry (system tests) assert "Sorry, but that…

a, b, and c, and there is no other / better service, so it's a difficult one to solve. also things can't wait until 9am or the selected waking hours, we have too many people/systems relying on working infrastructure, so an issue popping up at an ungodly hour is fixed there and then, even if it means calling other people. Those are the worst support calls, 3am on some weeknight, and you can't actually fix the problem…

> a, b, and c, and there is no other / better service, so it's a difficult one to solve.

You don't say what the service is, but if your company is relying on it to the extent that you need to be awake at 3am, then I suspect it's well worth calling the people providing it and offering to throw money at them.

Otherwise you're essentially relying on their goodwill for business continuity...

Post reply on HN