If you write such awesome well tested code you won't mind being primary on-call to support it, since it won't break and you won't get paged.
Putting out fires at 37signals: The on-call programmer
41–48 of 48 posts
Re: Putting out fires at 37signals: The on-call programmer
#42I'm curious to know what compensation people receive for being on-call, either as a percentage of salary or flat rate. (I'd submit a poll, but it appears from http://news.ycombinator.com/newpoll that polls are currently turned off.)
Nothing for actually being on call. 2x hourly rate for time worked (time worked is rounded up to nearest 1/2 hour before being doubled). Quiet weeks suck. We're continually reminded how "generous" this is. I would say my salary reflects oncall duties, but only a very small amount
Re: Putting out fires at 37signals: The on-call programmer
#43"We spend little time investigating crash bugs." Isn't "not crashing" kind of an implicit responsibility of any programmer? There are some bugs that aren't worth fixing, but even the most rare set of circumstances shouldn't be causing a crash for very long.
I think he means that there are rarely crash bugs that need fixing, therefore, little time is spent fixing them
Re: Putting out fires at 37signals: The on-call programmer
#44Is this seriously a post highlighting the heroics of being on-call?! Wake up -- being on call sucks. Being an on call programmer is even worse. All developers should have to work support sometime in their life to realize the pain of supporting software vs writing it. Only then will you realize why doing it "right" the first time really matters. I kind of agree with the first comment on that post from Alice Young. Eve…
Depends in what you like to do at work. Writing code isis boring most of the time. The most exciting thing that can happen is trying to figure out a design for a very complicated problem and sure, that's fun stuff. But I find handling production emergencies much more exciting. Whatever you test for, whatever you monitor, one day you find the system down or misbehaving in a completely unexpected way. And if your service is supposed to run 24/7 worldwide, someone has to fix it, preferably as soon as possible and making sure as few people are affected as possible.
In short, different people like different kind of work. And that's ok.
Re: Putting out fires at 37signals: The on-call programmer
#45Programmers shouldn't be on-call, but they should probably listen to the sysadmins who are. I'll never understand why it's so common to use programmers as IT/Sysadmins. Operating a working system is fundamentally different than building it. No one would expect a ship designer to be a captain. Sure there is enough overlap to make it possible, but why not have them each handle their specialty? If you've never experienc…
So what exactly are you proposing for a situation where the system fails for a large number of people and it's not a platform / sysadmin level issue? Assuming you're running 24/7 service with sla in place... Basically you need someone who knows your code and knows how to code.
To go with your ship analogy, no, the ship designer is like an solution architect who may never code any of it. In reality cruise ships carry whole engineering / maintenance teams on board in case of problems. I wouldn't be surprised if many of them were involved in building parts of some ship in the past.
Re: Putting out fires at 37signals: The on-call programmer
#46Don't be too quick to condemn 37signals for needing on-call programmers. For many startups, the process goes like this: all devs are always on-call. It seems that 37signals at least makes the requirements of the job clear. The fact is, running a live service almost always requires some degree of live support. (Even the most robust production software will experience the occasional hiccup.) But it does seem like they'…
Unless it's major server failure, I really don't see a need to have immediate customer support. Most issues can be solved the next day/a few hours later.
Or perhaps 37signals is of a size that losing a small % of subscriptions due to an issue is considerably larger cost than placing a couple of people on-call to deal with it immediately.
Re: Putting out fires at 37signals: The on-call programmer
#47I like how quite a number of peoples answers to the on-call programmer blog was "you need better tests" here's a what if scenario:- - you have a third party service your systems rely on - at 4am on Sunday morning said 3rd party service upgrades their system, introducing a breaking change, having never bothered to notify users - you get a call as the on-call person saying "application X is not longer working, please r…
> Tests don't magically help you invent features/work around introduced issues in 3rd party systems. Uh, yes they do. You want a unit or system test which covers the case where an external system is down or returns something that you can't parse. Something like: # code to take third party thing down # eg. mock out lib and return nonsense (unit tests) # or add an /etc/hosts entry (system tests) assert "Sorry, but that…
I used to work support for a SMS aggregator. The core business was delivering texts to mobile networks. Mobile networks break all the time, it turns out. And can't really be replaced - there's only one T-Mobile.
Now, when they died or sent garbage, our software wouldn't crash. But our service would stop functioning. And alarms would go off, and we would have to confirm why, and call them, and ask them if they knew if it had stopped working and why.
(The follow up was indeed to always ask to be added to the mailing list that would let us know this ahead of time. These proved remarkably unhelpful. In once case, we ended up setting up having to set up a mobile number to receive SMS alerts - this was apparently the only way they would notify anyone?)
And reasonably often actual engineers would be needed to be woken up. And in several cases, change the parsing behaviour so we could handle sudden unexpected changes in the format they returned. Yes, in the middle of the night.
Our system wasn't perfect, but I'm not sure our problem was simply a lack of system or unit tests.
During the day, we largely dealt with more minor customer complaints, and ongoing maintenance, outages and other nonsense fromt he carriers. Engineers would often have to dig into these edge cases too, and there was a nominated maintainer to look at this stuff, to let the rest of the team add features and work on long term fixes. But sometimes you just need someone to wedge LargeCustomer's encoding settings because they can't figure out how to properly specify it on their end.
Re: Putting out fires at 37signals: The on-call programmer
#48Earlier quoted context omitted.
Unless it's major server failure, I really don't see a need to have immediate customer support. Most issues can be solved the next day/a few hours later.
Or perhaps 37signals is of a size that losing a small % of subscriptions due to an issue is considerably larger cost than placing a couple of people on-call to deal with it immediately.
As a customer, if 37signals got back to me at the next day as opposed to 3am that night, I wouldn't see a problem with it. I seriously double they will lose any subscriptions.
Like I said, on-call should only be used for catastrophic server failures.
Most people don't need that kind of support.